Discussion: View Thread

Non Responders in Surveys: how do you handle them?

  • 1.  Non Responders in Surveys: how do you handle them?

    Posted 12-17-2014 18:20

    I would appreciate getting feedback on what would be the most appropriate way to report survey results with non-responders.

    For example if one question in a multi-question survey asked:
    Would you be willing to replace your current product with the evaluation product? 
    "Yes" and "No" were the only options. 
    72 forms were received back: 4 had no response to that question and the other 68 responded "Yes"

    Would statistician recommend to report results as:
    100% of those responding said "Yes"
    or
    94% of evaluators said "Yes"
    or
    something else?

    Thanks!

    (looking for a good solution to an ongoing debate with marketing...I cannot generate new data as a solution)

    -------------------------------------------
    Shelley-Ann Walters
    3M
    -------------------------------------------


  • 2.  RE: Non Responders in Surveys: how do you handle them?

    Posted 12-17-2014 18:34
    The most honest answer would include response rate as part of the survey results (lookin' at you, news media).
    I'd be happy seeing something like...
    There were 100 surveyed and 54 responses to Question 1.  And of those responding, 27 would vote for Candidate X...

    The response rate can provide some form of feedback regarding the survey question. Low response questions on a survey might be due to questions worded poorly, confusing to the reader, sensitive topic to the reader, etc.


    -------------------------------------------
    Mark Lancaster
    Northern Kentucky University
    -------------------------------------------




  • 3.  RE: Non Responders in Surveys: how do you handle them?

    Posted 12-17-2014 18:55
    I would definitely word it as Mark did as "Population Surveyed" and "Response Rate" are two totally different things.

    -------------------------------------------
    Suzann Williams
    -------------------------------------------




  • 4.  RE: Non Responders in Surveys: how do you handle them?

    Posted 12-17-2014 19:34
    As you probably have several other variables measured on the survey, you may also apply some sort of clustering methods to find similar groups. Then, locate the group(s) where these four respondents fall. This might 'discover' if there was something common in these four responses that led to the non-response. 

    -Khurram


    ------Original Message------

    I would definitely word it as Mark did as "Population Surveyed" and "Response Rate" are two totally different things.

    -------------------------------------------
    Suzann Williams
    -------------------------------------------




  • 5.  RE: Non Responders in Surveys: how do you handle them?

    Posted 12-17-2014 18:36
    I would conservatively report 94% with a fine print defining a positive response as Yes to the question and everything else being treated as a negative response. Of course, whether your sample is representative of your target population and/or your results could be generalized to other populations is a totally different matter.

    Daohai


    ------Original Message------


    I would appreciate getting feedback on what would be the most appropriate way to report survey results with non-responders.

    For example if one question in a multi-question survey asked:
    Would you be willing to replace your current product with the evaluation product? 
    "Yes" and "No" were the only options. 
    72 forms were received back: 4 had no response to that question and the other 68 responded "Yes"

    Would statistician recommend to report results as:
    100% of those responding said "Yes"
    or
    94% of evaluators said "Yes"
    or
    something else?

    Thanks!

    (looking for a good solution to an ongoing debate with marketing...I cannot generate new data as a solution)

    -------------------------------------------
    Shelley-Ann Walters
    3M
    -------------------------------------------


  • 6.  RE: Non Responders in Surveys: how do you handle them?

    Posted 12-17-2014 18:37
    Hello Shelly,

    Report it like it is, 68 respondents (94% of all respondents) said yes and 4 respondents (6%) did not answer the question.

    You don't know why those 4 people didn't respond. Maybe they weren't sure if they would change out their product and didn't want to commit to either yes or no. Or, maybe they misunderstood the question. Or...any number of other reasons. But just report it like it is. And 94% yes is a majority in any situation, so it is understood as it is that a vast majority of respondents said yes.

    Best,

    Elaine

    -------------------------------------------
    Elaine Eisenbeisz
    Owner and Principal Statistician
    Omega Statistics
    -------------------------------------------




  • 7.  RE: Non Responders in Surveys: how do you handle them?

    Posted 12-18-2014 09:30
    I agree with Elaine - it's clear, concise, includes all the information...and *any* reader will understand it.   :-)

    -------------------------------------------
    Wayne Fischer
    Statistician
    University of Texas Medical Branch
    -------------------------------------------




  • 8.  RE: Non Responders in Surveys: how do you handle them?

    Posted 12-17-2014 19:20
    As you probably have several other variables measured on the survey, you may also apply some sort of clustering methods to find similar groups. Then, locate the group(s) where these four respondents fall. This might 'discover' if there was something common in these four responses that led to the non-response. 

    -------------------------------------------
    Khurram Nadeem
    Postdoctoral Fellow
    Acadia University
    Wolfville, Nova Scotia
    -------------------------------------------




  • 9.  RE: Non Responders in Surveys: how do you handle them?

    Posted 12-18-2014 07:33

    Hi Shelley,

    I think you have entered the "realm" of determining best ways to handle missing data.  

    If you are treating the data as though it were a simple random sample, you could

    1) create a category consisting of the "non-responders" and include them in your analysis

    2) assume that data are missing completely at random and ignore the non-response.

    If the you did have variability in your (non-missing)  responses you could

    3) attempt an imputation method to estimate the missing data

    OR

    if you were dealing with a complex survey sample, you could get proportions using sampling weights that have been adjusted for non-response.

    Happy to discuss further via Skype if you wish.

     

    Best regards,

     

    Novie

     

     

    <signature_tag>

    Sent from my LG Mobile

    </signature_tag>

     



    ------Original Message------


    I would appreciate getting feedback on what would be the most appropriate way to report survey results with non-responders.

    For example if one question in a multi-question survey asked:
    Would you be willing to replace your current product with the evaluation product? 
    "Yes" and "No" were the only options. 
    72 forms were received back: 4 had no response to that question and the other 68 responded "Yes"

    Would statistician recommend to report results as:
    100% of those responding said "Yes"
    or
    94% of evaluators said "Yes"
    or
    something else?

    Thanks!

    (looking for a good solution to an ongoing debate with marketing...I cannot generate new data as a solution)

    -------------------------------------------
    Shelley-Ann Walters
    3M
    -------------------------------------------


  • 10.  RE: Non Responders in Surveys: how do you handle them?

    Posted 12-18-2014 09:34
    Responders are emergency management professionals. People filling survey are called respondents.

    Literature on survey non-response is extensive, although that's not the stuff you would see taught in the "proper" statistics courses. An edited volume by Groves et. al. (2001) (http://www.amazon.com/Survey-Nonresponse-Robert-M-Groves/dp/0471396273) is a good starting point. Another major professional organization, American Association for Public Opinion Research, has recently released the report of the task force on survey refusals, http://www.aapor.org/AAPORKentico/AAPOR_Main/media/MainSiteFiles/RefusalTF_FINAL090814.pdf. These resources go deeper to try and figure out what's going on in respondents' heads as they formulate their responses, or decide not to respond. AAPOR also provides guidance on how to report issues of non-response, see http://www.aapor.org/AAPORKentico/Communications/AAPOR-Journals/Standard-Definitions.aspx.

    -------------------------------------------
    Stanislav Kolenikov
    Abt SRBI
    -------------------------------------------




  • 11.  RE: Non Responders in Surveys: how do you handle them?

    Posted 12-18-2014 11:20
    I would report both and provide a lower 95% bound on each. For 100% the quick lower 95% bound is approximately 1-[3/(n+1)] i.e. 1-[3/69]~=0.957 and for the other one can compute in any number of ways

    -------------------------------------------
    Borko Jovanovic
    Associate Professor in Preventive Medicine
    Northwestern University, Feinberg School of Medicine
    -------------------------------------------