ASA Connect

I suggest a fifth reply such as "neither disagree nor agree", something indicating the repondent doesn't lean toward either direction. That way, if the replies are numerically coded, the "disagrees" can have values, say, -2 and -1, the "neither/nor" a value of 0 and the "agrees" values of +1 and +2. 

------------------------------
Nestor Rohowsky
President and Principal Consultant
Integrated Data Consultation Services, Inc.
------------------------------

Original Message:
Sent: 03-23-2017 00:28
From: Kevin Rees
Subject: question about survey questions

I posted this to the ASA Statistical Education section, but wanted to get a wider swath of responses.  My apologies if you have already seen this.

Everyone,
I had a question posed to me during a department meeting today that I was unsure how to answer, so I promised to reach out for some feedback.

Our school is redoing Student Evaluations of teachers and a committee has created a set of questions that have four possible responses.  For example:  "TEACHER provides feedback that helps me to understand how to improve."  with four responses:  Strongly agree, Agree more than disagree, disagree more than agree, and strongly disagree.

Here is my question to all of you - is there a statistical or survey analysis reason to use an odd (for this case five) number of responses to a question like this instead of four responses?  The feeling of our department was that odd (3, 5 or 7) number of responses was preferable to even (4, or 6) number of responses, but it is a gut feeling.

Does anyone have any information that might be helpful about this?

Thank you,
Kevin

in addition, there may be some questions for which the appropriate answer, for some people at least, is 'not applicable'.

------------------------------
Ellen Hertzmark
------------------------------

Original Message:
Sent: 03-24-2017 07:49
From: Nestor Rohowsky
Subject: question about survey questions

I suggest a fifth reply such as "neither disagree nor agree", something indicating the repondent doesn't lean toward either direction. That way, if the replies are numerically coded, the "disagrees" can have values, say, -2 and -1, the "neither/nor" a value of 0 and the "agrees" values of +1 and +2. 

------------------------------
Nestor Rohowsky
President and Principal Consultant
Integrated Data Consultation Services, Inc.
------------------------------

Original Message:
Sent: 03-23-2017 00:28
From: Kevin Rees
Subject: question about survey questions

I posted this to the ASA Statistical Education section, but wanted to get a wider swath of responses.  My apologies if you have already seen this.

Everyone,
I had a question posed to me during a department meeting today that I was unsure how to answer, so I promised to reach out for some feedback.

Our school is redoing Student Evaluations of teachers and a committee has created a set of questions that have four possible responses.  For example:  "TEACHER provides feedback that helps me to understand how to improve."  with four responses:  Strongly agree, Agree more than disagree, disagree more than agree, and strongly disagree.

Here is my question to all of you - is there a statistical or survey analysis reason to use an odd (for this case five) number of responses to a question like this instead of four responses?  The feeling of our department was that odd (3, 5 or 7) number of responses was preferable to even (4, or 6) number of responses, but it is a gut feeling.

Does anyone have any information that might be helpful about this?

Thank you,
Kevin

Original Message------

in addition, there may be some questions for which the appropriate answer, for some people at least, is 'not applicable'.

------------------------------
Ellen Hertzmark
------------------------------

I agree with Ed: tailor the responses to the question being asked, rather than using a "generic" agree/disagree type scale.

------------------------------
Martha Smith
University of Texas
------------------------------

Original Message:
Sent: 03-27-2017 08:57
From: Edward Gracely
Subject: question about survey questions

I have a different comment, for what it's worth. I use agree-disagree scales much less than other people. They force respondents to first decide how they feel about the issue (in this case, providing useful feedback) then figure out where that would fall on an agreement scale. If you think the teacher provides some but not enough feedback, is that strongly agree (the teacher does provide it, so the answer is clearly "yes") or disagree (because it isn't enough)? Or what?

I would ask something like, "How well does this teacher's feedback  help you understand how to improve?  No feedback provided; Not helpful;  a little helpful; somewhat helpful; quite helpful; very helpful". I wouldn't defend the exact question or answers, made up on the fly, but overall I think a question like this is more interpretable.

Ed

Ed J. Gracely, PhD
Associate Professor
Family, Community, & Preventive Medicine
College of Medicine

Associate Professor
Epidemiology and Biostatistics
Dornsife School of Public Health

Drexel University
2900 W. Queen Lane,
Philadelphia PA, 19129
Tel: 215.991.8466  | Fax: 215.843.6028
Cell: 609.707.6965
Egracely@drexelmed.edu
drexelmed.edu  |  drexel.edu/publichealth

---

This email and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this email communication by others is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and delete all copies. Thank you for your cooperation.

Exactly.  Agree/disagree is often inappropriate.  It's also crucial to phrase the question unambiguously and to make sure that the literal question asked is the one you want the respondent to answer.  And then craft the possible responses so that they literally correspond to reasonable answers.

I posted an example of a poorly phrased question on the Education discussion list.  It led to what could have been a disaster.

------------------------------
Emil M Friedman, PhD
emilfriedman@gmail.com
http://www.statisticalconsulting.org
------------------------------

Original Message:
Sent: 03-28-2017 17:53
From: Martha Smith
Subject: question about survey questions

I agree with Ed: tailor the responses to the question being asked, rather than using a "generic" agree/disagree type scale.

------------------------------
Martha Smith
University of Texas
------------------------------

Original Message:
Sent: 03-27-2017 08:57
From: Edward Gracely
Subject: question about survey questions

I have a different comment, for what it's worth. I use agree-disagree scales much less than other people. They force respondents to first decide how they feel about the issue (in this case, providing useful feedback) then figure out where that would fall on an agreement scale. If you think the teacher provides some but not enough feedback, is that strongly agree (the teacher does provide it, so the answer is clearly "yes") or disagree (because it isn't enough)? Or what?

I would ask something like, "How well does this teacher's feedback  help you understand how to improve?  No feedback provided; Not helpful;  a little helpful; somewhat helpful; quite helpful; very helpful". I wouldn't defend the exact question or answers, made up on the fly, but overall I think a question like this is more interpretable.

Ed

Ed J. Gracely, PhD
Associate Professor
Family, Community, & Preventive Medicine
College of Medicine

Associate Professor
Epidemiology and Biostatistics
Dornsife School of Public Health

Drexel University
2900 W. Queen Lane,
Philadelphia PA, 19129
Tel: 215.991.8466  | Fax: 215.843.6028
Cell: 609.707.6965
Egracely@drexelmed.edu
drexelmed.edu  |  drexel.edu/publichealth

---

This email and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of this email communication by others is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and delete all copies. Thank you for your cooperation.

Original Message------

I posted this to the ASA Statistical Education section, but wanted to get a wider swath of responses.  My apologies if you have already seen this.

Everyone,
I had a question posed to me during a department meeting today that I was unsure how to answer, so I promised to reach out for some feedback.

Our school is redoing Student Evaluations of teachers and a committee has created a set of questions that have four possible responses.  For example:  "TEACHER provides feedback that helps me to understand how to improve."  with four responses:  Strongly agree, Agree more than disagree, disagree more than agree, and strongly disagree.

Here is my question to all of you - is there a statistical or survey analysis reason to use an odd (for this case five) number of responses to a question like this instead of four responses?  The feeling of our department was that odd (3, 5 or 7) number of responses was preferable to even (4, or 6) number of responses, but it is a gut feeling.

Does anyone have any information that might be helpful about this?

Thank you,
Kevin

There are some topics for which neutral is not a plausible answer, and your response options should not include that. If you're asking about opinions about capital punishment or a single-payer insurance model, I would use questions with an even number of response options."TEACHER provides feedback that helps me to understand how to improve." Students might have reason to agree or not agree to that sentiment. I could also picture myself as a student not having an opinion either way for some courses.

The idea is that if neutral is plausible as someone's "true score," thenthereshould be an option for that. If that is not included, then you may be introducing measurement error by forcing them to pick a direction.Furthermore, in some surveys by not providing a Neutral option you might have more non-responses (some who truly feel neutral might not answer some questions). For course feedback, I would lean toward an odd number of responses but without much conviction. I could see it either way (I more agree than disagree??). It would be a worthy pilot study to see whethertheformat changeaffects things like reliability or response rates for course evals at your school.

As far as the number of response options goes, a common recommendation I have heard is to use between 4 and 7 options. More than that is beyond the number of bits of information that people typically can keep in working memory. Less than that results in too much loss of information due to categorization (see below). Shaw et al. (1987) give a pretty straightforward look at the effect of just categorizing a continuous variable, representing the effect of using an ordinal scale to measure a hypothetical, continuous latent trait. If you categorize a normally distributed variable into 3 bins, the r-square between X and the discretized version of X is 48%. You're losing over half of the variability. For 4, 5, 6, & 7 bins, the r-squares are 67, 77, 83, and 87% respectively. If you conceptualize the items as individual predictors used to estimate each person's latent true score, those are the maximum bivariate r-squares. Other aspects of measurement error would further attenuate the bivariate relationships. 

Shaw, D. G., Huffman, M. D., & Haviland, M. G. (1987). Grouping continuous data in discrete intervals: Information loss and recovery. Journal of Educational Measurement, 24(2), 167-173.


------------------------------
Robert Pearson
Asst. Professor
Grand Valley State University
------------------------------

Original Message:
Sent: 03-24-2017 09:30
From: Heather Rollins
Subject: question about survey questions

I suspect that the "gut feel" for an odd number of choices is a psychological thing, along the lines of having to choose "random" numbers: https://blogs.msdn.microsoft.com/shawnhar/2009/12/17/the-psychology-of-randomness/

That said, I always like to have a neutral or N/A option on surveys that I fill out.  Sometimes the particular aspect in the question is something that isn't a high priority for me, and so I haven't paid attention to it.  And sometimes (thinking especially of customer satisfaction surveys at a hotel) it is genuinely not applicable. 

Best,

Heather

--
Heather Rollins

Analysis Manager, Web Teks

Graduate student in math, UWF

“Not applicable” provides a qualitatively different response that can enhance validity of the original four-point scale. In contrast, adding a middle quantitative response substantively changes the original four-point scale, doubling the difference between weak disagreement and weak agreement, relative to the difference between weakly and strongly disagreeing or between weakly and strongly agreeing.

If “not applicable” were an available answer that was not selected, the implicit neutral response could be scored as halfway between weak disagreement and weak agreement on the four-point scale, rather than missing data.


Original Message------

There are some topics for which neutral is not a plausible answer, and your response options should not include that. If you're asking about opinions about capital punishment or a single-payer insurance model, I would use questions with an even number of response options."TEACHER provides feedback that helps me to understand how to improve." Students might have reason to agree or not agree to that sentiment. I could also picture myself as a student not having an opinion either way for some courses.

The idea is that if neutral is plausible as someone's "true score," thenthereshould be an option for that. If that is not included, then you may be introducing measurement error by forcing them to pick a direction.Furthermore, in some surveys by not providing a Neutral option you might have more non-responses (some who truly feel neutral might not answer some questions). For course feedback, I would lean toward an odd number of responses but without much conviction. I could see it either way (I more agree than disagree??). It would be a worthy pilot study to see whethertheformat changeaffects things like reliability or response rates for course evals at your school.

As far as the number of response options goes, a common recommendation I have heard is to use between 4 and 7 options. More than that is beyond the number of bits of information that people typically can keep in working memory. Less than that results in too much loss of information due to categorization (see below). Shaw et al. (1987) give a pretty straightforward look at the effect of just categorizing a continuous variable, representing the effect of using an ordinal scale to measure a hypothetical, continuous latent trait. If you categorize a normally distributed variable into 3 bins, the r-square between X and the discretized version of X is 48%. You're losing over half of the variability. For 4, 5, 6, & 7 bins, the r-squares are 67, 77, 83, and 87% respectively. If you conceptualize the items as individual predictors used to estimate each person's latent true score, those are the maximum bivariate r-squares. Other aspects of measurement error would further attenuate the bivariate relationships. 

Shaw, D. G., Huffman, M. D., & Haviland, M. G. (1987). Grouping continuous data in discrete intervals: Information loss and recovery. Journal of Educational Measurement, 24(2), 167-173.


------------------------------
Robert Pearson
Asst. Professor
Grand Valley State University
------------------------------

Robert said, "There are some topics for which neutral is not a plausible answer, and your response options should not include that. If you're asking about opinions about capital punishment or a single-payer insurance model, I would use questions with an even number of response options."

I disagree with the second sentence -- "Undecided" or "Not sure" are valid opinions about capital punishment or a single-payer insurance model -- and logically they fall between "somewhat agree" and "somewhat disagree". So I believe an odd number of response options is best here.

------------------------------
Martha Smith
University of Texas
------------------------------

Original Message:
Sent: 03-24-2017 11:16
From: Robert Pearson
Subject: question about survey questions

There are some topics for which neutral is not a plausible answer, and your response options should not include that. If you're asking about opinions about capital punishment or a single-payer insurance model, I would use questions with an even number of response options."TEACHER provides feedback that helps me to understand how to improve." Students might have reason to agree or not agree to that sentiment. I could also picture myself as a student not having an opinion either way for some courses.

The idea is that if neutral is plausible as someone's "true score," thenthereshould be an option for that. If that is not included, then you may be introducing measurement error by forcing them to pick a direction.Furthermore, in some surveys by not providing a Neutral option you might have more non-responses (some who truly feel neutral might not answer some questions). For course feedback, I would lean toward an odd number of responses but without much conviction. I could see it either way (I more agree than disagree??). It would be a worthy pilot study to see whethertheformat changeaffects things like reliability or response rates for course evals at your school.

As far as the number of response options goes, a common recommendation I have heard is to use between 4 and 7 options. More than that is beyond the number of bits of information that people typically can keep in working memory. Less than that results in too much loss of information due to categorization (see below). Shaw et al. (1987) give a pretty straightforward look at the effect of just categorizing a continuous variable, representing the effect of using an ordinal scale to measure a hypothetical, continuous latent trait. If you categorize a normally distributed variable into 3 bins, the r-square between X and the discretized version of X is 48%. You're losing over half of the variability. For 4, 5, 6, & 7 bins, the r-squares are 67, 77, 83, and 87% respectively. If you conceptualize the items as individual predictors used to estimate each person's latent true score, those are the maximum bivariate r-squares. Other aspects of measurement error would further attenuate the bivariate relationships. 

Shaw, D. G., Huffman, M. D., & Haviland, M. G. (1987). Grouping continuous data in discrete intervals: Information loss and recovery. Journal of Educational Measurement, 24(2), 167-173.


------------------------------
Robert Pearson
Asst. Professor
Grand Valley State University
------------------------------

Original Message:
Sent: 03-24-2017 09:30
From: Heather Rollins
Subject: question about survey questions

I suspect that the "gut feel" for an odd number of choices is a psychological thing, along the lines of having to choose "random" numbers: https://blogs.msdn.microsoft.com/shawnhar/2009/12/17/the-psychology-of-randomness/

That said, I always like to have a neutral or N/A option on surveys that I fill out.  Sometimes the particular aspect in the question is something that isn't a high priority for me, and so I haven't paid attention to it.  And sometimes (thinking especially of customer satisfaction surveys at a hotel) it is genuinely not applicable. 

Best,

Heather

--
Heather Rollins

Analysis Manager, Web Teks

Graduate student in math, UWF

The example question asks two questions, so it confounds the response, ie the student who disagrees could be disagreeing because the prof does not provide feed back, or because the prof does provide feedback but it does not help find ways to improve. If you really want to know what the students opinion is, then having a neutral or na response is minimally necessary, but so is making sure the question does not make the interpretation of the response questionable. Another tool I often use is to ask a followup question, ie why? Example...
I learned a lot from the Professors lectures in this class.   Student checks disagree
When asked why..... says class was too early in the morning and slept through many of them.
Thus this response is not about the Professor's ability, but the timing of the class which would be missed without the followup question.
Surveys have to be designed with care to make sure the information you are receiving is related to what you are asking. Have seen many cases where interpretation of survey results made faulty assumptions about the implications of the answers.

------------------------------
Patricia Fox
Research/Consultant
Fox Statistical Consulting & Applied Research
------------------------------

Original Message:
Sent: 03-23-2017 00:28
From: Kevin Rees
Subject: question about survey questions

I posted this to the ASA Statistical Education section, but wanted to get a wider swath of responses.  My apologies if you have already seen this.

Everyone,
I had a question posed to me during a department meeting today that I was unsure how to answer, so I promised to reach out for some feedback.

Our school is redoing Student Evaluations of teachers and a committee has created a set of questions that have four possible responses.  For example:  "TEACHER provides feedback that helps me to understand how to improve."  with four responses:  Strongly agree, Agree more than disagree, disagree more than agree, and strongly disagree.

Here is my question to all of you - is there a statistical or survey analysis reason to use an odd (for this case five) number of responses to a question like this instead of four responses?  The feeling of our department was that odd (3, 5 or 7) number of responses was preferable to even (4, or 6) number of responses, but it is a gut feeling.

Does anyone have any information that might be helpful about this?

Thank you,
Kevin

In my view, this whole approach to satisfaction surveys is fundamentally flawed because of its use of the Likert scale. (Likert inventedt his approach to satisfaction surveys as part of his PhD in Psychology.) Consider a survey seeking to ascertain students' satisfaction with a course, and using Likert-style questioning. 

The problem is that with the Likert approach, students are being asked to indicate agreement or disagreement with a number of statements, rather than to rate the performance of the institution (via the instructor).  Also, it can lead to difficulties of interpretation.  Subjective judgments are built into the survey statements.  For example, a student may disagree strongly with I have received detailed comments on my work yet the comments may have been quite sufficient for the student’s needs.  A more egregious example is the overall statement Overall, I am satisfied with the quality of the course.  A student may strongly agree that he/she is “satisfied”.  Unfortunately, it is known from market research that there is no such thing as a satisfied customer [Kordupleski, R., Mastering Customer Value Management.  Pinnaflex Educational Resources, Inc.: Cincinnati, OH, 2003, pp130 et seq.], only that there are degrees of satisfaction, and that the degree of satisfaction relative to the competition is what really counts.  This approach to satisfaction surveys cannot address this issue.  See Kordupleski (op. cit.)  for a fine comparison of different approaches to posing survey questions.

------------------------------
Nicholas Fisher
Visiting Professor
University of Sydney
------------------------------

Original Message:
Sent: 03-23-2017 00:28
From: Kevin Rees
Subject: question about survey questions

I posted this to the ASA Statistical Education section, but wanted to get a wider swath of responses.  My apologies if you have already seen this.

Everyone,
I had a question posed to me during a department meeting today that I was unsure how to answer, so I promised to reach out for some feedback.

Our school is redoing Student Evaluations of teachers and a committee has created a set of questions that have four possible responses.  For example:  "TEACHER provides feedback that helps me to understand how to improve."  with four responses:  Strongly agree, Agree more than disagree, disagree more than agree, and strongly disagree.

Here is my question to all of you - is there a statistical or survey analysis reason to use an odd (for this case five) number of responses to a question like this instead of four responses?  The feeling of our department was that odd (3, 5 or 7) number of responses was preferable to even (4, or 6) number of responses, but it is a gut feeling.

Does anyone have any information that might be helpful about this?

Thank you,
Kevin

Original Message------

In my view, this whole approach to satisfaction surveys is fundamentally flawed because of its use of the Likert scale. (Likert inventedt his approach to satisfaction surveys as part of his PhD in Psychology.) Consider a survey seeking to ascertain students' satisfaction with a course, and using Likert-style questioning. 

The problem is that with the Likert approach, students are being asked to indicate agreement or disagreement with a number of statements, rather than to rate the performance of the institution (via the instructor).  Also, it can lead to difficulties of interpretation.  Subjective judgments are built into the survey statements.  For example, a student may disagree strongly with I have received detailed comments on my work yet the comments may have been quite sufficient for the student’s needs.  A more egregious example is the overall statement Overall, I am satisfied with the quality of the course.  A student may strongly agree that he/she is “satisfied”.  Unfortunately, it is known from market research that there is no such thing as a satisfied customer [Kordupleski, R., Mastering Customer Value Management.  Pinnaflex Educational Resources, Inc.: Cincinnati, OH, 2003, pp130 et seq.], only that there are degrees of satisfaction, and that the degree of satisfaction relative to the competition is what really counts.  This approach to satisfaction surveys cannot address this issue.  See Kordupleski (op. cit.)  for a fine comparison of different approaches to posing survey questions.

------------------------------
Nicholas Fisher
Visiting Professor
University of Sydney
------------------------------