Discussion: View Thread

I am helping a friend with her dissertation analysis and would like advice on how to analyze survey data collected using a 5 or 7 category Likert scale.  As best I can tell from a quick literature search, these outcomes are often analyzed as if they were continuous normal random variables, but that doesn't seem right to me, as it would imply specific multiplicative effects between the responses (e.g., "a lot" is 2 times "neutral"). Assuming a multinomial distribution makes more sense.

This is not at all my area of consulting expertise, so I am hoping that someone can point me to a good reference to get me started. Thank you.
-------------------------------------------
Linda Pickle
StatNet Consulting, LLC
-------------------------------------------

The best reference is Agresti,  Analysis of Ordinal Categorical Data.  Wiley,  NY. 2010.    (make sure you get this edition - there is an old edition of the book from the 80's but this area has seen a lot of advancement since then). 



-------------------------------------------
Roy Tamura
Associate Professor
University of South Florida
-------------------------------------------






Original Message:
Sent: 01-10-2014 10:27
From: Linda Pickle
Subject: analyzing Likert scale data

I am helping a friend with her dissertation analysis and would like advice on how to analyze survey data collected using a 5 or 7 category Likert scale.  As best I can tell from a quick literature search, these outcomes are often analyzed as if they were continuous normal random variables, but that doesn't seem right to me, as it would imply specific multiplicative effects between the responses (e.g., "a lot" is 2 times "neutral"). Assuming a multinomial distribution makes more sense.

This is not at all my area of consulting expertise, so I am hoping that someone can point me to a good reference to get me started. Thank you.
-------------------------------------------
Linda Pickle
StatNet Consulting, LLC
-------------------------------------------

If you haven't already done so, check the pscyhometric literature.

One option is a Rasch Analysis

these  have some citations

http://web.it.nctu.edu.tw/~qtqm/upcomingpapers/2010V7N1/2010V7N1_F2.pdf

http://japha.org/article.aspx?articleid=1043469

-------------------------------------------
Chris Barker, Ph.D.
Consultant and
Adjunct Associate Professor of Biostatistics
www,barkerstats.com

---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
-------------------------------------------






Original Message:
Sent: 01-10-2014 10:31
From: Linda Pickle
Subject: analyzing Likert scale data

I am helping a friend with her dissertation analysis and would like advice on how to analyze survey data collected using a 5 or 7 category Likert scale.  As best I can tell from a quick literature search, these outcomes are often analyzed as if they were continuous normal random variables, but that doesn't seem right to me, as it would imply specific multiplicative effects between the responses (e.g., "a lot" is 2 times "neutral"). Assuming a multinomial distribution makes more sense.

This is not at all my area of consulting expertise, so I am hoping that someone can point me to a good reference to get me started. Thank you.
-------------------------------------------
Linda Pickle
StatNet Consulting, LLC
-------------------------------------------

I am also fairly new to the Rasch analysis but had to do some extensive learning for one of my clients a month ago. For Likert scale responses there are several models that can be used depending on the specific questions. I found the following book to be a good solid introduction to the topic:

Ostini, Polytomous Item Response Theory Models, SAGE, 2006

-------------------------------------------
Moni Neradilek
Statistical Consultant
The Mountain-Whisper Light Statistics
-------------------------------------------






Original Message:
Sent: 01-10-2014 11:14
From: Chris Barker
Subject: analyzing Likert scale data

If you haven't already done so, check the pscyhometric literature.

One option is a Rasch Analysis

these  have some citations

http://web.it.nctu.edu.tw/~qtqm/upcomingpapers/2010V7N1/2010V7N1_F2.pdf

http://japha.org/article.aspx?articleid=1043469

-------------------------------------------
Chris Barker, Ph.D.
Consultant and
Adjunct Associate Professor of Biostatistics
www,barkerstats.com

---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
-------------------------------------------






Original Message:
Sent: 01-10-2014 10:31
From: Linda Pickle
Subject: analyzing Likert scale data

I am helping a friend with her dissertation analysis and would like advice on how to analyze survey data collected using a 5 or 7 category Likert scale.  As best I can tell from a quick literature search, these outcomes are often analyzed as if they were continuous normal random variables, but that doesn't seem right to me, as it would imply specific multiplicative effects between the responses (e.g., "a lot" is 2 times "neutral"). Assuming a multinomial distribution makes more sense.

This is not at all my area of consulting expertise, so I am hoping that someone can point me to a good reference to get me started. Thank you.
-------------------------------------------
Linda Pickle
StatNet Consulting, LLC
-------------------------------------------

Hi Linda,

 

In line with Chris' helpful recommendations, you might find the following article to also be of interest:

 

Rhemtulla M, Brosseau-Liard PÉ, Savalei V. When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychol Methods 2012;17:354-73.

 

A simulation study compared the performance of robust normal theory maximum likelihood (ML) and robust categorical least squares (cat-LS) methodology for estimating confirmatory factor analysis models with ordinal variables. Data were generated from 2 models with 2-7 categories, 4 sample sizes, 2 latent distributions, and 5 patterns of category thresholds. Results revealed that factor loadings and robust standard errors were generally most accurately estimated using cat-LS, especially with fewer than 5 categories; however, factor correlations and model fit were assessed equally well with ML. Cat-LS was found to be more sensitive to sample size and to violations of the assumption of normality of the underlying continuous variables. Normal theory ML was found to be more sensitive to asymmetric category thresholds and was especially biased when estimating large factor loadings. Accordingly, we recommend cat-LS for data sets containing variables with fewer than 5 categories and ML when there are 5 or more categories, sample size is small, and category thresholds are approximately symmetric. With 6-7 categories, results were similar across methods for many conditions; in these cases, either method is acceptable. (PsycINFO Database Record (c) 2013 APA, all rights reserved)(journal abstract)

 

With best wishes,

Tor Neilands

UCSF                                                                                 

 

Tor Neilands, PhD

Professor of Medicine

Center for AIDS Prevention Studies (CAPS)

University of California, San Francisco

50 Beale Street, Suite 1300

San Francisco, CA 94105

 

Phone: (415) 597-9236

Fax: (415) 597-9213

e-mail: tor@ucsf.edu

Organization page: http://www.caps.ucsf.edu

Bio page: http://caps.ucsf.edu/personnel/tneilands/

-------------------------------------------
Torsten Neilands
Professor
UCSF Center for AIDS Prevention Studies
-------------------------------------------






Original Message:
Sent: 01-10-2014 13:51
From: Chris Barker
Subject: analyzing Likert scale data

If you haven't already done so, check the pscyhometric literature.

One option is a Rasch Analysis

these  have some citations

http://web.it.nctu.edu.tw/~qtqm/upcomingpapers/2010V7N1/2010V7N1_F2.pdf

http://japha.org/article.aspx?articleid=1043469

-------------------------------------------
Chris Barker, Ph.D.
Consultant and
Adjunct Associate Professor of Biostatistics
www,barkerstats.com

---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
-------------------------------------------






Original Message:
Sent: 01-10-2014 10:31
From: Linda Pickle
Subject: analyzing Likert scale data

I am helping a friend with her dissertation analysis and would like advice on how to analyze survey data collected using a 5 or 7 category Likert scale.  As best I can tell from a quick literature search, these outcomes are often analyzed as if they were continuous normal random variables, but that doesn't seem right to me, as it would imply specific multiplicative effects between the responses (e.g., "a lot" is 2 times "neutral"). Assuming a multinomial distribution makes more sense.

This is not at all my area of consulting expertise, so I am hoping that someone can point me to a good reference to get me started. Thank you.
-------------------------------------------
Linda Pickle
StatNet Consulting, LLC
-------------------------------------------

I generally use a Bayesian scale-usage model, in which there is assumed to be a continuous latent variable underlying each Likert-scale response, and a set of cutpoints that turn the continuous value into the discrete Likert-scale values. The cutpoints, and possible heterogeneity in same, are inferred from the data. Here are some papers that describe such scale-usage models:

Rossi, P. E., Z. Gilula, G. M. Allenby (2001), "Overcoming scale usage heterogeneity: a Bayesian approach," J. of the American Statistical Association 96 (453), pp. 20--31.

C. Hans, G. M. Allenby, P. F. Craigmile, J. Lee (2012), "Covariance decomposition for for accurate computation in Bayesian scale-usage models," J. of Computational and Graphical Statistics 21 (2), pp. 538--557.

The bayesm package (in R) has an rscaleUsage function that does this kind of analysis.

-------------------------------------------
Kevin Van Horn
-------------------------------------------






Original Message:
Sent: 01-10-2014 11:29
From: Linda Pickle
Subject: analyzing Likert scale data

I am helping a friend with her dissertation analysis and would like advice on how to analyze survey data collected using a 5 or 7 category Likert scale.  As best I can tell from a quick literature search, these outcomes are often analyzed as if they were continuous normal random variables, but that doesn't seem right to me, as it would imply specific multiplicative effects between the responses (e.g., "a lot" is 2 times "neutral"). Assuming a multinomial distribution makes more sense.

This is not at all my area of consulting expertise, so I am hoping that someone can point me to a good reference to get me started. Thank you.
-------------------------------------------
Linda Pickle
StatNet Consulting, LLC
-------------------------------------------

Likert used "agreement" as the construct underlying the response categories.
It sounds as if your friend has some from of "extent"  as the underlying construct.
What was the actual response scale used?

Is the construct that is being operationalized with these variables intrinsically continuous?

Are these items that are parts of a summative scale?
Have the psychometric properties of any scales been established in previous research?

Are the stems written for this study or are they from an established instrument?

How are the variables being used in any modeling? IVs? DVs? Something to control for?


Depending what the hypotheses being examined are you might want to use Categorical Regression (CATREG) or Categorical Principal Components Analysis (CATPCA) and test whether it meaningfully changes the fit if you assume ordinal vs interval level of measurement.



-------------------------------------------
Arthur Kendall
Social Research Consultants
-------------------------------------------






Original Message:
Sent: 01-10-2014 13:26
From: Linda Pickle
Subject: analyzing Likert scale data

I am helping a friend with her dissertation analysis and would like advice on how to analyze survey data collected using a 5 or 7 category Likert scale.  As best I can tell from a quick literature search, these outcomes are often analyzed as if they were continuous normal random variables, but that doesn't seem right to me, as it would imply specific multiplicative effects between the responses (e.g., "a lot" is 2 times "neutral"). Assuming a multinomial distribution makes more sense.

This is not at all my area of consulting expertise, so I am hoping that someone can point me to a good reference to get me started. Thank you.
-------------------------------------------
Linda Pickle
StatNet Consulting, LLC
-------------------------------------------

There appears to be two schools of thought on this subject - one that treats the responses as ordinal data and the other treats the responses as ratio data.  Literature articles are available to support both.

Being in healthcare, my experience with some of the ratio crowd is the patient survey companies.  They take patients' value judgments ("very poor" to "very good") and ultimately report average scores to four significant digits (two decimal places), along with standard deviations - implying a precision and accuracy that does not exist.  This, of course, assumes that the ordinal responses possess the property of distance, which they do not.

My method of analysis of these data is (1) a simple histogram of each question's response levels frequencies, and (2) if the survey is repeated, SPC p-charts of the percent "very good" and "very poor" for each question (tracking each extreme).

-------------------------------------------
Wayne Fischer
Statistician
University of Texas Medical Branch
-------------------------------------------






Original Message:
Sent: 01-10-2014 13:51
From: Linda Pickle
Subject: analyzing Likert scale data

I am helping a friend with her dissertation analysis and would like advice on how to analyze survey data collected using a 5 or 7 category Likert scale.  As best I can tell from a quick literature search, these outcomes are often analyzed as if they were continuous normal random variables, but that doesn't seem right to me, as it would imply specific multiplicative effects between the responses (e.g., "a lot" is 2 times "neutral"). Assuming a multinomial distribution makes more sense.

This is not at all my area of consulting expertise, so I am hoping that someone can point me to a good reference to get me started. Thank you.
-------------------------------------------
Linda Pickle
StatNet Consulting, LLC
-------------------------------------------

I've been offline for 10 days but wanted to thank those who responded to my question earlier. You have given me several good references to get started.

-------------------------------------------
Linda Pickle
StatNet Consulting, LLC
-------------------------------------------






Original Message:
Sent: 01-22-2014 09:11
From: Wayne Fischer
Subject: analyzing Likert scale data

There appears to be two schools of thought on this subject - one that treats the responses as ordinal data and the other treats the responses as ratio data.  Literature articles are available to support both.

Being in healthcare, my experience with some of the ratio crowd is the patient survey companies.  They take patients' value judgments ("very poor" to "very good") and ultimately report average scores to four significant digits (two decimal places), along with standard deviations - implying a precision and accuracy that does not exist.  This, of course, assumes that the ordinal responses possess the property of distance, which they do not.

My method of analysis of these data is (1) a simple histogram of each question's response levels frequencies, and (2) if the survey is repeated, SPC p-charts of the percent "very good" and "very poor" for each question (tracking each extreme).

-------------------------------------------
Wayne Fischer
Statistician
University of Texas Medical Branch
-------------------------------------------






Original Message:
Sent: 01-10-2014 13:51
From: Linda Pickle
Subject: analyzing Likert scale data

I am helping a friend with her dissertation analysis and would like advice on how to analyze survey data collected using a 5 or 7 category Likert scale.  As best I can tell from a quick literature search, these outcomes are often analyzed as if they were continuous normal random variables, but that doesn't seem right to me, as it would imply specific multiplicative effects between the responses (e.g., "a lot" is 2 times "neutral"). Assuming a multinomial distribution makes more sense.

This is not at all my area of consulting expertise, so I am hoping that someone can point me to a good reference to get me started. Thank you.
-------------------------------------------
Linda Pickle
StatNet Consulting, LLC
-------------------------------------------

Hi all, 
For ordinal data, I try to use the format below.  This example is for (what I call) bidirectional ordinal data, meaning that it rates from positive to negative extremes, with a neutral (e.g., "No Change") position in the middle.  For unidirectional ordinal data, the graph would just be the top half of this graph.

The following stacked bar graph has many advantage:

• The most intense ratings are stacked nearest the axis, with decreasing intensity away from the axis.  Hence the top of any bar corresponds to the % of subjects with at least that intense of a response.  
• E.G.: at a glance: For the EOS group A observation, about 10% scored "Very Much Improved" (top of the dark green), while 30% scored at least Much Improvement (top of the light green), and 30% any improvement (NB: no one scored "Minimally Improved")
• The above axis item correspond to improvement, and below axis items correspond to worsening
• E.G.: In the Day 28 Group A observation, few scored Very Much Improvement or Worsening, and about equal numbers scored improvement as worsening. 
• Ratings of "No Change" are implied, but not shown.
• E.G.:  For Day 28 Group A, about 30% improved, 40% worsened, leaving 30% with no change.

 

Generic Description (for Statistical Analysis Plan)

For birectional ordinal data (such as CGI-I), the same concept as for unidirectional ordinal data is used but with desirable scores (e.g., improvement) plotted above the horizontal axis, and undesirable scores (e.g. worsening) plotted below the horizontal axis. For either category, the most extreme ratings will be closest to the horizontal axis, and assigned the most intense colors. This will allow an immediate visual impression of the relative proportions of patients who improved versus those who worsened, and by how much.

http://dennis-sweitzer.blogspot.com/2014/01/displaying-ordinal-data-bidirectional.html
-------------------------------------------
Dennis Sweitzer
Principal Biostatistician
Medidata Solutions
-------------------------------------------






Original Message:
Sent: 01-22-2014 12:57
From: Linda Pickle
Subject: analyzing Likert scale data

I've been offline for 10 days but wanted to thank those who responded to my question earlier. You have given me several good references to get started.

-------------------------------------------
Linda Pickle
StatNet Consulting, LLC
-------------------------------------------






Original Message:
Sent: 01-22-2014 09:11
From: Wayne Fischer
Subject: analyzing Likert scale data

There appears to be two schools of thought on this subject - one that treats the responses as ordinal data and the other treats the responses as ratio data.  Literature articles are available to support both.

Being in healthcare, my experience with some of the ratio crowd is the patient survey companies.  They take patients' value judgments ("very poor" to "very good") and ultimately report average scores to four significant digits (two decimal places), along with standard deviations - implying a precision and accuracy that does not exist.  This, of course, assumes that the ordinal responses possess the property of distance, which they do not.

My method of analysis of these data is (1) a simple histogram of each question's response levels frequencies, and (2) if the survey is repeated, SPC p-charts of the percent "very good" and "very poor" for each question (tracking each extreme).

-------------------------------------------
Wayne Fischer
Statistician
University of Texas Medical Branch
-------------------------------------------






Original Message:
Sent: 01-10-2014 13:51
From: Linda Pickle
Subject: analyzing Likert scale data

I am helping a friend with her dissertation analysis and would like advice on how to analyze survey data collected using a 5 or 7 category Likert scale.  As best I can tell from a quick literature search, these outcomes are often analyzed as if they were continuous normal random variables, but that doesn't seem right to me, as it would imply specific multiplicative effects between the responses (e.g., "a lot" is 2 times "neutral"). Assuming a multinomial distribution makes more sense.

This is not at all my area of consulting expertise, so I am hoping that someone can point me to a good reference to get me started. Thank you.
-------------------------------------------
Linda Pickle
StatNet Consulting, LLC
-------------------------------------------