Discussion: View Thread

  • 1.  Recommendations on how to approach analysis of Likert scale responses

    Posted 10-13-2022 19:08
    Hi Statistical Consulting Community! 

    I have been working with a researcher in occupational therapy on analysis of Likert scale responses (5-point scale) from pre vs. post surveys. The questions the researcher is interested in answering are whether the level of agreement changed for survey respondents from the pre to post survey responses on a selection of questions. The analysis I have been doing is a Wilcoxon signed-rank test to compare differences in the distributions and reporting the median, IQR, and mean values (responses were changed to 5-point scale) for the pre and post questions for reference. The reviewers of the occupational therapy journal have not agreed with the analysis methods and provided a citation that ordinal data should be analyzed with metric ordered-probit models. The citation they provided is here: Liddell, T., & Kruschke, J. K. (2017, November 6). Analyzing ordinal data with metric models: What could possibly go wrong?. https://doi.org/10.31219/osf.io/9h3et

    Would anyone have any advice on how to respond to a journal reviewer that is recommending a more complex modeling method than is needed for the research question? Any specific articles that I can reference for using a non-parametric test for the analysis? Or any recommendations on other analysis methods to use for simple pre vs. post comparisons of Likert scale responses? 

    Thanks in advance for any advice and suggestions!

    ------------------------------
    Charlotte Bolch
    Midwestern University
    ------------------------------


  • 2.  RE: Recommendations on how to approach analysis of Likert scale responses

    Posted 10-13-2022 19:33
    The ordered-probit model has asymptotic normality assumptions. The rate of convergence in distribution can be rather slow for this family of tests. Unless your sample size is fairly large, I would agree with your choice to use non-parametric methods.

    ------------------------------
    Jimmy Efird
    Chief Statistician
    ------------------------------

    Original Message


  • 3.  RE: Recommendations on how to approach analysis of Likert scale responses

    Posted 10-13-2022 21:22
    Great question. Your situation is a sort of "glass is half full, half empty".
    The glass is half full, the reviewer cares a lot. The glass is half empty, the reviewer cares a lot.

    Perhaps you are in the situation where the reviewer agrees that the paper is acceptable though with a different analysis.  There is no obvious (to me) harm in doing the reviewer requested analysis. I happen to have done a lot of work with PRO's (patient reported outcomes) and with Likert scale. Psychometricians have very specific ideas about analysis and more specifically for the connection between actual human behavior and the statistical models of that human behavior, and broadly divide their philosophy into 'classical test theory" and "item response theory".  Caveat Emptor, You do not need not take a deep dive into psychometric theory for your specific paper.

    To more directly answer your question First it would be helpful to have a definition of the metric ordered probit model. I may be wrong, -I think Jackman has written thoughtfully about these models. https://web.stanford.edu/class/polisci203/ordered.pdf
    your citation to Liddel Kruscke also gives a not-too-technical definition of metric ordered probit models.
     I would be (prepared to be ) surprised if your non-parametric method and the reviewers choice give markedly different answers.
    It looks like (after my -lite-  reading) that the metric model has a few more "bells and whistles" that psychometricians view with fondness.

    If one analysis (subject to caveats that follow) is all that's needed, to get the publication, then I'd recommend doing that -again with caveats below.

    One option (you have  several ) is to agree to do the reviewers analysis as an appendix and present yours as the preferred analysis, given "all the very well known limitations" of the metric ordered model. This is the "do it both ways" approach and there is no guarantee the results are the same nor can I guarantee the editor will accept it. . My caveats and my personal "line in the sand" as a reviewer.  I generally require that the following are true. First  I  require that authors state their analysis was "pre planned" and my second line in the sand is that the authors gave appropriate "consideration to multiple comparisons". 

    Entirely separate , I would personally be interested to find out if the two approaches yield similar interpretations.





    ------------------------------
    Chris Barker, Ph.D.
    2022 Statistical Consulting Section
    Chair-elect
    Consultant and
    Adjunct Associate Professor of Biostatistics
    www.barkerstats.com


    ---
    "In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
    -Steve Lacy
    ------------------------------

    Original Message


  • 4.  RE: Recommendations on how to approach analysis of Likert scale responses

    Posted 10-14-2022 09:20
    Liddell and Kruschke's (2017) paper is interesting and worth reading carefully. It could be quite instructive to run the analysis suggested by the reviewer and compare the results to what you obtained with Wilcoxon signed-rank test. I would also consider using the McNemar-Bowker test or a more sophisticated alternative to that based on log-linear models described in the sources below. Think about which analysis allows you to extract the most interpretable and directly relevant results from the data and communicate the answers to your research question while providing appropriate effect size estimates and associated assessment of uncertainty. My intuition here is that the log-linear modeling or ordered-probit models can offer more supplementary insight than the Wilcoxon signed-rank test. 

    Meiser, T., von Eye, A., & Spiel, C. (1997). Loglinear symmetry and quasi-symmetry models for the analysis of change. Biometrical Journal, 39(3), 351-368. https://doi.org/10.1002/bimj.4710390309

    Von Eye, A., & Mun, E. Y. (2013). Log-linear modeling: Concepts, interpretation, and application. John Wiley & Sons, Inc. 



    ------------------------------
    Steven J. Pierce
    Associate Director
    Center for Statistical Training and Consulting
    Michigan State University
    ------------------------------

    Original Message


  • 5.  RE: Recommendations on how to approach analysis of Likert scale responses

    Posted 10-18-2022 13:06
    A quick but informative read is given in French and Shotwell, "Regression Models for Ordinal Outcomes", JAMA, August 2022. It provides some references not in Liddell and Kruschke (2017) , including the classic McCullagh JRSSB (1980). 
    David

    ------------------------------
    David Bristol
    Statistical Consulting Services, Inc.
    ------------------------------

    Original Message