ASA Connect

 View Only

  • 1.  Number of raters

    Posted 11-04-2016 12:17

    Hello everyone,

    One of my clients would like to investigate the inter-rater reliability for an instrument tool to evaluate a clinical outcome. The outcome would be classified in likert-type scale.  I need to help him to determine how many raters will be invited to participate this study.  I did some research but not find very useful resources.  Please share your thoughts.

    I attached the links I found for this topic:

    http://stats.stackexchange.com/questions/41016/determining-the-number-of-raters-for-inter-rater-reliability

     

     

    http://onlinelibrary.wiley.com/doi/10.1002/sim.2294/epdf

    This paper suggested a formula to calculate the number of raters given total assessment number fixed.  However, for us, the number of patients is fixed.  We would like to find out how many raters we should use to make this assessment. 

    Thank you for your help!

    Jessie

    ------------------------------
    Jun Liu
    Sr. Biostatistician
    ------------------------------


  • 2.  RE: Number of raters

    Posted 11-07-2016 12:49

    PASS versions 12 and later have a sample-size-and-power module for intraclass correlation. It can calculate number of observations (i.e., raters) per subject needed to achieve specified power when the number of subjects is fixed.

    PASS version 14 has a second sample-size module for intraclass correlation that bases calculations on confidence intervals instead of power.  

    Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.



    Original Message


  • 3.  RE: Number of raters

    Posted 11-08-2016 08:04

    Thank you for your help!  I don't have PASS.  Can I know what reference paper they used for that calculation?

    ------------------------------
    Jun Liu
    Sr. Biostatistician

    Original Message


  • 4.  RE: Number of raters

    Posted 11-10-2016 12:55

    PASS provides the following references:

    Walter, S.D., Eliasziw, M., and Donner, A. 1998. 'Sample Size and Optimal Designs For Reliability Studies.'
    Statistics in Medicine, 17, 101-110.
    Winer, B.J. 1991. Statistical Principles in Experimental Design (Third Edition). McGraw-Hill. New York, NY.

     

    Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.



    Original Message


  • 5.  RE: Number of raters

    Posted 11-11-2016 12:52
    After all is said and done, here are the the two things that stand out about the 2016 presidential election. (1) The best polling models in the world can't account for the honesty of the response, especially when one of the choices is perceived as controversial or "unpopular".  Of course, there are probabilistic ways to control for this problem but then the polling is too time consuming, too laborious, and impractical. (2) The perception of Hillary's lead (within days of the election captured by the now infamous 71% probability of winning) was based on the result of a national poll. US presidential elections are not based on the popular national vote. State by state polling results would have been more indicative of the final electoral college outcome.

    Frank Soler



    Original Message