Hello Shelly-Ann,
I cannot agree more with David, for his nice comments on FDA's PRO guidance and Dr. Ron Hay's work.
For scale development, based on my experiences at work I found this small booklet by Dr. Robert F.DeVellis is very useful:
DeVellis RF. Scale development: theory and applications, 3rd edn. Thousand Oaks, CA: Sage Publications 2012.
For scale validating, historically, there had been many types of validity, and here is a good summary by Dr. Bruno D. Zumbo:
Zumbo BD. Validity: foundational issues and statistical methodology. In Rao CR and Sinharay S. (Ed.). Psychometrics, Handbook of Statistics [26]. Amsterdam, The Netherlands: Elsevier 2007; 45-79.
And here is another article on this topic:
Cook DA, Beckman TJ. Current Concepts in Validity and Reliability for Psychometric Instruments: Theory and Application. Am J Med. 2006 Feb;119(2):166.e7-16. Review.
As to your sample size question, I think there is no easy/consistent answer. Based on different authors (e.g., DeVellis RF, Embertson SE & Reise SP (Item Response Theory for Psychologists, New York, NY. Psychology Press 2000), etc), few hundreds should suffice.
But I think that the most challenging thing in your plan is, you want to develop a 'tool that can track clinically relevant changes' and 'for use in future comparative studies'. As you may know, it is very difficult to define a 'minimum clinically important difference (MCID), and, in longitudinal studies that use scales/instruments ('tools' in your word), there is another big concern: you need to show 'longitudinal measurement invariance' (many articles on this topic, such as, Brown TA. Confirmatory Factor Analysis for Applied Research. New York. NY. The Guilford Press 2006; 252-266).
Sincerely yours,
Chengwu Yang （杨成武）
______________________
Chengwu Yang, MD, MS, PhD
Assistant Professor of Biostatistics
Department of Public Health Sciences
College of Medicine, The Pennsylvania State University
A210, ASB 3400H, 600 Centerview Drive, Hershey, PA 17033
Email:
yangc@psu.edu; Phone: 717-531-3016; Fax: 717-531-0146
http://profiles.psu.edu/profiles/ProfileDetails.aspx?From=SE&Person=244 -------------------------------------------
Original Message:
Sent: 09-26-2012 15:44
From: David Reasner
Subject: Developing and Validating a Scoring Tool
Dear Shelly-Ann,
It would be worth looking at the FDA's PRO guidance (Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims). While the FDA emphasizes content validity (i.e., the patient perspective) rather than evidence based on relations to other variables, the release of the guidance triggered both an academic and industry response that is still ongoing. In addition to several dedicated conferences, ISPOR and other organizations have responded to the guidance with white papers, conference tracks, etc. Psychometrics is separate from biometrics and you will find a long-standing literature on the topics that you mention. It may be worth working with a consultancy (e.g., RTI) where researchers deal routinely with these topics and FDA expectations. I've also found Ron Hays' work very helpful and will paste his link below.
Best regards,
David
http://www.chime.ucla.edu/directory/Hays.htm
-------------------------------------------
David Reasner
Albemarle Scientific Consulting LLC
www.AlbemarleScientific.com
-------------------------------------------