Hello Shelly-Ann,
First of all, thank David for his wonderful comments on FDA's PRO guidance and Dr. Ron Hay's work, and I also learned a lot from these two. Moreover, I would suggest you also have a look at Dr. Robert Gibbons' (http://health.uchicago.edu/People/Gibbons-Robert) work, which includes a very impressive project of the CAT-DI (Depression Inventory) that uses item response theory (IRT) and computerized adaptive testing (CAT) to develop a depression test with less burden and higher precision.
For scale development, based on my experiences of working with clinicians, I found this small booklet by Dr. Robert F.DeVellis is very useful:
DeVellis RF. Scale development: theory and applications, 3rd edn. Thousand Oaks, CA: Sage Publications 2012.
For scale validating, historically, there had been many types of validity, and here is a good summary by Dr. Bruno D. Zumbo:
Zumbo BD. Validity: foundational issues and statistical methodology. In Rao CR and Sinharay S. (Ed.). Psychometrics, Handbook of Statistics [26]. Amsterdam, The Netherlands: Elsevier 2007; 45-79.
And here is another article on this topic:
Cook DA, Beckman TJ. Current Concepts in Validity and Reliability for Psychometric Instruments: Theory and Application. Am J Med. 2006 Feb;119(2):166.e7-16. Review.
As to your sample size question, I think there is no easy/consistent answer. Based on different authors (e.g., DeVellis RF, Embertson SE & Reise SP (Item Response Theory for Psychologists, New York, NY. Psychology Press 2000), etc), few hundreds should suffice.
But I think that the most challenging thing in your plan is, you want to develop a 'tool that can track clinically relevant changes' and 'for use in future comparative studies'. As you may know, it is very difficult to define a 'minimum clinically important difference' (MCID), and, in longitudinal studies that use scales/instruments ('tools' in your word), there is another big concern: you need to show 'longitudinal measurement invariance' (many articles on this topic, such as, Brown TA. Confirmatory Factor Analysis for Applied Research. New York. NY. The Guilford Press 2006; 252-266).
I hope this helps, thanks.
Sincerely yours,
Chengwu Yang （杨成武）
______________________
Chengwu Yang, MD, MS, PhD
Assistant Professor of Biostatistics
Department of Public Health Sciences
College of Medicine, The Pennsylvania State University
A210, ASB 3400H, 600 Centerview Drive, Hershey, PA 17033
Email:
yangc@psu.edu; Phone: 717-531-3016; Fax: 717-531-0146
http://profiles.psu.edu/profiles/ProfileDetails.aspx?From=SE&Person=244 -------------------------------------------