Discussion: View Thread

Hello, I received the following request and would like to find somebody familiar with survey sampling to assist. Please respond directly to me at mhayat@gsu.edu. Thank you, Matt


"I need a recommendation for someone to assist me on my project involving a multilevel design where we enrolled, using convenience sampling, 1150 male freshmen from 30 different colleges.

I am looking for help with assessing the representativeness of the sample and possibly using post-stratification weighting. "

This is an example of the (in)famous situation of "if you bring the data you have already collected to a statistician, they would only be able to tell what the study died from." No amount of post-stratification fiddling will overcome the lack of initial randomization / random selection. You would be able to reduce the method biases, of course, but you will never know if you removed 20% of the bias or 80% of the bias (and most likely there will be variables in which you removed most of the bias from the descriptive summaries, and there will be variables in which you remove little if the existing post-stratification variables aren't explaining much). If you are going to fit regression models to these data, with the variables that you intend to post-stratify on as predictors, then you gain little to nothing from post-stratification; while differential weights will give you a fair amount of headache in your multilevel modeling down the road, with weight scaling and all that.

------------------------------
Stanislav Kolenikov
Principal Survey Scientist
Abt SRBI
Education Officer, Survey Research Methods Section
------------------------------

Hello Stanislav, I enjoyed learning your quip: "if you bring the data you have already collected to a statistician, they would only be able to tell what the study died from." 

This is a situation that consultants meet many times. The client or potential client briefly describes (in an email) what they think they have done and tells what they think needs doing. Until a statistician talks with them, we don't know what we have in hand. And, the value of the data depends on the situation. If th data are from a field that is not well developed (and little is known), then even a convenience sample (if that is what it is) may be informative. Of course, the narrative in the report has to tell the story and limitations. It will be interesting to see what was actually done and what can be made of it. 

I read in Stanislav's message: "be careful!" Agreed!

Best wishes,

Nayak

------------------------------
Nayak Polissar
Principal Statistician
The Mountain-Whisper-Light Statistics
------------------------------

Original Message:
Sent: 10-09-2015 18:30
From: Stanislav Kolenikov
Subject: Survey sampling expertise needed

This is an example of the (in)famous situation of "if you bring the data you have already collected to a statistician, they would only be able to tell what the study died from." No amount of post-stratification fiddling will overcome the lack of initial randomization / random selection. You would be able to reduce the method biases, of course, but you will never know if you removed 20% of the bias or 80% of the bias (and most likely there will be variables in which you removed most of the bias from the descriptive summaries, and there will be variables in which you remove little if the existing post-stratification variables aren't explaining much). If you are going to fit regression models to these data, with the variables that you intend to post-stratify on as predictors, then you gain little to nothing from post-stratification; while differential weights will give you a fair amount of headache in your multilevel modeling down the road, with weight scaling and all that.

------------------------------
Stanislav Kolenikov
Principal Survey Scientist
Abt SRBI
Education Officer, Survey Research Methods Section
------------------------------