ASA Connect

 View Only
Back to eGroups
Expand all | Collapse all

Survey re: Issues of Reproducibility in Mixed-Methods Research

  • 1.  Survey re: Issues of Reproducibility in Mixed-Methods Research

    Posted 04-10-2018 12:44
    Dear Colleagues,
    I'm posting this request on behalf of researchers at Northwestern.  Please see below.  
    Thanks.
    Ron

    Dear Dr. Wasserstein,<u5:p></u5:p>

     

    We are contacting you in your capacity as Executive Director of the American Statistical Association.

     

    There is a great deal of conversation in the field at the moment about replicability and reproducibility of research findings. As graduate students who work with mixed-methods research designs (i.e., research incorporating both qualitative and quantitative analysis), we have found that there has been little discussion about whether or how issues of reproducibility apply in mixed-methods research.

     

    We believe that special considerations apply to this type of research and we want to make sure that mixed-methods researchers' voices are heard on these issues. We are circulating a survey to our professional peers to collate opinions on these issues and best practices for reproducibility in mixed-methods research. <u5:p></u5:p>

     

    We are hoping for feedback from individuals with experience in this area. We would appreciate your input and collaboration on this matter. You can access the survey here: https://weinberg.co1.qualtrics.com/jfe/form/SV_1XDp511VI1ma2S9

     

    The survey will be active until April 27, 2018. It may take anywhere from 5-30 minutes of your time, depending on how much you choose to contribute. We also welcome you to contact us directly to discuss this project or issues of reproducibility more generally. <u5:p></u5:p>

     

    Please share this message and the survey link with any colleagues who work in this area and would be interested in sharing their opinions on these important issues. <u5:p></u5:p>

    <u5:p> </u5:p>

    Many thanks,<u5:p></u5:p>

    <u5:p></u5:p>

    Hollen Reischer and Raffles Cowan
    hollen@u.northwestern.edu

    raffles.cowan@u.northwestern.edu

    Foley Center for the Study of Lives<u5:p></u5:p>

    Northwestern University


    <u5:p></u5:p>



    ------------------------------
    Ron Wasserstein
    Executive Director
    The American Statistical Association
    Promoting the Practice and Profession of Statistics
    732 N. Washington St.
    Alexandria, VA 22314
    703-684-1221 x1860
    ------------------------------


  • 2.  RE: Survey re: Issues of Reproducibility in Mixed-Methods Research

    Posted 04-11-2018 16:51
    When I first heard about mixed methods about 20 years ago, I thought it was about mixed models. Then I learned it emerged as a blending of qualitative and quantitative research methods. The so-called qualitative methods interested me because my dissertation was on MDS and I thought qualitative methods were similar to nonmetric models (MDS, MONANOVA, conjoint measurement, etc.). I was so wrong. Qualitative methods emerged as a reaction by a few social scientists against statistics. They felt that assigning a number to something obscured underlying meaning, not realizing that an awful lot of research in the social sciences has not involved assigning numbers to observations. Luce and Tukey's conjoint measurement was, if anything, a reaction against the uncritical assignment of numbers on such things as Likert scales.

    Before long, the qualitative research movement took over whole departments, including UC Irvine, which had one of the top nonmetric research faculties in the country. Qualitative researchers back then promoted methodologies like sticking Post-Its on a wall and moving them around to "discover" clusters. And, of course, they loved unstructured interviews and personal impressions. Unable to get their articles published in mainline journals, they formed their own journals and editorial boards. Unlike Polanyi, whose book "Personal Knowledge" steered clear of relativism while acknowledging the subjective aspects of scientific inquiry, these qualitative researchers believed understanding required a rejection of measurement itself. And when they fused their methods into a "new" field called mixed methods, they hoped to recruit social scientists who were more familiar with experimental design and statistics.

    So when I took this questionnaire and discovered that these two researchers wanted to see whether reproducibility is relevant to mixed-methods, I thought they were involved in an oxymoron. The whole point of mixed methods, it seems to me, is to worry less about reproducibility than about "meaningful" storytelling. 

    I'm not a positivist.  I believe unstructured interviews and other impressionistic methods can be valuable for generating social science research ideas. But I think it is misleading to draw an equivalence between current controversies regarding scientific fraud, reproducibility, power, p-hacking, etc. and problems in the more amorphous area of mixed methods. Mixed methods have taken root in some prestigious education schools and social science departments, but I regard it as a step backward, not forward.

    Take the questionnaire. Ask yourself what they hope to understand from the data in this survey.



    Original Message


Related Content