Hello all,
I want to share a post I wrote under a pseudonym (P. Hackman, PhD). The goal was to expose a real failure mode: how model selection combined with unadjusted post-selection inference can be used to all but guarantee statistically significant results.
Link: https://data-diction.com/posts/upsi-example/
I'd be interested in hearing your perspectives on a few questions:
1) How do you explain post-selection inference issues to collaborators in practice?
2) Where do you draw the line between routine model building and analysis choices that require adjustment or reframing?
3) Do satirical examples like this clarify the problem, or do they risk normalizing behavior we'd rather discourage?
As someone stepping into a leadership role in the section next year, I'd like to help this forum continue to be a place where we regularly have constructive discussions about issues like this. I encourage replies here, or new threads on similar challenges that arise in practice. Our forum of practicing statisticians offers something increasingly rare: context-rich advice shaped by real collaboration, incentives, and consequences.
Thank you!
------------------------------
Ryan Peterson
ASA Statistical Consulting Section Chair-Elect
Associate Professor
University of Iowa
------------------------------