Good topic. Here are a couple observations from my experience working with PI's as they develop their grant proposals:
1) Poorly defined hypotheses: I've seen many "Aims" and "Hypotheses" where it was difficult to determine the primary exposure or group comparison being tested and/or the primary outcome variable (not just the anticipated distribution of the variable – I mean, I couldn’t even tell what the primary outcome was, period).
2) Insufficiently described "primary analysis" for Aim 1: certainly, it is unreasonable to expect the PI to lay out every single possible analysis that might be considered within their grant’s data, but I've seen several analysis plans with no substance at all - clearly someone who glossed over the very principle. If someone merely says “Statistical analysis will be performed using standard methods in MedCalc software” (yes, I’ve seen this as a “statistical analysis plan” before!) that should be instant rejection. They have to at least be able to name the primary hypothesis’ testing method.
3) Insufficiently described power and sample size calculations: I think we statisticians all agree that power calculations are a maddening topic to describe to the layperson (plus, they already have lots of built-in imprecision), so I am somewhat sympathetic here. However, every quarter someone brings me a grant proposal with a nonsense power calculation entirely plucked from thin air (i.e. "Collection of 20 samples will have sufficient statistical power to show differences between groups" with no mention of the hypothesized effect size, measure of variability, the test being used, etc).
In the interest of being more constructive - here are a few things that I would recommend to any PI:
- Meet with a statistician for help, and then tell them to help you with the following items:
- Make sure your primary hypothesis CLEARLY IDENTIFIES the primary outcome variable.
- Make sure it is clear whether the primary outcome is a continuous variable, categorical variable, or a time-to-event (survival) variable.
- Make sure it is clear what the comparison groups / exposure variable is
- Make sure the analysis plan identifies the specific statistical method which will be used to test the primary hypothesis.
- Make sure the description of your power and sample size calculation names ALL of the necessary elements: the statistical test, anticipated effect size, estimates of variability, significance level, etc.
This is a very short list, and admittedly does not cover all possible errors (for example, people who name a test that is irrelevant or incorrect – such as a PI that references a “t-test” or “ANOVA” when the named test is not suitable for their primary question – but I think that’s outside the scope of what can be covered in a guide for grant writers).
------------------------------
Andrew D. Althouse, PhD
Supervisor of Statistical Projects
UPMC Heart & Vascular Institute
Presbyterian Hospital, Office C701
Phone: 412-802-6811
Email:
althousead@upmc.edu
Original Message:
Sent: 07-22-2016 14:27
From: Steve Pierson
Subject: Top statistical issues seen in (non-statistics) grant proposals
Dear Statistical Community,
A recent survey of NIH reviewers revealed this top concern: "Having more experienced reviewers especially those with statistical, biostatistical, or clinical expertise." In an effort to help both proposals reviewers and applicants, the ASA would like to compile a list of top statistical issues seen in (non-statistics) proposals. We’d especially like to focus on biomedical research proposals but input on other proposals is also welcome.
Please send what you see as the top statistical issues seen in proposals to me by August 8. You can also frame it as statistical advice for proposals.
Let us also encourage you to consider becoming a reviewer, if you aren't already. Here are some relevant resources:
Thank you,
Steve
------------------------------
Steve Pierson
Director of Science Policy
American Statistical Association
------------------------------