ASA Connect

As a follow-up to an message I sent out several weeks ago, I developed a draft policy on research grants. It stated

1. the normal support level should be 20%, higher for complex projects, lower if the PI does his/her own data analysis, but never below 10%.
2. we need at least a full month lead time, ideally two months, if you want us to work on the grant.
3. we will not work on a grant that does not budget for a data manager.
4. you cannot change our percentage effort after the fact without renegotiating the scope of work.

I have an early draft at PMean: Draft policy on statistical support for research. It needs a lot of polishing, but the general goal is to set some minimum expectations on our end.

I'm getting a lot of push back on the first item. One person wrote "efforts of 20% or higher for statistical support are very rare." Another said 20% support is mostly "if the stats person is the PI and the project is a stats-centric project."

What's the experience of others? If you've served on review panels, do you see us as being doomed to a life of 5% effort here and 10% effort there? if you are on grants, do you ever get anything close to 20% support without being the PI? If you routinely get 5% to 10% support does that really cover the amount of work that you do?

------------------------------
Stephen Simon, blog.pmean.com
Independent Statistical Consultant
P. Mean Consulting
------------------------------

A very fine question, indeed!  May I suggest conducting a small survey, ie, as can be designed quickly with google forms or surveymonkey?  You may wish to collect covariates such as rank/title/location so it's possible to post-stratify the results to reflect various demographic assumptions.

------------------------------
Andrew McDavid
Biostatistics and Computational Biology
University of Rochester
------------------------------

Original Message:
Sent: 05-10-2018 18:28
From: Stephen Simon
Subject: What percentage effort is normal for statisticians on research grants

As a follow-up to an message I sent out several weeks ago, I developed a draft policy on research grants. It stated

1. the normal support level should be 20%, higher for complex projects, lower if the PI does his/her own data analysis, but never below 10%.
2. we need at least a full month lead time, ideally two months, if you want us to work on the grant.
3. we will not work on a grant that does not budget for a data manager.
4. you cannot change our percentage effort after the fact without renegotiating the scope of work.

I have an early draft at PMean: Draft policy on statistical support for research. It needs a lot of polishing, but the general goal is to set some minimum expectations on our end.

I'm getting a lot of push back on the first item. One person wrote "efforts of 20% or higher for statistical support are very rare." Another said 20% support is mostly "if the stats person is the PI and the project is a stats-centric project."

What's the experience of others? If you've served on review panels, do you see us as being doomed to a life of 5% effort here and 10% effort there? if you are on grants, do you ever get anything close to 20% support without being the PI? If you routinely get 5% to 10% support does that really cover the amount of work that you do?

------------------------------
Stephen Simon, blog.pmean.com
Independent Statistical Consultant
P. Mean Consulting
------------------------------

In a medical school setting, I use 30% for grants where I am the PI, 20% for collaborative complex studies, 10% for large but straight-forward studies, and 5% for basic lab/animal lab studies. HOWEVER, I always include a MS-level statistical analyst at double (or more) my effort to do the "hands-on" work with my direction. It is rare to see 20% Ph.D. statistician support these days on grants unless there is something really complex involved. 
Hope that helps.


------------------------------
Bruce Barton
Professor, Div. Biostatistics
Univ. Massachusetts Medical School
------------------------------

Original Message:
Sent: 05-10-2018 18:28
From: Stephen Simon
Subject: What percentage effort is normal for statisticians on research grants

As a follow-up to an message I sent out several weeks ago, I developed a draft policy on research grants. It stated

1. the normal support level should be 20%, higher for complex projects, lower if the PI does his/her own data analysis, but never below 10%.
2. we need at least a full month lead time, ideally two months, if you want us to work on the grant.
3. we will not work on a grant that does not budget for a data manager.
4. you cannot change our percentage effort after the fact without renegotiating the scope of work.

I have an early draft at PMean: Draft policy on statistical support for research. It needs a lot of polishing, but the general goal is to set some minimum expectations on our end.

I'm getting a lot of push back on the first item. One person wrote "efforts of 20% or higher for statistical support are very rare." Another said 20% support is mostly "if the stats person is the PI and the project is a stats-centric project."

What's the experience of others? If you've served on review panels, do you see us as being doomed to a life of 5% effort here and 10% effort there? if you are on grants, do you ever get anything close to 20% support without being the PI? If you routinely get 5% to 10% support does that really cover the amount of work that you do?

------------------------------
Stephen Simon, blog.pmean.com
Independent Statistical Consultant
P. Mean Consulting
------------------------------

The late, wonderful Todd Nick and I wrote a little piece on this very topic.

Nick, T. and O'Brien, R. (2010). Developing grant proposals: Guidelines for statisticians collaborating under limited resources,. Chance, 23:39–40.

------------------------------
Ralph O'Brien
Professor of Biostatistics (officially retired; still keenly active)
Case Western Reserve University
http://rfuncs.weebly.com/about-ralph-obrien.html
------------------------------

Original Message:
Sent: 05-10-2018 18:28
From: Stephen Simon
Subject: What percentage effort is normal for statisticians on research grants

As a follow-up to an message I sent out several weeks ago, I developed a draft policy on research grants. It stated

1. the normal support level should be 20%, higher for complex projects, lower if the PI does his/her own data analysis, but never below 10%.
2. we need at least a full month lead time, ideally two months, if you want us to work on the grant.
3. we will not work on a grant that does not budget for a data manager.
4. you cannot change our percentage effort after the fact without renegotiating the scope of work.

I have an early draft at PMean: Draft policy on statistical support for research. It needs a lot of polishing, but the general goal is to set some minimum expectations on our end.

I'm getting a lot of push back on the first item. One person wrote "efforts of 20% or higher for statistical support are very rare." Another said 20% support is mostly "if the stats person is the PI and the project is a stats-centric project."

What's the experience of others? If you've served on review panels, do you see us as being doomed to a life of 5% effort here and 10% effort there? if you are on grants, do you ever get anything close to 20% support without being the PI? If you routinely get 5% to 10% support does that really cover the amount of work that you do?

------------------------------
Stephen Simon, blog.pmean.com
Independent Statistical Consultant
P. Mean Consulting
------------------------------

Original Message------

The late, wonderful Todd Nick and I wrote a little piece on this very topic.

Nick, T. and O'Brien, R. (2010). Developing grant proposals: Guidelines for statisticians collaborating under limited resources,. Chance, 23:39–40.

------------------------------
Ralph O'Brien
Professor of Biostatistics (officially retired; still keenly active)
Case Western Reserve University
http://rfuncs.weebly.com/about-ralph-obrien.html
------------------------------

For its document, Statistical Issues Seen in Non-Statistics Proposals, the ASA Committee on Funded Research (CFR) links to University of California Davis School of Medicine Guidelines for Estimating Biostatistician Effort and Resources on Grants. 

It may also help to make your collaborators and potential collaborators aware of the ASA CFR document (i.e., Statistical Issues Seen in Non-Statistics Proposals.)

Best,
Steve

------------------------------
Steve Pierson
Director of Science Policy
American Statistical Association
------------------------------

Original Message:
Sent: 05-14-2018 14:23
From: Frank Harrell
Subject: What percentage effort is normal for statisticians on research grants

Very nice article.  I miss Todd Nick.  I was so lucky to get to work with him a lot.  We lost him way too soon.  

We tend to have a minimum of PhD time of 10% for an R01 grant.

---

Frank E Harrell Jr	Professor	School of Medicine
	Department of Biostatistics	Vanderbilt University

Original Message------

The late, wonderful Todd Nick and I wrote a little piece on this very topic.

Nick, T. and O'Brien, R. (2010). Developing grant proposals: Guidelines for statisticians collaborating under limited resources,. Chance, 23:39–40.

------------------------------
Ralph O'Brien
Professor of Biostatistics (officially retired; still keenly active)
Case Western Reserve University
http://rfuncs.weebly.com/about-ralph-obrien.html
------------------------------

Original Message------

As a follow-up to an message I sent out several weeks ago, I developed a draft policy on research grants. It stated

1. the normal support level should be 20%, higher for complex projects, lower if the PI does his/her own data analysis, but never below 10%.
2. we need at least a full month lead time, ideally two months, if you want us to work on the grant.
3. we will not work on a grant that does not budget for a data manager.
4. you cannot change our percentage effort after the fact without renegotiating the scope of work.

I have an early draft at PMean: Draft policy on statistical support for research. It needs a lot of polishing, but the general goal is to set some minimum expectations on our end.

I'm getting a lot of push back on the first item. One person wrote "efforts of 20% or higher for statistical support are very rare." Another said 20% support is mostly "if the stats person is the PI and the project is a stats-centric project."

What's the experience of others? If you've served on review panels, do you see us as being doomed to a life of 5% effort here and 10% effort there? if you are on grants, do you ever get anything close to 20% support without being the PI? If you routinely get 5% to 10% support does that really cover the amount of work that you do?

------------------------------
Stephen Simon, blog.pmean.com
Independent Statistical Consultant
P. Mean Consulting
------------------------------

I confess to finding this discussion puzzling, as I have with others in the past.

For most projects I consult on (which may be thousands of subjects in a chart review, not necessarily small in magnitude), I usually get the data in Excel.

2-3 hours of back and forth about bugs and errors, cleaning the data, getting it ready for analysis.

2-3 hours talking about the analyses, a meeting or two.

5-10 hours running t-tests, ANOVAs, multiple linear regressions, Kaplan-Meier curves, etc... and summarizing the results.

2-3 hours helping the client write up the results.

2-3 hours responding to reviewer requests.

Even counting 3 and 10 (rather than 5 and 2) it's 22 hours for a typical project. That's less than 1% of 1 year. I cannot figure out what people are actually doing with 5% time (let alone 10% or more) unless you are working with data collectors on a day to day basis -- I've never done (or been asked to do) that.

I'm on a grant now for about 3% time. Is it enough?  I've spent 6 months waiting for the PI to actually send me any data. When last he did, it took me a few hours to analyze.  I've had to give back time!  And this is 3%.

Can somebody talk me through why you typically need 200 hours for a data analysis project (10% time for a year or 5% for two years)? I can see that if the project has 40 components and needs multiple interim analyses, or if you are doing very extensive oversight. But you are talking about 5% for a typical small lab project.  I don't see it.

Thoughts?

Ed

------------------------------
Edward Gracely
Drexel University
------------------------------

Original Message:
Sent: 05-10-2018 18:28
From: Stephen Simon
Subject: What percentage effort is normal for statisticians on research grants

As a follow-up to an message I sent out several weeks ago, I developed a draft policy on research grants. It stated

1. the normal support level should be 20%, higher for complex projects, lower if the PI does his/her own data analysis, but never below 10%.
2. we need at least a full month lead time, ideally two months, if you want us to work on the grant.
3. we will not work on a grant that does not budget for a data manager.
4. you cannot change our percentage effort after the fact without renegotiating the scope of work.

I have an early draft at PMean: Draft policy on statistical support for research. It needs a lot of polishing, but the general goal is to set some minimum expectations on our end.

I'm getting a lot of push back on the first item. One person wrote "efforts of 20% or higher for statistical support are very rare." Another said 20% support is mostly "if the stats person is the PI and the project is a stats-centric project."

What's the experience of others? If you've served on review panels, do you see us as being doomed to a life of 5% effort here and 10% effort there? if you are on grants, do you ever get anything close to 20% support without being the PI? If you routinely get 5% to 10% support does that really cover the amount of work that you do?

------------------------------
Stephen Simon, blog.pmean.com
Independent Statistical Consultant
P. Mean Consulting
------------------------------

I review research grant applications as a member of an NIH study section. In my experience it is very unusual for a grant proposal to be criticized for too much statistical support, and in my experience that only happens if there are multiple statisticians who seem to be duplicating effort. I don't think that I have never seen 30% of a doctoral statistician and 60% for a masters statistician cause a problem. However, some applications suffer from inadequate statistical support and do get downgraded for that. I have also seen some grant applications where study section members notice that a statistician is so spread out so many different grants that they question whether the statistician will actually be able to make a significant contribution.  Remember that 10% level of effort is only four hour per week and it is hard to make a contribution with that little participation in the research. 

------------------------------
Charles Hall
Professor
Albert Einstein College of Medicine
Bronx NY United States
------------------------------

Original Message:
Sent: 05-14-2018 14:27
From: Edward Gracely
Subject: What percentage effort is normal for statisticians on research grants

I confess to finding this discussion puzzling, as I have with others in the past.

For most projects I consult on (which may be thousands of subjects in a chart review, not necessarily small in magnitude), I usually get the data in Excel.

2-3 hours of back and forth about bugs and errors, cleaning the data, getting it ready for analysis.

2-3 hours talking about the analyses, a meeting or two.

5-10 hours running t-tests, ANOVAs, multiple linear regressions, Kaplan-Meier curves, etc... and summarizing the results.

2-3 hours helping the client write up the results.

2-3 hours responding to reviewer requests.

Even counting 3 and 10 (rather than 5 and 2) it's 22 hours for a typical project. That's less than 1% of 1 year. I cannot figure out what people are actually doing with 5% time (let alone 10% or more) unless you are working with data collectors on a day to day basis -- I've never done (or been asked to do) that.

I'm on a grant now for about 3% time. Is it enough?  I've spent 6 months waiting for the PI to actually send me any data. When last he did, it took me a few hours to analyze.  I've had to give back time!  And this is 3%.

Can somebody talk me through why you typically need 200 hours for a data analysis project (10% time for a year or 5% for two years)? I can see that if the project has 40 components and needs multiple interim analyses, or if you are doing very extensive oversight. But you are talking about 5% for a typical small lab project.  I don't see it.

Thoughts?

Ed

------------------------------
Edward Gracely
Drexel University

Original Message:
Sent: 05-10-2018 18:28
From: Stephen Simon
Subject: What percentage effort is normal for statisticians on research grants

As a follow-up to an message I sent out several weeks ago, I developed a draft policy on research grants. It stated

1. the normal support level should be 20%, higher for complex projects, lower if the PI does his/her own data analysis, but never below 10%.
2. we need at least a full month lead time, ideally two months, if you want us to work on the grant.
3. we will not work on a grant that does not budget for a data manager.
4. you cannot change our percentage effort after the fact without renegotiating the scope of work.

I have an early draft at PMean: Draft policy on statistical support for research. It needs a lot of polishing, but the general goal is to set some minimum expectations on our end.

I'm getting a lot of push back on the first item. One person wrote "efforts of 20% or higher for statistical support are very rare." Another said 20% support is mostly "if the stats person is the PI and the project is a stats-centric project."

What's the experience of others? If you've served on review panels, do you see us as being doomed to a life of 5% effort here and 10% effort there? if you are on grants, do you ever get anything close to 20% support without being the PI? If you routinely get 5% to 10% support does that really cover the amount of work that you do?

------------------------------
Stephen Simon, blog.pmean.com
Independent Statistical Consultant
P. Mean Consulting
------------------------------

I have to respond to Ed Gracely's estimate that a typical effort on a research grant might involve, at most, 22 hours of effort. To be honest, there are settings where this happens for me, but more often than not, I put in a lot more than 22 hours of effort. This includes:

* helping write documents for IRB, and responding to questions from the IRB,

* preparation of data collection systems,

* review of questionnaires,

* development of randomization tables,

* regular and on-going reviews of data as it is being collected,

* discussions about small refinements and sometimes major changes in the protocol during the study.

This all comes before the data analysis that Dr. Gracely describes in his email. Once the data analysis is done, however, there is still more work to prepare the manuscript. I find that Dr. Gracely's estimates of 4 to 6 hours are a gross underestimate, mostly because the people I work with have a lot of trouble writing and I end up reviewing and revising their work a lot.

It is also fairly common to see two or three distinct publications coming from the same research grant. I'm sure that I left out a lot of things here, but even with just the things that I did mention, it would certainly require much more than 22 hours of effort.

Don't get me wrong. Some clients are in my door and out again in an hour and they have everything they need. Some require a second or third hour of work. But there are many more clients where I sweat and slave and it comes out to a lot more than 22 hours of work. Thankfully it is spread out over a long time frame, but it's still a lot of work.

------------------------------
Stephen Simon, blog.pmean.com
Independent Statistical Consultant
P. Mean Consulting
------------------------------

Original Message:
Sent: 05-14-2018 14:27
From: Edward Gracely
Subject: What percentage effort is normal for statisticians on research grants

I confess to finding this discussion puzzling, as I have with others in the past.

For most projects I consult on (which may be thousands of subjects in a chart review, not necessarily small in magnitude), I usually get the data in Excel.

2-3 hours of back and forth about bugs and errors, cleaning the data, getting it ready for analysis.

2-3 hours talking about the analyses, a meeting or two.

5-10 hours running t-tests, ANOVAs, multiple linear regressions, Kaplan-Meier curves, etc... and summarizing the results.

2-3 hours helping the client write up the results.

2-3 hours responding to reviewer requests.

Even counting 3 and 10 (rather than 5 and 2) it's 22 hours for a typical project. That's less than 1% of 1 year. I cannot figure out what people are actually doing with 5% time (let alone 10% or more) unless you are working with data collectors on a day to day basis -- I've never done (or been asked to do) that.

I'm on a grant now for about 3% time. Is it enough?  I've spent 6 months waiting for the PI to actually send me any data. When last he did, it took me a few hours to analyze.  I've had to give back time!  And this is 3%.

Can somebody talk me through why you typically need 200 hours for a data analysis project (10% time for a year or 5% for two years)? I can see that if the project has 40 components and needs multiple interim analyses, or if you are doing very extensive oversight. But you are talking about 5% for a typical small lab project.  I don't see it.

Thoughts?

Ed

------------------------------
Edward Gracely
Drexel University

Original Message:
Sent: 05-10-2018 18:28
From: Stephen Simon
Subject: What percentage effort is normal for statisticians on research grants

As a follow-up to an message I sent out several weeks ago, I developed a draft policy on research grants. It stated

1. the normal support level should be 20%, higher for complex projects, lower if the PI does his/her own data analysis, but never below 10%.
2. we need at least a full month lead time, ideally two months, if you want us to work on the grant.
3. we will not work on a grant that does not budget for a data manager.
4. you cannot change our percentage effort after the fact without renegotiating the scope of work.

I have an early draft at PMean: Draft policy on statistical support for research. It needs a lot of polishing, but the general goal is to set some minimum expectations on our end.

I'm getting a lot of push back on the first item. One person wrote "efforts of 20% or higher for statistical support are very rare." Another said 20% support is mostly "if the stats person is the PI and the project is a stats-centric project."

What's the experience of others? If you've served on review panels, do you see us as being doomed to a life of 5% effort here and 10% effort there? if you are on grants, do you ever get anything close to 20% support without being the PI? If you routinely get 5% to 10% support does that really cover the amount of work that you do?

------------------------------
Stephen Simon, blog.pmean.com
Independent Statistical Consultant
P. Mean Consulting
------------------------------

A few others have already chimed in, but I also wanted to comment briefly on this.

I think what we have here is a bit of a terminology issue.

Ed: respectfully, that describes the likely analytic flow for a single analysis project.  However, most grant submissions (whether they pre-specify this or not) are going to do far more than one analysis project with the data that they collect, and this can and likely will encompass much more than what you have laid out.  If I am listed at 10-20 percent effort on a grant, I am (most likely) doing a lot more than what you have described.

I don't think any of us expects 20 percent effort for a project where we are analyzing the results from one or two Excel sheets and preparing a descriptive table or two, a few t-tests or ANOVA, and a Kaplan-Meier curve.  But, as alluded, the duties for a 20% grant are usually a combo of study design consultation, preparation of IRB documents, preparation of interim reports (where appropriate), discussions of protocol modifications, and planning analyses of some ancillary projects that grew out of the original submission.  Plus, most of these grants are not intending to collect their data and write just 1 paper, but often multiple papers for second and third aims, plus usually some ancillary projects.

Moving past that, if we do limit to a "single analysis project" view of things:

I must also second Stephen Simon's prior comment that some of your time estimates sound like gross underestimates; without  taking credit entirely from yourself as you must be a very efficient worker and programmer, you must also have exceptionally good clients (and likely very good client-statistician relationships) if that is all it takes you to complete some of those tasks.  It typically takes much more effort on my part for "talking about bugs and errors & cleaning the data" (some/many of which can be prevented if I help design the database used for data collection and entry, but then that's creating a new bin of "effort" not accounted for in your discussion).  Similarly, the tasks of "helping the client write up the results" and "responding to reviewer requests" portion are nearly always more than 2-3 hours in my division (I had to send three separate emails yesterday to answer a straightforward question .  Admittedly, these tasks can be quite variable from field to field (i.e. complexity/rigidity of statistical analysis plans & scrutiny of review is very different depending on whom the client is and what journal they are submitting to) and the PI themselves will have a large effect over this.  As Stephen Simon said, the people that I work with have a lot of trouble writing, change their mind about which analyses to include, request a couple rounds of redirected / secondary / sensitivity analyses that require revisiting and occasionally re-running much of original analysis, etc.  You are a very fortunate man if you get brought a dataset, take 2-3 hours to clean it, 2-3 hours to discuss it, 5-10 hours to analyze it, 2-3 hours on the back end for writing, and 2-3 hours for helping reply to comments.  That is a very smooth and ideal project flow and in my dream world, that seems like it should be possible, but it's been extremely rare in practice.

The data analysis is nearly always the easiest part of my collaborative projects.  For straightforward stuff (descriptives, t-tests, ANOVA, linear or logistic regression) I can get that done in a day or two most of the time.  If it's a dataset that I've already worked with before, maybe less.  But those other tasks often take a much larger portion, and I'm almost never lucky enough to get away with analyzing the data just once (here's a recent example - from a non-grant-funded project, which explains some of what you're about to read: I analyzed data from a retrospective observational study comparing outcomes with two treatments; standard Table 1 comparing the groups, Kaplan-Meier curve, Cox model, etc which probably took about a day from start to finish.  Since then, the PI's have decided to add a handful of secondary outcomes that required a fresh round of data collection; add a propensity-score matched analysis comparing the two treatments; remove a specific type of patients, requiring us to reproduce the entire packet from start to finish; and around and around we go...)

------------------------------
Andrew D. Althouse, PhD
Supervisor of Statistical Projects
UPMC Heart & Vascular Institute
Presbyterian Hospital, Office C701
Phone: 412-802-6811
Email: althousead@upmc.edu
Twitter: @ADAlthousePhD
------------------------------

Original Message:
Sent: 05-14-2018 14:27
From: Edward Gracely
Subject: What percentage effort is normal for statisticians on research grants

I confess to finding this discussion puzzling, as I have with others in the past.

For most projects I consult on (which may be thousands of subjects in a chart review, not necessarily small in magnitude), I usually get the data in Excel.

2-3 hours of back and forth about bugs and errors, cleaning the data, getting it ready for analysis.

2-3 hours talking about the analyses, a meeting or two.

5-10 hours running t-tests, ANOVAs, multiple linear regressions, Kaplan-Meier curves, etc... and summarizing the results.

2-3 hours helping the client write up the results.

2-3 hours responding to reviewer requests.

Even counting 3 and 10 (rather than 5 and 2) it's 22 hours for a typical project. That's less than 1% of 1 year. I cannot figure out what people are actually doing with 5% time (let alone 10% or more) unless you are working with data collectors on a day to day basis -- I've never done (or been asked to do) that.

I'm on a grant now for about 3% time. Is it enough?  I've spent 6 months waiting for the PI to actually send me any data. When last he did, it took me a few hours to analyze.  I've had to give back time!  And this is 3%.

Can somebody talk me through why you typically need 200 hours for a data analysis project (10% time for a year or 5% for two years)? I can see that if the project has 40 components and needs multiple interim analyses, or if you are doing very extensive oversight. But you are talking about 5% for a typical small lab project.  I don't see it.

Thoughts?

Ed

------------------------------
Edward Gracely
Drexel University

Original Message:
Sent: 05-10-2018 18:28
From: Stephen Simon
Subject: What percentage effort is normal for statisticians on research grants

As a follow-up to an message I sent out several weeks ago, I developed a draft policy on research grants. It stated

1. the normal support level should be 20%, higher for complex projects, lower if the PI does his/her own data analysis, but never below 10%.
2. we need at least a full month lead time, ideally two months, if you want us to work on the grant.
3. we will not work on a grant that does not budget for a data manager.
4. you cannot change our percentage effort after the fact without renegotiating the scope of work.

I have an early draft at PMean: Draft policy on statistical support for research. It needs a lot of polishing, but the general goal is to set some minimum expectations on our end.

I'm getting a lot of push back on the first item. One person wrote "efforts of 20% or higher for statistical support are very rare." Another said 20% support is mostly "if the stats person is the PI and the project is a stats-centric project."

What's the experience of others? If you've served on review panels, do you see us as being doomed to a life of 5% effort here and 10% effort there? if you are on grants, do you ever get anything close to 20% support without being the PI? If you routinely get 5% to 10% support does that really cover the amount of work that you do?

------------------------------
Stephen Simon, blog.pmean.com
Independent Statistical Consultant
P. Mean Consulting
------------------------------

Original Message------

A few others have already chimed in, but I also wanted to comment briefly on this.

I think what we have here is a bit of a terminology issue.

Ed: respectfully, that describes the likely analytic flow for a single analysis project.  However, most grant submissions (whether they pre-specify this or not) are going to do far more than one analysis project with the data that they collect, and this can and likely will encompass much more than what you have laid out.  If I am listed at 10-20 percent effort on a grant, I am (most likely) doing a lot more than what you have described.

I don't think any of us expects 20 percent effort for a project where we are analyzing the results from one or two Excel sheets and preparing a descriptive table or two, a few t-tests or ANOVA, and a Kaplan-Meier curve.  But, as alluded, the duties for a 20% grant are usually a combo of study design consultation, preparation of IRB documents, preparation of interim reports (where appropriate), discussions of protocol modifications, and planning analyses of some ancillary projects that grew out of the original submission.  Plus, most of these grants are not intending to collect their data and write just 1 paper, but often multiple papers for second and third aims, plus usually some ancillary projects.

Moving past that, if we do limit to a "single analysis project" view of things:

I must also second Stephen Simon's prior comment that some of your time estimates sound like gross underestimates; without  taking credit entirely from yourself as you must be a very efficient worker and programmer, you must also have exceptionally good clients (and likely very good client-statistician relationships) if that is all it takes you to complete some of those tasks.  It typically takes much more effort on my part for "talking about bugs and errors & cleaning the data" (some/many of which can be prevented if I help design the database used for data collection and entry, but then that's creating a new bin of "effort" not accounted for in your discussion).  Similarly, the tasks of "helping the client write up the results" and "responding to reviewer requests" portion are nearly always more than 2-3 hours in my division (I had to send three separate emails yesterday to answer a straightforward question .  Admittedly, these tasks can be quite variable from field to field (i.e. complexity/rigidity of statistical analysis plans & scrutiny of review is very different depending on whom the client is and what journal they are submitting to) and the PI themselves will have a large effect over this.  As Stephen Simon said, the people that I work with have a lot of trouble writing, change their mind about which analyses to include, request a couple rounds of redirected / secondary / sensitivity analyses that require revisiting and occasionally re-running much of original analysis, etc.  You are a very fortunate man if you get brought a dataset, take 2-3 hours to clean it, 2-3 hours to discuss it, 5-10 hours to analyze it, 2-3 hours on the back end for writing, and 2-3 hours for helping reply to comments.  That is a very smooth and ideal project flow and in my dream world, that seems like it should be possible, but it's been extremely rare in practice.

The data analysis is nearly always the easiest part of my collaborative projects.  For straightforward stuff (descriptives, t-tests, ANOVA, linear or logistic regression) I can get that done in a day or two most of the time.  If it's a dataset that I've already worked with before, maybe less.  But those other tasks often take a much larger portion, and I'm almost never lucky enough to get away with analyzing the data just once (here's a recent example - from a non-grant-funded project, which explains some of what you're about to read: I analyzed data from a retrospective observational study comparing outcomes with two treatments; standard Table 1 comparing the groups, Kaplan-Meier curve, Cox model, etc which probably took about a day from start to finish.  Since then, the PI's have decided to add a handful of secondary outcomes that required a fresh round of data collection; add a propensity-score matched analysis comparing the two treatments; remove a specific type of patients, requiring us to reproduce the entire packet from start to finish; and around and around we go...)

------------------------------
Andrew D. Althouse, PhD
Supervisor of Statistical Projects
UPMC Heart & Vascular Institute
Presbyterian Hospital, Office C701
Phone: 412-802-6811
Email: althousead@upmc.edu
Twitter: @ADAlthousePhD
------------------------------

Gregory—

Your point is well taken, and I agree with you on the point of hourly rates...particularly when the work is done in a consulting atmosphere in the private sector. When I worked in consulting full time, we submitted budgets with hourly estimates for technical work and were nearly always able to barter for more later on if needed.

However, there are important considerations in the Academy. First, grant submissions are based on budgets that often are calculated on FTE effort, not on hourly wages. Second, in most academic settings, there is a limit on the total amount of extra-contractual compensation that is allowed to be paid to Faculty, which is why FTE is used in the calculation (and this is often limited only to the summer months unless one is soft-money funded). Third, as grant budgets shrink over time (and they almost always do), the stats often get short shrift as PIs try to allocate more into the intervention/research budget without realizing the need for sound, statistical work that needs to be compensated appropriately.

Whatever the case, those of us working in the grant world do not purposely “leave money on the table”...we just rarely even know how many tables we are going to have at any given time, and we rarely have control over the ultimate budgets in grants that do get funded.

--
Chad L. Cross, PhD, PStat(R)
Biostatistician


Original Message------

I've been following this thread with some incredulity about the apparent practice in servicing grant-funded academic clients that certain statisticians quote a fixed price (which is what saying "20% effort" is) for highly technical and open-ended work. This is not done in any other area I know about where consultants are involved.

 

I would assume a PhD-level statistician would have as much bargaining power in the current marketplace as CPAs, attorneys, and management consultants who bill by the hour for complex, non-routine work. For forensic accounting work, where I use statistical techniques, I never provide an estimate. It's an hourly rate plus a retainer that needs to be paid before work begins, and once the retainer is exhausted, work stops until it is replenished.

 

This might just be culture shock, but I can't shake the feeling that you're collectively leaving money on the table.

 

 

 

Please note that I check email once a day and strive to return all messages by the end of the next business day. If urgent, please call or text me at 857.636.1694.

 

-- 
Gregory Csikos, CPA, CFE, GStat
www.csikoscpa.com | greg@csikoscpa.com

139A Charles Street, Suite 249

Boston, MA 02114

P: 857.636.1694
F: 857.201.3202

This e-mail and the documents accompanying it contain information which is confidential or privileged. The information is intended to be for the use of the individual or entity named in this transmission. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of the contents of this information is prohibited. If you have received this e-mail in error, please notify me by telephone immediately and promptly delete this email from your computer.

 

Original Message------

A few others have already chimed in, but I also wanted to comment briefly on this.

I think what we have here is a bit of a terminology issue.

Ed: respectfully, that describes the likely analytic flow for a single analysis project.  However, most grant submissions (whether they pre-specify this or not) are going to do far more than one analysis project with the data that they collect, and this can and likely will encompass much more than what you have laid out.  If I am listed at 10-20 percent effort on a grant, I am (most likely) doing a lot more than what you have described.

I don't think any of us expects 20 percent effort for a project where we are analyzing the results from one or two Excel sheets and preparing a descriptive table or two, a few t-tests or ANOVA, and a Kaplan-Meier curve.  But, as alluded, the duties for a 20% grant are usually a combo of study design consultation, preparation of IRB documents, preparation of interim reports (where appropriate), discussions of protocol modifications, and planning analyses of some ancillary projects that grew out of the original submission.  Plus, most of these grants are not intending to collect their data and write just 1 paper, but often multiple papers for second and third aims, plus usually some ancillary projects.

Moving past that, if we do limit to a "single analysis project" view of things:

I must also second Stephen Simon's prior comment that some of your time estimates sound like gross underestimates; without  taking credit entirely from yourself as you must be a very efficient worker and programmer, you must also have exceptionally good clients (and likely very good client-statistician relationships) if that is all it takes you to complete some of those tasks.  It typically takes much more effort on my part for "talking about bugs and errors & cleaning the data" (some/many of which can be prevented if I help design the database used for data collection and entry, but then that's creating a new bin of "effort" not accounted for in your discussion).  Similarly, the tasks of "helping the client write up the results" and "responding to reviewer requests" portion are nearly always more than 2-3 hours in my division (I had to send three separate emails yesterday to answer a straightforward question .  Admittedly, these tasks can be quite variable from field to field (i.e. complexity/rigidity of statistical analysis plans & scrutiny of review is very different depending on whom the client is and what journal they are submitting to) and the PI themselves will have a large effect over this.  As Stephen Simon said, the people that I work with have a lot of trouble writing, change their mind about which analyses to include, request a couple rounds of redirected / secondary / sensitivity analyses that require revisiting and occasionally re-running much of original analysis, etc.  You are a very fortunate man if you get brought a dataset, take 2-3 hours to clean it, 2-3 hours to discuss it, 5-10 hours to analyze it, 2-3 hours on the back end for writing, and 2-3 hours for helping reply to comments.  That is a very smooth and ideal project flow and in my dream world, that seems like it should be possible, but it's been extremely rare in practice.

The data analysis is nearly always the easiest part of my collaborative projects.  For straightforward stuff (descriptives, t-tests, ANOVA, linear or logistic regression) I can get that done in a day or two most of the time.  If it's a dataset that I've already worked with before, maybe less.  But those other tasks often take a much larger portion, and I'm almost never lucky enough to get away with analyzing the data just once (here's a recent example - from a non-grant-funded project, which explains some of what you're about to read: I analyzed data from a retrospective observational study comparing outcomes with two treatments; standard Table 1 comparing the groups, Kaplan-Meier curve, Cox model, etc which probably took about a day from start to finish.  Since then, the PI's have decided to add a handful of secondary outcomes that required a fresh round of data collection; add a propensity-score matched analysis comparing the two treatments; remove a specific type of patients, requiring us to reproduce the entire packet from start to finish; and around and around we go...)

------------------------------
Andrew D. Althouse, PhD
Supervisor of Statistical Projects
UPMC Heart & Vascular Institute
Presbyterian Hospital, Office C701
Phone: 412-802-6811
Email: althousead@upmc.edu
Twitter: @ADAlthousePhD
------------------------------

Hi,
I think there are distinctions to be made that depend upon the statistician's position.

For faculty of Statistics or Biostatistics departments, 20% - 30% is justified as the effort both ensures high-level statistical expertise on the project, but also protects some time from teaching and other responsibilities.

I served as a MS-level statistician on over 60 NIH-funded grants in my career and seldom would reviewers allow more than 10% even when I was the lead statistician -- which was the majority of cases. Only in the past 5-years have I seen reviewers request that the statistician's effort be boosted above 10%.

Given that, I served as the statistician to a single division in our Department of Medicine and was expected to generate 80% of my salary from grant support. Most often that meant funding from 6 to 10 studies in any given year. As different studies had different start dates and life-cycles, this was a juggling act, but entirely doable with a certain amount of self-discipline. Difficulties occur when 3 or 4 grants get funded on the same cycle, or it takes 3 or 4 grant cycles to get one application funded. I'm sure you are all familiar with this problem.

So I don't think there is a simple algorithmic solution to what is a) appropriate to the statistician's professional development; b) appropriate to their level of expertise; c) position within the organization; and d) the requirements of the particular project. The data analyst in me says I should estimate the project requirements, my needs, my organization's needs, and come up with a percent that is justifiable given the science proposed.

Donald J. McMahon, MS (Retired)
Columbia University Medical Center
Department of Medicine Division of Endocrinology
New York City, NY

------------------------------
Donald McMahon, MS (Retired)
Columbia University Medical Center
Department of Medicine Division of Endocrinology
New York City, NY
------------------------------

Original Message:
Sent: 05-10-2018 18:28
From: Stephen Simon
Subject: What percentage effort is normal for statisticians on research grants

As a follow-up to an message I sent out several weeks ago, I developed a draft policy on research grants. It stated

1. the normal support level should be 20%, higher for complex projects, lower if the PI does his/her own data analysis, but never below 10%.
2. we need at least a full month lead time, ideally two months, if you want us to work on the grant.
3. we will not work on a grant that does not budget for a data manager.
4. you cannot change our percentage effort after the fact without renegotiating the scope of work.

I have an early draft at PMean: Draft policy on statistical support for research. It needs a lot of polishing, but the general goal is to set some minimum expectations on our end.

I'm getting a lot of push back on the first item. One person wrote "efforts of 20% or higher for statistical support are very rare." Another said 20% support is mostly "if the stats person is the PI and the project is a stats-centric project."

What's the experience of others? If you've served on review panels, do you see us as being doomed to a life of 5% effort here and 10% effort there? if you are on grants, do you ever get anything close to 20% support without being the PI? If you routinely get 5% to 10% support does that really cover the amount of work that you do?

------------------------------
Stephen Simon, blog.pmean.com
Independent Statistical Consultant
P. Mean Consulting
------------------------------

Indeed, this is a vexing issue for me too.  I've been working as a Biostatistician for nearly 20 years now in two Universities (Medicine, Nursing) and Government (NASA Human Research).  Each position was much the same--with a primary mission of collaborating with content-expert PI's on grants and the scholarship that leads up or comes from them funded research programs.  Team science has always been my passion.

I have gained confidence over the years, and have had most success when I'm upfront with PI's at our initial consults.  I do not have a fixed policy on %FTE because some research involves more/less quantitative work.  Some require sophisticated randomization schemes and monitoring, some require DSMB's, many require mid-stream analytics, etc..  Having said that, I do go into most collaborations expecting 10% FTE, mainly out of necessity. I'm not always successful, and if I'm honest, it's typically not enough FTE.

While working in government (NASA), the usual expectation by collaborators there was a measly 5% FTE, but to be fair, NASA funds are very limited, and I was on a hard-funded line anyway, so the 5% FTE was more of a "retainer" of sorts so that the PI would know that she had statistical collaboration lined up.  Still, that meant that adding up all of the different grants that I supported put my true effort beyond what I could handle when crunch time hit, and all of my PI's were hoping to present their interim data at the same annual conferences. 

In academia, I can typically get 10% FTE covered if I make a strong case on R-level grants, though one PI recently dropped my effort from a grant  on re submission that had a solid review but one of the reviewers questioned why there would need to be a statistician at 10% FTE.  Interestingly, there were no criticisms of any of my contributions on this grant...  This this was a proposal involving multiple  animal study with 20 planned experiments.  Somehow this reviewer figured that analyzing data collected from animals somehow represented less work than analyzing data that comes from humans.  Not sure I follow that logic, but the PI was a well-funded PI for many years and had just recently begun using a statistician because his studies were getting too complicated for what he was trained to handle.  He asked if I would accept 5% FTE on the re-submission, and as much as I really enjoy working with this team, I declined because it was just too much work for that level of effort.  The grant did not get funded on resubmit.

In all of my negotiations, I'm most successful when I present the PI with the bottom line.  FTE is an odd currency, even for seasoned researchers,  so I find it useful to break it down into hours per week or month.

5% FTE equates to about one day per month, or 2 hours per week. 
10% FTE equates to two days per month, or 4 hours per week.
20% FTE equates to 4 days per month, or 8 hours per week (2-hours per day for 4 days)

Will the grant have weekly study meetings?  (1 hour per week).
Will the grant require statistician/pharmacy interactions for blinded drug/placebo assignments?  (1-2 hours per week)
Will the grant require consulting with data managers?  (mystery per week)
Will the grant require interim reports for funding agencies (many hours at crunch time)
Will the PI wish to report interim data at annual conferences (many hours at crunch time)
How much time will the final manuscripts take, including analysis, graphics, writing, responding to reviewers... (many hours)

The bottom line is that on any given study, weeks may go by with no statistical work to be handled, but we negotiate our %FTE so that we can cover the entirety of the work to be conducted, including those crunch times when our work will be intense. It's a very difficult thing to estimate for me, so I imagine PI's can't possibly understand the logic.   The only other option is to budget a dynamic FTE percent based on when we think our effort will be used, weighing heavy effort when the data are to be analyzed for interim reports, and heaviest at the end of study.  The difficult there is exactly how to negotiate that: 80% FTE for 6 months doesn't sound appealing to PI's, though that may actually be a reasonable estimate.  Also... funds tend to run out towards the end of the grant, so the money may not be there anyway.

And then, of course, the there are the non-R-level grants that either don't support collaborative %FTE, or have very tight limits on what they will support.  Those are necessary precursors to the R01, and they really must be supported somehow.  So we just do it.  We eat the FTE in order to gain collaborative experience and get our junior faculty the experience and pilot data they need in order to compete successfully for that future R01.  











------------------------------
Robert Ploutz-Snyder,PhD, PStat(r)
Research Professor & Director
Applied Biostatistics Laboratory
University of Michigan-School of Nursing
------------------------------

Original Message:
Sent: 05-10-2018 18:28
From: Stephen Simon
Subject: What percentage effort is normal for statisticians on research grants

As a follow-up to an message I sent out several weeks ago, I developed a draft policy on research grants. It stated

1. the normal support level should be 20%, higher for complex projects, lower if the PI does his/her own data analysis, but never below 10%.
2. we need at least a full month lead time, ideally two months, if you want us to work on the grant.
3. we will not work on a grant that does not budget for a data manager.
4. you cannot change our percentage effort after the fact without renegotiating the scope of work.

I have an early draft at PMean: Draft policy on statistical support for research. It needs a lot of polishing, but the general goal is to set some minimum expectations on our end.

I'm getting a lot of push back on the first item. One person wrote "efforts of 20% or higher for statistical support are very rare." Another said 20% support is mostly "if the stats person is the PI and the project is a stats-centric project."

What's the experience of others? If you've served on review panels, do you see us as being doomed to a life of 5% effort here and 10% effort there? if you are on grants, do you ever get anything close to 20% support without being the PI? If you routinely get 5% to 10% support does that really cover the amount of work that you do?

------------------------------
Stephen Simon, blog.pmean.com
Independent Statistical Consultant
P. Mean Consulting
------------------------------

This is a very interesting string of discussion. Most of what I feel (about the complexity of projects and the need for far more than 22 hours) has already been said, but I'll add a few notes that I hope are new to the discussion.

Although in many ways, I like the idea of setting some guidelines, I am not in favor of it for myself; however, I can see where the need arises, especially in a service unit.

First, and most importantly, the problem is not really so much time as it is the mental bandwidth one has for collaboration across 30+ projects that could come through your door at any time that also cover different disciplines (none of which I have any formal training on).

So for instance, when I was in our Biostatistics Facility in the Cancer Center, we had a small group to handle a large number of investigators and were somewhat obligated to take projects as they arose. At one point I had 30% on the cancer center support grant, where I worked with about 20-30 different investigators (some of whom were very needy - those projects alone could have filled my time with no trouble) and then had the other 70% time was spread over 17 different grants. In that situation, some rules for our group to follow as were proposed at the beginning of this email string would have been great.
 
As another example where such rules are useful/necessary, in my previous position in medicine, I started as 50% on our CTSA biostat core - where anyone of the hundreds of research faculty in the health science at Pitt could come through my door at any time - and the other 50% of my time was funded by a data center that also serviced a huge number of investigators. It was awful! I worked with >200 different investigators in my first 2 years. I threatened to quit on a regular basis until I got my own section, out of the data center, and worked to get PI roles on 3 different projects. 

Now, in a department of biomedical informatics, I fund about 30% of my time on a training/education grant as the PI, another 10% on running a data coordinating center (which should be more) and then am very picky on what else I will take. My other projects are largely a result of the connections I make as the director of biostatistics (for research) for a transplant institute, a member of our Comparative Effectiveness Research Center, and a few select collaborators on biomarker and physical activity studies in rheumatology, nephrology and hematology.

Now I think in terms of % effort for my staff to do the work and what % effort I can possibly spare and still stay at 80% grant funded (with 5% covered on non-grant activities by my department for teaching and the two above-mentioned institutes/centers). 

So, in my current situation, the rules stated initially are of no interest to me, but I totally get the need to consider that in other cases. Otherwise - if you are in a largely grant-funded research setting - things get out of hand very quickly. It's a really bad feeling when someone walks in your door saying "hey if I could just have a second of your time to answer some questions about that project we worked on last year" and 1) you remember them but can't remember for the life of you what their project is about, and 2) you can bet that their question is going to take a hell of a lot more than a few seconds, and you've got 20 other projects piled up and due that day.

Finally, I think times are changing in terms of % effort being questioned by study sections. I just finished one 4-year appointment and am starting another 4-year appointment now (and have done >50 ad hoc reviews). My experience is that - if it's a complex epi study - you want at least 10-20% for the statistician as a co-investigator + support staff. It's been a long time since I've heard anyone criticize that but I see people get criticized all the time for complex studies where the statistician is a consultant and/or <10%, especially for clinical trials. Basic science and animal studies are different.
 
Thanks to all for the interesting discussion.

------------------------------
Douglas Landsittel
Professor of Biomedical Informatics
University of Pittsburgh
------------------------------

Original Message:
Sent: 05-15-2018 08:52
From: Robert Ploutz-Snyder
Subject: What percentage effort is normal for statisticians on research grants

Indeed, this is a vexing issue for me too.  I've been working as a Biostatistician for nearly 20 years now in two Universities (Medicine, Nursing) and Government (NASA Human Research).  Each position was much the same--with a primary mission of collaborating with content-expert PI's on grants and the scholarship that leads up or comes from them funded research programs.  Team science has always been my passion.

I have gained confidence over the years, and have had most success when I'm upfront with PI's at our initial consults.  I do not have a fixed policy on %FTE because some research involves more/less quantitative work.  Some require sophisticated randomization schemes and monitoring, some require DSMB's, many require mid-stream analytics, etc..  Having said that, I do go into most collaborations expecting 10% FTE, mainly out of necessity. I'm not always successful, and if I'm honest, it's typically not enough FTE.

While working in government (NASA), the usual expectation by collaborators there was a measly 5% FTE, but to be fair, NASA funds are very limited, and I was on a hard-funded line anyway, so the 5% FTE was more of a "retainer" of sorts so that the PI would know that she had statistical collaboration lined up.  Still, that meant that adding up all of the different grants that I supported put my true effort beyond what I could handle when crunch time hit, and all of my PI's were hoping to present their interim data at the same annual conferences.

In academia, I can typically get 10% FTE covered if I make a strong case on R-level grants, though one PI recently dropped my effort from a grant  on re submission that had a solid review but one of the reviewers questioned why there would need to be a statistician at 10% FTE.  Interestingly, there were no criticisms of any of my contributions on this grant...  This this was a proposal involving multiple  animal study with 20 planned experiments.  Somehow this reviewer figured that analyzing data collected from animals somehow represented less work than analyzing data that comes from humans.  Not sure I follow that logic, but the PI was a well-funded PI for many years and had just recently begun using a statistician because his studies were getting too complicated for what he was trained to handle.  He asked if I would accept 5% FTE on the re-submission, and as much as I really enjoy working with this team, I declined because it was just too much work for that level of effort.  The grant did not get funded on resubmit.

In all of my negotiations, I'm most successful when I present the PI with the bottom line.  FTE is an odd currency, even for seasoned researchers,  so I find it useful to break it down into hours per week or month.

5% FTE equates to about one day per month, or 2 hours per week.
10% FTE equates to two days per month, or 4 hours per week.
20% FTE equates to 4 days per month, or 8 hours per week (2-hours per day for 4 days)

Will the grant have weekly study meetings?  (1 hour per week).
Will the grant require statistician/pharmacy interactions for blinded drug/placebo assignments?  (1-2 hours per week)
Will the grant require consulting with data managers?  (mystery per week)
Will the grant require interim reports for funding agencies (many hours at crunch time)
Will the PI wish to report interim data at annual conferences (many hours at crunch time)
How much time will the final manuscripts take, including analysis, graphics, writing, responding to reviewers... (many hours)

The bottom line is that on any given study, weeks may go by with no statistical work to be handled, but we negotiate our %FTE so that we can cover the entirety of the work to be conducted, including those crunch times when our work will be intense. It's a very difficult thing to estimate for me, so I imagine PI's can't possibly understand the logic.   The only other option is to budget a dynamic FTE percent based on when we think our effort will be used, weighing heavy effort when the data are to be analyzed for interim reports, and heaviest at the end of study.  The difficult there is exactly how to negotiate that: 80% FTE for 6 months doesn't sound appealing to PI's, though that may actually be a reasonable estimate.  Also... funds tend to run out towards the end of the grant, so the money may not be there anyway.

And then, of course, the there are the non-R-level grants that either don't support collaborative %FTE, or have very tight limits on what they will support.  Those are necessary precursors to the R01, and they really must be supported somehow.  So we just do it.  We eat the FTE in order to gain collaborative experience and get our junior faculty the experience and pilot data they need in order to compete successfully for that future R01.











------------------------------
Robert Ploutz-Snyder,PhD, PStat(r)
Research Professor & Director
Applied Biostatistics Laboratory
University of Michigan-School of Nursing

Original Message:
Sent: 05-10-2018 18:28
From: Stephen Simon
Subject: What percentage effort is normal for statisticians on research grants

As a follow-up to an message I sent out several weeks ago, I developed a draft policy on research grants. It stated

1. the normal support level should be 20%, higher for complex projects, lower if the PI does his/her own data analysis, but never below 10%.
2. we need at least a full month lead time, ideally two months, if you want us to work on the grant.
3. we will not work on a grant that does not budget for a data manager.
4. you cannot change our percentage effort after the fact without renegotiating the scope of work.

I have an early draft at PMean: Draft policy on statistical support for research. It needs a lot of polishing, but the general goal is to set some minimum expectations on our end.

I'm getting a lot of push back on the first item. One person wrote "efforts of 20% or higher for statistical support are very rare." Another said 20% support is mostly "if the stats person is the PI and the project is a stats-centric project."

What's the experience of others? If you've served on review panels, do you see us as being doomed to a life of 5% effort here and 10% effort there? if you are on grants, do you ever get anything close to 20% support without being the PI? If you routinely get 5% to 10% support does that really cover the amount of work that you do?

------------------------------
Stephen Simon, blog.pmean.com
Independent Statistical Consultant
P. Mean Consulting
------------------------------

Dear ASA Colleagues:

My perspective is from an MS level statistician and programmer/analyst.  My roles in CDC and NIH grants have been to set up the project database, either online or a secured network database, import all data into a statistical analysis package, such as SAS or Stata, analyze the data for publications, and maintain the project websites.  Based on my experience, ~50% effort per project works well for a data analyst / data manager.

------------------------------
Brandy Sinco, BS, MA, MS
Statistician and Programmer/Analyst
------------------------------

Original Message:
Sent: 05-25-2018 16:13
From: Douglas Landsittel
Subject: What percentage effort is normal for statisticians on research grants

This is a very interesting string of discussion. Most of what I feel (about the complexity of projects and the need for far more than 22 hours) has already been said, but I'll add a few notes that I hope are new to the discussion.

Although in many ways, I like the idea of setting some guidelines, I am not in favor of it for myself; however, I can see where the need arises, especially in a service unit.

First, and most importantly, the problem is not really so much time as it is the mental bandwidth one has for collaboration across 30+ projects that could come through your door at any time that also cover different disciplines (none of which I have any formal training on).

So for instance, when I was in our Biostatistics Facility in the Cancer Center, we had a small group to handle a large number of investigators and were somewhat obligated to take projects as they arose. At one point I had 30% on the cancer center support grant, where I worked with about 20-30 different investigators (some of whom were very needy - those projects alone could have filled my time with no trouble) and then had the other 70% time was spread over 17 different grants. In that situation, some rules for our group to follow as were proposed at the beginning of this email string would have been great.

As another example where such rules are useful/necessary, in my previous position in medicine, I started as 50% on our CTSA biostat core - where anyone of the hundreds of research faculty in the health science at Pitt could come through my door at any time - and the other 50% of my time was funded by a data center that also serviced a huge number of investigators. It was awful! I worked with >200 different investigators in my first 2 years. I threatened to quit on a regular basis until I got my own section, out of the data center, and worked to get PI roles on 3 different projects.

Now, in a department of biomedical informatics, I fund about 30% of my time on a training/education grant as the PI, another 10% on running a data coordinating center (which should be more) and then am very picky on what else I will take. My other projects are largely a result of the connections I make as the director of biostatistics (for research) for a transplant institute, a member of our Comparative Effectiveness Research Center, and a few select collaborators on biomarker and physical activity studies in rheumatology, nephrology and hematology.

Now I think in terms of % effort for my staff to do the work and what % effort I can possibly spare and still stay at 80% grant funded (with 5% covered on non-grant activities by my department for teaching and the two above-mentioned institutes/centers).

So, in my current situation, the rules stated initially are of no interest to me, but I totally get the need to consider that in other cases. Otherwise - if you are in a largely grant-funded research setting - things get out of hand very quickly. It's a really bad feeling when someone walks in your door saying "hey if I could just have a second of your time to answer some questions about that project we worked on last year" and 1) you remember them but can't remember for the life of you what their project is about, and 2) you can bet that their question is going to take a hell of a lot more than a few seconds, and you've got 20 other projects piled up and due that day.

Finally, I think times are changing in terms of % effort being questioned by study sections. I just finished one 4-year appointment and am starting another 4-year appointment now (and have done >50 ad hoc reviews). My experience is that - if it's a complex epi study - you want at least 10-20% for the statistician as a co-investigator + support staff. It's been a long time since I've heard anyone criticize that but I see people get criticized all the time for complex studies where the statistician is a consultant and/or <10%, especially for clinical trials. Basic science and animal studies are different.

Thanks to all for the interesting discussion.

------------------------------
Douglas Landsittel
Professor of Biomedical Informatics
University of Pittsburgh

Original Message:
Sent: 05-15-2018 08:52
From: Robert Ploutz-Snyder
Subject: What percentage effort is normal for statisticians on research grants

Indeed, this is a vexing issue for me too.  I've been working as a Biostatistician for nearly 20 years now in two Universities (Medicine, Nursing) and Government (NASA Human Research).  Each position was much the same--with a primary mission of collaborating with content-expert PI's on grants and the scholarship that leads up or comes from them funded research programs.  Team science has always been my passion.

I have gained confidence over the years, and have had most success when I'm upfront with PI's at our initial consults.  I do not have a fixed policy on %FTE because some research involves more/less quantitative work.  Some require sophisticated randomization schemes and monitoring, some require DSMB's, many require mid-stream analytics, etc..  Having said that, I do go into most collaborations expecting 10% FTE, mainly out of necessity. I'm not always successful, and if I'm honest, it's typically not enough FTE.

While working in government (NASA), the usual expectation by collaborators there was a measly 5% FTE, but to be fair, NASA funds are very limited, and I was on a hard-funded line anyway, so the 5% FTE was more of a "retainer" of sorts so that the PI would know that she had statistical collaboration lined up.  Still, that meant that adding up all of the different grants that I supported put my true effort beyond what I could handle when crunch time hit, and all of my PI's were hoping to present their interim data at the same annual conferences.

In academia, I can typically get 10% FTE covered if I make a strong case on R-level grants, though one PI recently dropped my effort from a grant  on re submission that had a solid review but one of the reviewers questioned why there would need to be a statistician at 10% FTE.  Interestingly, there were no criticisms of any of my contributions on this grant...  This this was a proposal involving multiple  animal study with 20 planned experiments.  Somehow this reviewer figured that analyzing data collected from animals somehow represented less work than analyzing data that comes from humans.  Not sure I follow that logic, but the PI was a well-funded PI for many years and had just recently begun using a statistician because his studies were getting too complicated for what he was trained to handle.  He asked if I would accept 5% FTE on the re-submission, and as much as I really enjoy working with this team, I declined because it was just too much work for that level of effort.  The grant did not get funded on resubmit.

In all of my negotiations, I'm most successful when I present the PI with the bottom line.  FTE is an odd currency, even for seasoned researchers,  so I find it useful to break it down into hours per week or month.

5% FTE equates to about one day per month, or 2 hours per week.
10% FTE equates to two days per month, or 4 hours per week.
20% FTE equates to 4 days per month, or 8 hours per week (2-hours per day for 4 days)

Will the grant have weekly study meetings?  (1 hour per week).
Will the grant require statistician/pharmacy interactions for blinded drug/placebo assignments?  (1-2 hours per week)
Will the grant require consulting with data managers?  (mystery per week)
Will the grant require interim reports for funding agencies (many hours at crunch time)
Will the PI wish to report interim data at annual conferences (many hours at crunch time)
How much time will the final manuscripts take, including analysis, graphics, writing, responding to reviewers... (many hours)

The bottom line is that on any given study, weeks may go by with no statistical work to be handled, but we negotiate our %FTE so that we can cover the entirety of the work to be conducted, including those crunch times when our work will be intense. It's a very difficult thing to estimate for me, so I imagine PI's can't possibly understand the logic.   The only other option is to budget a dynamic FTE percent based on when we think our effort will be used, weighing heavy effort when the data are to be analyzed for interim reports, and heaviest at the end of study.  The difficult there is exactly how to negotiate that: 80% FTE for 6 months doesn't sound appealing to PI's, though that may actually be a reasonable estimate.  Also... funds tend to run out towards the end of the grant, so the money may not be there anyway.

And then, of course, the there are the non-R-level grants that either don't support collaborative %FTE, or have very tight limits on what they will support.  Those are necessary precursors to the R01, and they really must be supported somehow.  So we just do it.  We eat the FTE in order to gain collaborative experience and get our junior faculty the experience and pilot data they need in order to compete successfully for that future R01.











------------------------------
Robert Ploutz-Snyder,PhD, PStat(r)
Research Professor & Director
Applied Biostatistics Laboratory
University of Michigan-School of Nursing

Original Message:
Sent: 05-10-2018 18:28
From: Stephen Simon
Subject: What percentage effort is normal for statisticians on research grants

As a follow-up to an message I sent out several weeks ago, I developed a draft policy on research grants. It stated

1. the normal support level should be 20%, higher for complex projects, lower if the PI does his/her own data analysis, but never below 10%.
2. we need at least a full month lead time, ideally two months, if you want us to work on the grant.
3. we will not work on a grant that does not budget for a data manager.
4. you cannot change our percentage effort after the fact without renegotiating the scope of work.

I have an early draft at PMean: Draft policy on statistical support for research. It needs a lot of polishing, but the general goal is to set some minimum expectations on our end.

I'm getting a lot of push back on the first item. One person wrote "efforts of 20% or higher for statistical support are very rare." Another said 20% support is mostly "if the stats person is the PI and the project is a stats-centric project."

What's the experience of others? If you've served on review panels, do you see us as being doomed to a life of 5% effort here and 10% effort there? if you are on grants, do you ever get anything close to 20% support without being the PI? If you routinely get 5% to 10% support does that really cover the amount of work that you do?

------------------------------
Stephen Simon, blog.pmean.com
Independent Statistical Consultant
P. Mean Consulting
------------------------------