Discussion: View Thread

I'm in the middle of a situation where it looks (at least to me) that an Institutional Review Board (IRB) is micromanaging the details of a clinical study. They asked the principal investigator (PI) the following question:

"The IRB would like to know, how you set the parameters for the power calculation, such as effect size, alpha level. For effect size, you need to have some data to justify or should choose a conservative one. And also which statistical test was the calculation based on."

The PI sent the comments to me, of course.

I wrote a long response about how a Type I error rate of 5% and a Type II error rate of (about) 10% are pretty much standard in the research community. I pointed out the problem with using "effect sizes" and that the sample size justification needs to be based on the minimum clinically important difference instead. I also noted that an IRB demanding a conservative value represents a one-sided perspective on the issue because sample sizes that are too large raise just as many ethical issues as sample sizes that are too small.

So now the IRB asks the question "Please indicate whether or not the PI is confident to observe a difference of at least 15 on the XXXX between two groups and a difference of at least 25 on the YYYY between two groups."

The original power calculation suggested a sample of 30 patients total (15 per group) based on the power and sample size calculator for the independent samples t-test of Russell V. Lenth. Here is what we wrote in the original protocol: "If the standard deviation for the XXXX score is 11, then we would have 89% power for detecting a shift of 15. If the standard deviation of YYYY is 18, then we would have 90% power for detecting a difference of 25."

Now at this point, I'm tempted to say, "Who the heck do you think you are to question the PIs judgment about what is likely to be seen in a study like this?" Instead, I am just suggesting that the PI respond "Differences of this size have been seen in similar research studies, but we won't know for sure in our setting until we do the research." after making sure, of course, that there are indeed a couple of studies that actually do have differences of 15 and 25 on the two outcome measures.

But this is bothering me a lot. I'm at a loss as for why the IRB is doing this. In the hundreds of studies that I've helped get through the IRB, I have never had anyone question the sample size justification in such detail. Are they upset because the effect size is so much larger than 0.8? Are they convinced that a sample of 30 patients total is guaranteed to be underpowered?

I'm curious if anyone else has had a similar encounter with an IRB. Any advice on how to handle this would also be appreciated.

-------------------------------------------
Stephen Simon
Independent Statistical Consultant
P. Mean Consulting
-------------------------------------------

Steve,

Have you ever worked with this IRB before?  And more specifically, have you ever worked with this IRB-PI combination before?

My suspicion is that you are running into a political buzz saw where the IRB wants the PI to fail.  It's not about science, it's about power (and not the power of the test!).

-------------------------------------------
David Mangen
Owner
Mangen Research Associates, Inc.
-------------------------------------------

Hi Steve,

Another way to read this situation is that the IRB would like to make sure the PI gave sufficient consideration to the various assumptions required by the sample size calculation and how likely it is that they would hold for the study at hand.  Sample size calculations are often made on the basis of fairly strong assumptions.   

You could ask the PI to think about a "worst case scenario" and "best case scenario" in terms of sizes of standard deviations, of differences in means, etc.  Then your sample size calculation could take these into account and aim for perhaps a middle ground.  A small simulation study could be set up to investigate what happens as you relax various assumptions.  This might be more objective than simply asking the PI to vouch that they are confident that the results of their study will come out a certain way.  He may have some expectation of what the outcome might look like, but that expectation could be highly subjective.

If you take the emotion out of the situation and try to think about the problem from the IRB's perspective, it will be easier to understand what it is they are looking for. 

Good luck!

Isabella


-------------------------------------------
Isabella Ghement
Ghement Statistical Consulting Company Ltd.
-------------------------------------------







Original Message:
Sent: 05-09-2014 16:30
From: David Mangen
Subject: IRB questioning the details of a power calculation

Steve,

Have you ever worked with this IRB before?  And more specifically, have you ever worked with this IRB-PI combination before?

My suspicion is that you are running into a political buzz saw where the IRB wants the PI to fail.  It's not about science, it's about power (and not the power of the test!).

-------------------------------------------
David Mangen
Owner
Mangen Research Associates, Inc.
-------------------------------------------

-------------------------------------------
Raymond Hoffmann
Professor
Medical College of Wisconsin
-------------------------------------------


An interesting comment about the IRB.  It might be that they have someone who fancies themselves as a statistician.
   But if I was on that IRB, I would question whether the power calculation made any sense.  If you use Cohen's criteria, you are using a giant effect size to claim you have 90% power. I don't find that believable.  I'd suggest scaling it down to 80%, but you are right to suggest that the PI needs papers for a reference.  Otherwise, I think this is vastly underpowered and indicates a PI who hasn't done much, if any, research before.  The sample sizes you are suggesting are more of a pilot study sample size to get an idea of what the real effect was for future work. 
   I don't think the IRB is communicating the real problem with the PI and you.  But I would certainly reject this study, especially if there was any risk at all of an adverse event.  Or if this is a me too study of something that has been done before on a much larger sample.

The context of the study is everything and the first thing I learned was that the PI is not always right when it comes to planning a study.
(I deal with a lot of beginning researchers that need a lot of mentoring in how to design a study and ask pertinent questions).
Ray



Original Message:
Sent: 05-09-2014 16:30
From: David Mangen
Subject: IRB questioning the details of a power calculation

Steve,

Have you ever worked with this IRB before?  And more specifically, have you ever worked with this IRB-PI combination before?

My suspicion is that you are running into a political buzz saw where the IRB wants the PI to fail.  It's not about science, it's about power (and not the power of the test!).

-------------------------------------------
David Mangen
Owner
Mangen Research Associates, Inc.
-------------------------------------------

Steve, you're certain to get a lot of comments. My sheer speculation - there is a statistician on the IRB.

It would be helpful if you can give details about the study. For example is this a "first in human" or "first ever" study?
If yours was a Phase III study, then all those questions and a few others would have likely been answered in the statistical methods of the protocol.

As to the investigator "confidence" - has the investigator done research of any kind with these patients?

Some investigators can say something like "Yes, I've seen <nn>  patients with this and I've noted that it is important to the patients to have about a <zz>  point change in this measure to feel benefit from the procedure".

-------------------------------------------
Chris Barker, Ph.D.
Consultant and
Adjunct Associate Professor of Biostatistics
www,barkerstats.com

---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
-------------------------------------------

Chris's comment is funny to me because my first thought was that there WASN'T a statistician on the IRB.  Instead, you may be playing host to one of a whole slew of wannabees from a wide variety of disciplines who 'took a lot of stats in grad school'.  I'll stop there as to keep such individuals who read these posts from blowing up the feed on a Friday afternoon.

My suggestion is to just answer their questions as best you can in a concise way and try to move on.  It is likely NOT the entire IRB who is slowing things down.  If you get dragged down into a huge back and forth then there are places where you can file complaints.  You don't complain about the unusual requests they have but in the amount of time spent in such discussions.  Many IRB panels are criticized for taking too long and this is very counter productive to faculty research and output, if you have made reasonable attempts to answer their questions and still it takes too long then someone should be told that the research is being held up.

-------------------------------------------
Jason Brinkley
East Carolina University
-------------------------------------------







Original Message:
Sent: 05-09-2014 16:42
From: Chris Barker
Subject: IRB questioning the details of a power calculation

Steve, you're certain to get a lot of comments. My sheer speculation - there is a statistician on the IRB.

It would be helpful if you can give details about the study. For example is this a "first in human" or "first ever" study?
If yours was a Phase III study, then all those questions and a few others would have likely been answered in the statistical methods of the protocol.

As to the investigator "confidence" - has the investigator done research of any kind with these patients?

Some investigators can say something like "Yes, I've seen <nn>  patients with this and I've noted that it is important to the patients to have about a <zz>  point change in this measure to feel benefit from the procedure".

-------------------------------------------
Chris Barker, Ph.D.
Consultant and
Adjunct Associate Professor of Biostatistics
www,barkerstats.com

---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
-------------------------------------------

Steve, It is interesting the scrutiny that this has.  I'll weigh in on what may be on their mind.  

They are approving experimental treatment usage in the perspective that the research leads to scientific progression -- "For the sake of science."  If the trial is too small, that it has no chance of providing scientific progression then this is an ethical issue for the IRB.  Maybe he/she looked at these effect sizes and says: "these are impossible effect sizes" within this disease.... now he/she is worried that subjects are agreeing to contribute to science -- when there is no chance of learning.

Now we can have a discussion of whether statistical significance is learning and whether there is advancement here.  But addressing the question of "whether these subjects will contribute to science is an important one."

An effect size of 0.8 in medical applications is big -- is this possible?  Having the clinician weigh in or empirical evidence of such a possibility will ease the IRB members concern that this is an exercise in futility and doesn't benefit anyone (then why should the approve such a study for human entry?). 

So, it seems to me the query isn't that unreasonable -- even if unusual in the business!

--Scott


-------------------------------------------
Scott Berry
Berry Consultants
-------------------------------------------

While I can understand your frustrations, at the end of the day the IRB is the gatekeeper for this research proposal
and their concerns need to be addressed in some manner, regardless of how reasonable they may or may not be.

Since their main concern is 'what will happen if these parameters are different', it might be useful to produce a power curve figure.
For example, for a fixed per-group sample size there's a 1-to-1 relationship between the effect size and the resulting power, given alpha.
Similarly, for a fixed effect size there's a 1-to-1 relationship between the sample size and power.

If you made a power curve or two, it might head off a lot of 'what if' rebuttals/queries from the IRB.

-------------------------------------------
John Dawson
Postdoctoral Scholar
University of Alabama at Birmingham
-------------------------------------------

Stephen:

The idea of the power curve is a good one - perhaps also showing what effects have been seen in similar studies. 

In case it makes you feel any better,  when I worked in large pharma,  we often got this type of response from IRB's, especially for our Phase II studies.  The attitude seemed to be "you have plenty of money,  you should make the study larger".  My philosophy was to just keep providing more information to them and leave my annoyance out of it.  Eventually, they have to move on to other trials and you win the argument by default. 


-------------------------------------------
Roy Tamura
Associate Professor
University of South Florida
-------------------------------------------







Original Message:
Sent: 05-09-2014 16:48
From: John Dawson
Subject: IRB questioning the details of a power calculation

While I can understand your frustrations, at the end of the day the IRB is the gatekeeper for this research proposal
and their concerns need to be addressed in some manner, regardless of how reasonable they may or may not be.

Since their main concern is 'what will happen if these parameters are different', it might be useful to produce a power curve figure.
For example, for a fixed per-group sample size there's a 1-to-1 relationship between the effect size and the resulting power, given alpha.
Similarly, for a fixed effect size there's a 1-to-1 relationship between the sample size and power.

If you made a power curve or two, it might head off a lot of 'what if' rebuttals/queries from the IRB.

-------------------------------------------
John Dawson
Postdoctoral Scholar
University of Alabama at Birmingham
-------------------------------------------

Dear Stephen,

I understand your frustration, but you may be able to resolve the problem by providing them what they ask for. 

The protocol states: "If the standard deviation for the XXXX score is 11, then we would have 89% power for detecting a shift of 15. If the standard deviation of YYYY is 18, then we would have 90% power for detecting a difference of 25."

The IRB would like to know, "how you set the parameters for the power calculation, such as effect size, alpha level. For effect size, you need to have some data to justify or should choose a conservative one. And also which statistical test was the calculation based on."

Have you considered asking the PI why a difference less than 15 or 25 would not be worth detecting with 90% power at the stated level of 5% (Fisher 1925)? Could you give a justification for alpha/beta that refers to the risks of type I and type II errors? Is there any literature supporting the estimates of 11 or 18 for the standard deviation? 

Best
Knut
-------------------------------------------
Knut M. Wittkowski
Head, Dept. Biostatistics, Epidemiology, and Research Design
The Rockefeller University
-------------------------------------------




Original Message:
Sent: 05-09-2014 16:20
From: Stephen Simon
Subject: IRB questioning the details of a power calculation

I'm in the middle of a situation where it looks (at least to me) that an Institutional Review Board (IRB) is micromanaging the details of a clinical study. They asked the principal investigator (PI) the following question:

"The IRB would like to know, how you set the parameters for the power calculation, such as effect size, alpha level. For effect size, you need to have some data to justify or should choose a conservative one. And also which statistical test was the calculation based on."

The PI sent the comments to me, of course.

I wrote a long response about how a Type I error rate of 5% and a Type II error rate of (about) 10% are pretty much standard in the research community. I pointed out the problem with using "effect sizes" and that the sample size justification needs to be based on the minimum clinically important difference instead. I also noted that an IRB demanding a conservative value represents a one-sided perspective on the issue because sample sizes that are too large raise just as many ethical issues as sample sizes that are too small.

So now the IRB asks the question "Please indicate whether or not the PI is confident to observe a difference of at least 15 on the XXXX between two groups and a difference of at least 25 on the YYYY between two groups."

The original power calculation suggested a sample of 30 patients total (15 per group) based on the power and sample size calculator for the independent samples t-test of Russell V. Lenth. Here is what we wrote in the original protocol: "If the standard deviation for the XXXX score is 11, then we would have 89% power for detecting a shift of 15. If the standard deviation of YYYY is 18, then we would have 90% power for detecting a difference of 25."

Now at this point, I'm tempted to say, "Who the heck do you think you are to question the PIs judgment about what is likely to be seen in a study like this?" Instead, I am just suggesting that the PI respond "Differences of this size have been seen in similar research studies, but we won't know for sure in our setting until we do the research." after making sure, of course, that there are indeed a couple of studies that actually do have differences of 15 and 25 on the two outcome measures.

But this is bothering me a lot. I'm at a loss as for why the IRB is doing this. In the hundreds of studies that I've helped get through the IRB, I have never had anyone question the sample size justification in such detail. Are they upset because the effect size is so much larger than 0.8? Are they convinced that a sample of 30 patients total is guaranteed to be underpowered?

I'm curious if anyone else has had a similar encounter with an IRB. Any advice on how to handle this would also be appreciated.

-------------------------------------------
Stephen Simon
Independent Statistical Consultant
P. Mean Consulting
-------------------------------------------

Huh.  All the IRB's queries seem reasonable to me...I would have thought the details of sample size estimations would always be included in the statistical methodology section of any research protocol.

The University of Texas M. D. Anderson Cancer Center's strict policy was that all protocols had to (1) be reviewed and approved by a biostatistician (after discussions with the researchers), then (2) he/she would present and review the protocol with a group of biostatisticians (without the researchers present).  Only after revisions and approval by the group of biostatisticians was the proposed protocol submitted to an IRB committee.

IRBs political?  Oh yeah...and we ran into a case of what I can only call "professional snobbery" with an IRB committee - but that's another posting...   :-)

-------------------------------------------
Wayne Fischer
Statistician
University of Texas Medical Branch
-------------------------------------------

Sample size recommendations are a difficult idea for many people.  I had 2 scientists sit at my desk and when I asked them what size effect they would like to detect, they answered 'a significant effect.'   So minimal detected effect, which I refer to as 'effect size'  (the .8 below I think refers to Cohen's D) is a very hard concept.   First of all the idea that you can assume a true value of the difference, or several true values, is confusing.  People think their work is original so any previous work is not relevant,  So talking about previous studies seems almost offensive, even when the literature is rich.     

Then the whole idea of type II error is very confusing.   So if you don't reject because the difference is too small, how do you know whether you are committing a type II error?    Isn't this observed difference the effect size, not the 25 originally specified in the power analysis?   Also as we all observe, declaring statistical significance seems to create an aura that separates from the original study design.

I don't know the context of this situation so I can't say why the IRB is acting this way.   I will say that power and sample size need to be explained in different ways to different audiences.     I am not sure in this case if a graph would help.    A table of different effect sizes, with n=30 and the assumed standard deviation, and their power level might help show what happens when you deviate from your assumptions 

-------------------------------------------
Georgette Asherman
-------------------------------------------