ASA Connect

In anticipating a small randomized, controlled exploratory clinical study (using topical essential oil as an adjunct treatment to modulate postprandial glycemia)  we desire to set a more relaxed level of significance with an alpha of 0.10 for our power calculation, however the IRB wants a reference. Because of cost constraints and a limited study population, we are interested in keeping the study population as limited as possible and determining a therapeutic effect size that can be used in a larger trial can be planned. Is this a justifiable position and is there a reference that would satisfy the IRB?

------------------------------
JR Gates
------------------------------

See ICH E9 Statistical Principles Section 3.5 Sample Size

"the following items should be specified: A primary variable; the test statistic; the null hypothesis; the alternative (``working'') hypothesis at the chosen dose(s) (embodying consideration of the treatment difference to be detected or rejected at the dose and in the subject population selected); the probability of erroneously rejecting the null hypothesis (the Type I error) and the probability of erroneously failing to reject the null hypothesis (the Type II error); as well as the approach to dealing with treatment withdrawals and protocol violations. In some instances, the event rate is of primary interest for evaluating power, and assumptions should be made to extrapolate from the required number of events to the eventual sample size for the trial."

and

"Alternative values to the conventional levels of Type I and Type II error may be acceptable or even preferable in some cases."

------------------------------
Donald McMahon
Director of Statistics & Data Management
Columbia University Medical Center
------------------------------

Jeff Gates,   
Regarding this issue of using alpha of 0.10 for an exploratory study/power analysis, is perhaps "false discovery rate" (see Benjamini & Hochberg, etc.) relevant?   Also there was a psychological researcher in the 1960s & 70s who was (in)famous for using and justifying an alpha of 0.1 in his research, but I can't remember his name.  You might check the internet. 
------------------------------
Joseph Locascio
------------------------------

Hi,

If this is a pilot study, where the data collected will be used to guide a future "properly powered" study, why not consider a sample size that will yield a reasonable level of precision in your estimated effect size via confidence intervals?

Examples:

Considerations in determining sample size for pilot studies

MA Herzog

Res Nurs Health. 2008 Apr;31(2):180-91

http://www.ncbi.nlm.nih.gov/pubmed/18183564

Sample size calculations for pilot randomized trials: a confidence interval approach

Kim Cocks, David J. Torgerson

Doing a Google search for "pilot study sample size" will yield a variety of references with differing approaches.

Regards

------------------------------
Marc Schwartz
------------------------------

There are lots of technical approaches to an exploratory study that use something different than an alpha level of 0.05, or something different than an alpha level entirely. But these would not be worth citing to the IRB.

What you need to show the IRB is that standard statistical references allow for other alpha levels, especially alpha levels of 0.10 and 0.01. One such source is the Engineering Statistics Handbook, which in section 1.3.5, states that "The choice of α is somewhat arbitrary, although in practice values of 0.1, 0.05, and 0.01 are commonly used." See 1.3.5. Quantitative Techniques.

I'm being lazy and citing electronic sources. A quick search of Google books provides an extensive discussion of alpha levels. One book, Understanding Statistics in the Behavioral Sciences (page 257) says "If, however, the experiment is exploratory in nauture and the results are to guide the researcher in deciding whether to do a full-fledged experiment, it would be foolish to use such stringent levels. In such cases, alpha levels as high as 0.10 or 0.20 are often used".

You can probably find a similar statement in a book oriented more towards medical statistics, but I can't seem to find one easily right now. The key is to convince the IRB that alpha=0.10 is a "common" choice.

------------------------------
Stephen Simon
Independent Statistical Consultant
P. Mean Consulting
------------------------------

See O’Brien, R. G. and Castelloe, J. (2007). Pharmaceutical Statistics Using SAS: A Practical Guide, chapter Sample-Size Analysis for Traditional Hypothesis Testing: Concepts and Issues, pages 237–271. SAS Press.

The first part of that chapter addresses your question head-on, but other parts relate firmly as well. Section 10.3.1 uses an example with alpha = 0.20. 

Basically, if the sample size is small, then with alpha=0.05, the Type II error rates will be high, so BOTH "significant" and "non-significant" findings will be untrustworthy--will have high "crucial" error rates (as defined in the chapter). Increasing alpha will bring down the Type II rate a little, but unless the effect of interest is huge, nothing can make an "exploratory" study solid WRT inference.

Power analysis, especially for small studies conducted early in the "March of Science", is almost always just statistical gamesmanship. Besides that, it's asking questions only about some eventual p-value. Science will improve the more we discourage the use and reporting of pedestrian p-values (99% of them) and, thus, of pedestrian power analyses.

What I teach/preach now is that we should be seriously guesstimating (a real word) how the eventual key confidence/credible intervals might turn out. How might those intervals--even those that are not significant because they include the null point or null region--help the investigators plan the next leg of their March? And, you can use 90% confidence levels, which most people think is a lot better than using alpha = 0.10. Yes, this is statistical slight of hand, but it works almost every time with IRBs and such!

------------------------------
Ralph O'Brien
Professor of Biostatistics (officially retired; still keenly active)
Case Western Reserve University
------------------------------

To minimize the maximal expected risk (Neyman-Pearson), the ratio of alpha to beta should be the inverse of the ratio of type-I to type-II risks. Hence, If type-II errors are 2x more severe than type-I errors (as one might argue in a - non-aviation - "pilot" study), one might convincingly argue, e.g., for level/power of .20/.90. This get's you about the same sample size as the "conventional" (aka unjustified) .05/80 ratio.

------------------------------
Knut Wittkowski
Head, Dept. Biostatistics, Epidemiology, and Research Design
The Rockefeller University
------------------------------

Cost is a VERY significant factor at many companies (e.g. those developing jet engines). Hence, early in the development program testing(where a single test is >>$2M), an alpha level of 0.2 is used. In addition to the very nicely stated statistical observations already made, the cost of possibly discarding an important variable or interaction can cost you much more money later.

------------------------------
James Breneman
------------------------------

(1) Look up the following reference: Rubinstein LV, Korn EL, et al. J Clin Oncol. 2005 Oct 1;23(28):7199-206. PMID:16192604

(2) Ask Ralph O'Brien of Case Western Reserve University. He ought to have numerous additional references.

(3) Your situation sounds like one in which estimation might be more suitable than hypothesis-testing.

------------------------------
Eric Siegel
Biostatistician
Univ of Arkansas for Medical Sciences of Biostatistics
------------------------------