ASA Connect

Erick,

If I understand your question, what are trying to do is commonly referred to as a "post-hoc power analysis".  Googling that term will bring up a host of literature, but you may want to get started with this article that is quite accessible:

John M Hoenig , Dennis M Heisey
The American Statistician
Vol. 55, Iss. 1, 2001

http://www.tandfonline.com/doi/abs/10.1198/000313001300339897

Bottom line is that you would probably be better served by looking at confidence intervals and measures of effect size.

Hope this helps.

Bob

--  ?????????????? ??. ??????????????????, ?????????????????????? ??????????????????, ??????, ???????? ???????????????????? 1, ?????????????????? 6???? ????????????, ?????????????? 611, ??????. 210.368.9491  Yiannis C. Bassiakos Associate Professor, DoE, UoA 1 Sofokleous street, 15509, Athens, GREECE tel. +30210.368.9491 mob. +30697.419.2133 Skype: yiannis.c.bassiakos

I agree with the comments made by Yiannis, Bob, and Jonathan. There are lots of potential problems and pitfalls involved with power; feel free to look at pp. 14 - 45 of http://www.ma.utexas.edu/users/mks/CommonMistakes2016/SSISlidesDayThree2016.pdf for more extensive discussion. You might in particular consider looking at the Type M and Type S calculations recommended in Gelman and Carlin's 2014 paper Beyond Power Calculations ( http://www.stat.columbia.edu/~gelman/research/published/retropower_final.pdf) -- these are OK to use retrospectively.

Well another approach is to to fix power at a certain level, say 80% and solve for the detectable effect given the available sample size and a desired significance level. But there's a problem, and it's the significance level. If only one hypothesis is going to be tested then you could use the traditional alpha=0.05. But with many hypotheses and many looks at the data, which is what often happens in secondary data analyses,  a marginal alpha of 0.05 (i.e., 'p<.05') doesn't work well. What to do? one way would be to make the significance level more stringent, based on the number of tests that will be conducted, perhaps using some adjustment like Bonferroni, or FDR-based (which would require some assumptions).

As Bob Gerzoff mentioned, you would probably be better served by looking at confidence intervals and measures of effect size. However, with many hypotheses, the traditional 95% marginal confidence intervals have the same problem as 'p<0.05'. They fail to account for uncertainty due to multiplicity. A relatively simple approach to deal with this is to estimate simultaneous confidence intervals using an FDR approach. See the 2005 paper in JASA by Benjamini and Yekutieli 'False Discovery Rate-Adjusted Multiple Confidence Intervals for Selected Parameters'. 

http://www.math.tau.ac.il/~yekutiel/papers/JASA%20FCR%20prints.pdf

There are two types of testing in New Drug Application (NDA): parallel designs and cross-over or Latin square designs.  The latter is generally used as a two-period cross-over design with a reference (R) and test (T) materials.  In all cases one needs to compute power and/or confidence intervals.  At the design stage, one needs to compute sample sizes to achieve power (>=80% or so) using global estimates of variances )from published literature or elsewhere).  In the post hoc stage, one needs to compute power from the acquired data and establish an acceptable range of confidence intervals. The variance components at this stage are obtained from the observed data.  In other words, for NDAs there is need for both global and observed variances and powers from both of them based on the design and post hoc analysis steps.  The US FDA has a gold standard for these studies: Statistical Approaches to Establishing Bioequivalence, US Department of Health and Human Services, US FDA, Center for Drug Evaluation and Research (CDER), January 2001, Rockville, MD.

Ajit K. Thakur, Ph.D.

Retired Statistician