It sounds to me like you have two planned tests. First, you will test the interaction effect to determine if there exists an effect of the treatment that differs by disease type. Then, if no important interaction is found, you will test the main effect to determine if there exists an effect of the treatment that is not dependent on disease type. I would make two recommendations here:
1. You should not perform the second test if an interaction exists. Doing so can be very misleading. For example, suppose in an extreme case that the treatment is helpful to one group of patients but hurtful to the other. In that situation, the main effect test might well appear non-significant (but in fact there are effects in both groups). A more likely scenario is that the treatment is helpful to one group but not the other. This could give you the appearance of a significant "main effect" but that will again be misleading (because in the one group the treatment isn't helpful. If the interaction exists, that is what you should examine. (What you suggest at the end is in fact the wrong way to go about things.)
2. If you wish to control the overall false positive rate at 0.05, a simple way to do this would be a Bonferroni Correction, performing each of the two tests at significance level 0.025. If both tests are performed at significance level 0.05, then logically the overall false positive rate would have to be slightly higher.
I hope this is helpful, and best of luck with your study!
Best Regards,
Joe
------------------------------
Joseph Nolan
Associate Professor of Statistics
Director, Burkardt Consulting Center
Northern Kentucky University
Department of Mathematics & Statistics
Original Message:
Sent: 12-10-2016 12:24
From: Keith Goldfeld
Subject: Proper hypothesis testing procedure for subgroup analysis when we are interested in main effect as well
Greetings –
This is a really elemental question. We are planning a randomized trial that will be stratified by disease type (of which there are two) to compare the outcomes of two interventions. Our primary interest is the main effect of intervention 1 compared to intervention 2. But, we are pre-specifying a test to determine if the intervention effect differs between the two disease types, and want to make sure we have a large enough sample size to measure a smallish difference in effect sizes. The literature seems to suggest to fit an interaction model first (Y~b0 + b1*trt + b2*subgroup + b3*trt*subgroup) and test the hypothesis H0: b3 = 0 using a sig. level alpha=0.05. If you reject null, then you are done – you can conclude that there is an overall effect and report the effect sizes for both disease groups. However, if you fail to reject H0 but still want to assess if there is an overall effect, it seems like it is impossible to maintain FWER of 5%.
I really would like to maintain the FWER of 5% but still be able to do each test using alpha = 0.05. So, I thought of reversing the approach. First I would estimate a main effects model first (Y~a0+a1*trt) and test the null hypothesis H0: a1=0 with alpha = 0.05. If we fail to reject, we stop and do not say anything about the subgroups. However, if we reject the null, then we move on to the interaction model and test H0: b3=0, also at 5%. I've done simulations to confirm that this approach conserves the FWER of 5% - so it seems reasonable. But is it the wrong way to go about things?
Thoughts appreciated.
Keith Goldfeld, DrPH
Assistant Professor
School of Medicine, New York University
------------------------------------------------------------
This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is proprietary, confidential, and exempt from disclosure under applicable law. Any unauthorized review, use, disclosure, or distribution is prohibited. If you have received this email in error please notify the sender by return email and delete the original message. Please note, the recipient should check this email and any attachments for the presence of viruses. The organization accepts no liability for any damage caused by any virus transmitted by this email.
=================================