Discussion: View Thread

Hoping for some guidance:

I have a client with an already completed study (so no chance of "fixing" the strange design).  He has one sample that was assessed twice, once on medication and once off. He has a matched control sample that was just assessed once.  He is interested in whether the set of outcome measures (all continuous) differ among all three groups. He is insisting that a repeated-measures MANOVA is the right approach but I'm not convinced.  The data are transformed to be normal and nothing is missing.  Any thoughts?

Thanks,

-------------------------------------------
Lisa Spielman
Consultation Services
New City, NY
-------------------------------------------

Ignore him, he hasn't a clue what he is talking about.
It's better to walk away than to do a nonsensical analysis.
As far as what you should do, there isn't enough information about when the sample were collected. It is critical to know whether the off medication was before or after the medication (which implies potential carry over effects).  But it is clear that MVANOVA doesn't fit any situation that I can envision.

Ray

-------------------------------------------
Raymond Hoffmann
Professor
Medical College of Wisconsin
-------------------------------------------

Raymond is right - your client doesn't know what he/she is talking about.  The problem with the group that was sampled twice is their responses will be correlated and this correlation will need to be taken into account in the analysis.  However, the response from the control sample should be independent of the other group.  Assuming all other aspects of the study are valid and solid (big assumption) then I think the best you can do is two separate analyses.  First, you could propose using paired t-test on the group that was sampled twice to see if there was a difference between when they were on and off medication.  Second, it might be resonable to compare the control sample responses to one of the set of responses from the group that was sampled twice.  I emphasize "might" because more information is needed about the study to determine if this is appropriate.  I wouldn't however, walk away from the project at this point.  One of our roles as statisticians is to educate our clients and it sounds like you have an educational opportunity here. 

-------------------------------------------
John Brejda
Principal Statistician
Alpha Statistical Consulting
-------------------------------------------







Original Message:
Sent: 06-13-2013 14:55
From: Raymond Hoffmann
Subject: Analysis of unusual matched sample design

Ignore him, he hasn't a clue what he is talking about.
It's better to walk away than to do a nonsensical analysis.
As far as what you should do, there isn't enough information about when the sample were collected. It is critical to know whether the off medication was before or after the medication (which implies potential carry over effects).  But it is clear that MVANOVA doesn't fit any situation that I can envision.

Ray

-------------------------------------------
Raymond Hoffmann
Professor
Medical College of Wisconsin
-------------------------------------------

Brejda's suggestion has the virtue of being simple to execute and easy for editors/reviewers to understand. 

-------------------------------------------
Michael Kruger
Information Resources Inc
-------------------------------------------







Original Message:
Sent: 06-13-2013 16:38
From: John Brejda
Subject: Analysis of unusual matched sample design

Raymond is right - your client doesn't know what he/she is talking about.  The problem with the group that was sampled twice is their responses will be correlated and this correlation will need to be taken into account in the analysis.  However, the response from the control sample should be independent of the other group.  Assuming all other aspects of the study are valid and solid (big assumption) then I think the best you can do is two separate analyses.  First, you could propose using paired t-test on the group that was sampled twice to see if there was a difference between when they were on and off medication.  Second, it might be resonable to compare the control sample responses to one of the set of responses from the group that was sampled twice.  I emphasize "might" because more information is needed about the study to determine if this is appropriate.  I wouldn't however, walk away from the project at this point.  One of our roles as statisticians is to educate our clients and it sounds like you have an educational opportunity here. 

-------------------------------------------
John Brejda
Principal Statistician
Alpha Statistical Consulting
-------------------------------------------

Please clarify, By "one sample assessed twice" , is that only one subject and the other subjects assessed once?


-------------------------------------------
Chris Barker, Ph.D.
www,barkerstats.com

---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
-------------------------------------------

It is indeed a bizarre design!

To give you a little more info:  The group assessed twice were adult subjects diagnosed with ADHD and being treated with medication, they were tested once on their meds and week or so later off their meds.  The outcome is a measure of attention that has several indices.

They did not counterbalance the on/off medication factor (yes, I know....let me reiterate that I had nothing to do with the design).  There is reason to believe that the test does not have any practice effects but I would have chosen to counterbalance anyway to confirm, and I would have assessed the control group twice as well.  The hypothesis is that performance "off medication" is worse than "on medication", and that performance "off medication" is worse than matched controls.  Built into that is an assumption of no difference between normal controls and "on medication" - I would think it makes sense to do an equivalence test for that one.

My first thought was to split it all up, as Ryan mentioned.  And Ray, believe me, I would walk away if I could but I can't.

-------------------------------------------
Lisa Spielman
-------------------------------------------







Original Message:
Sent: 06-13-2013 14:56
From: Chris Barker
Subject: Analysis of unusual matched sample design


Please clarify, By "one sample assessed twice" , is that only one subject and the other subjects assessed once?


-------------------------------------------
Chris Barker, Ph.D.
www,barkerstats.com

---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
-------------------------------------------

A Challenging client.
Is this for a publication and you'd be listed as a co author?

Perhaps you can insist on writing the interpretation, an insist on listing all the caveats, or decline to be a coauthor or acknowledged.

Also I don't see that an equivalence test will help you, without a "pre specified margin".
(FDA has an extensive guidance on preparing analyses of "non inferiority"

-good luck
-------------------------------------------
Chris Barker, Ph.D.
Consultant and
Adjunct Associate Professor of Biostatistics
www,barkerstats.com

---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
-------------------------------------------







Original Message:
Sent: 06-13-2013 15:13
From: Lisa Spielman
Subject: Analysis of unusual matched sample design

It is indeed a bizarre design!

To give you a little more info:  The group assessed twice were adult subjects diagnosed with ADHD and being treated with medication, they were tested once on their meds and week or so later off their meds.  The outcome is a measure of attention that has several indices.

They did not counterbalance the on/off medication factor (yes, I know....let me reiterate that I had nothing to do with the design).  There is reason to believe that the test does not have any practice effects but I would have chosen to counterbalance anyway to confirm, and I would have assessed the control group twice as well.  The hypothesis is that performance "off medication" is worse than "on medication", and that performance "off medication" is worse than matched controls.  Built into that is an assumption of no difference between normal controls and "on medication" - I would think it makes sense to do an equivalence test for that one.

My first thought was to split it all up, as Ryan mentioned.  And Ray, believe me, I would walk away if I could but I can't.

-------------------------------------------
Lisa Spielman
-------------------------------------------

Lisa,

Since the experimental group was already on meds what he has done is a simple ex-post facto design. There is no way to show causation from this type of study. For the first hypothesis (that on meds is better than off meds for the experimental group) the samples are dependent so it would be a really basic repeated measures design one dependent variable (preformance) with one factor (meds coded for "on" and "off") and two time periods (pre and post). That is the only way that I can think of to handle that part of this design. The second comparison of the "off meds" group and the "no meds" control group could be done with a t-test, again assuming random assignment, etc. The "assumption" of equivalence without checking invalidates the design as an experiment anyway. There is no way to run all of this under any single statistic that I know.

You inherited a mess. Good luck.

~Dan
-------------------------------------------
Daniel Butorovich
Research Analyst
Cochise College
-------------------------------------------







Original Message:
Sent: 06-13-2013 15:13
From: Lisa Spielman
Subject: Analysis of unusual matched sample design

It is indeed a bizarre design!

To give you a little more info:  The group assessed twice were adult subjects diagnosed with ADHD and being treated with medication, they were tested once on their meds and week or so later off their meds.  The outcome is a measure of attention that has several indices.

They did not counterbalance the on/off medication factor (yes, I know....let me reiterate that I had nothing to do with the design).  There is reason to believe that the test does not have any practice effects but I would have chosen to counterbalance anyway to confirm, and I would have assessed the control group twice as well.  The hypothesis is that performance "off medication" is worse than "on medication", and that performance "off medication" is worse than matched controls.  Built into that is an assumption of no difference between normal controls and "on medication" - I would think it makes sense to do an equivalence test for that one.

My first thought was to split it all up, as Ryan mentioned.  And Ray, believe me, I would walk away if I could but I can't.

-------------------------------------------
Lisa Spielman
-------------------------------------------

The test of "ON" and "OFF" is fine -- paired t-test I suppose (is the difference from ON - OFF > 0?

An issue that commonly arrises is that you need a baseline to be "in a region" -- usually a bad region -- to be in the study.  Then you 'magically' find the next reading is better (regression to the mean).  Thus your ON may be better just because of RTTM and not some learning effect or anything else.  

Good luck!


-------------------------------------------
Scott Berry
Berry Consultants
-------------------------------------------







Original Message:
Sent: 06-13-2013 15:13
From: Lisa Spielman
Subject: Analysis of unusual matched sample design

It is indeed a bizarre design!

To give you a little more info:  The group assessed twice were adult subjects diagnosed with ADHD and being treated with medication, they were tested once on their meds and week or so later off their meds.  The outcome is a measure of attention that has several indices.

They did not counterbalance the on/off medication factor (yes, I know....let me reiterate that I had nothing to do with the design).  There is reason to believe that the test does not have any practice effects but I would have chosen to counterbalance anyway to confirm, and I would have assessed the control group twice as well.  The hypothesis is that performance "off medication" is worse than "on medication", and that performance "off medication" is worse than matched controls.  Built into that is an assumption of no difference between normal controls and "on medication" - I would think it makes sense to do an equivalence test for that one.

My first thought was to split it all up, as Ryan mentioned.  And Ray, believe me, I would walk away if I could but I can't.

-------------------------------------------
Lisa Spielman
-------------------------------------------

Wow, bizarre design.  If I got this right -- you have a baseline measure for everyone -- and a postbaseline for the drug group only....

So, you now dont really have any comparison for the drug?  They obviously needed the control group to get post-baseline as well... very likely regression to the mean effects, etc.  Thus the subjects with a single measure are meaningless???  You could now only really compare the change from baseline in the twice treated group to 0 ? (is the order of the on/off treatment always OFF then ON)? 

This would ignore any possible crossover effects.

So, I echo Ray in saying MANOVA makes no sense, and well, there is no ACRONYM analysis that will rescue this (and beware someone who thinks its your job to know the ACRONYM that creates a p-value less than 0.05)!

--Scott

-------------------------------------------
Scott Berry
Berry Consultants
-------------------------------------------

Lisa,

This sounds like a pre-experimental static group comparison at worst or a post-test only control group design at best. One ("experimental") group observed twice - once on treatment and once off, and a "control" group that was only observed once. Since it seems the groups were at least matched this may qualify as a post-test only experimental design if the participants were randomly assigned. If the participants were not randomly assigned then the study is not experimental and so just a static group comparison.

This is a comparison study so ANOVA or t-tests would handle the comparisons. If you have a single variable as an outcome (patient status) with one predictor (factor of on meds or no meds) then one-way ANOVA is best if you have a large enough sample size. 

MANOVA is when you have two separate continuos but somewhat related dependent variables and one or more separate factors as independent variables. Is there more than one outcome variable? (From what you described I suspect the answer is no). There appear to be TWO groups not three (there are 3 states - treatment and observation for the experimental group and observation for the control) and you would be comparing the post tests between the experimental group and the control to see if there is any difference in whatever he is using to measure outcome.

I agree with Ray - whoever ran this doesn't seem to know what he's doing. If he continues to insist on MANOVA I would politely say no thank you and walk away.

Hope that helps.

~Dan
-------------------------------------------
Daniel Butorovich
Research Analyst
Cochise College
-------------------------------------------

That is an odd design.  Can you share any details regarding the matching and how this was done, including any diagnostics on the quality of the matching process.  Was the matched control measured at pre or post -- and speaking of which, is the study group measured pre on medication and post off, or vice versa?  

Even with that, however, I suspect that the best you can do is tell them that this study should at best be viewed as a quick and dirty exploration of some of the issues in the use of this drug, and any fantasies regarding publication should be tossed aside.  As for MANOVA -- no.

-------------------------------------------
David Mangen
Owner
Mangen Research Associates, Inc.
-------------------------------------------