Discussion: View Thread

Hi, I have  a -great- client who really likes statistics. :)

Because of a non-disclosure I can't give too many technical details in the question that follows below.

The client has a "widget". Think of it as a type of radio transmitter. 

They give say, 100 of these widgets (all identical widgets) to each of say 5 "critter's" (no critters are harmed)  and each of the 100 widgets turns on, or not. They monitor the widget, and record if it "turns on" and then compute the proportion of widgets that do turn on for each critter.

Then the client gets another group of 5 critters, and makes some minor (but not unimportant ) engineering changes to "widget". They give 100 widgets to another 5 "critters".  Compute the proportion that turn on.

And they may make a few more engineering changes, give to critters,  and compute the proportion.

Sometimes a critter (critter-s) is used more than once in these experiments.

-nowhere in the process is anything randomized- and there's no experimental design-

The client is receptive to randomizing and experimental design for future work.

For now,

The client would like me to prepare differences of the proportions between different experiments, in other words between different engineering changes. And the client would like pairwise comparisons between critters. They have about 5 different engineering changes and want several pairwise comparisons (e.g. in sequence experiment 2 with experiment 1, experiment 3 with experiment 2, ...etc.)

 I think I'd have to reveal too much about the widget to explain whether or not they can randomize things.

But for now assuming randomizing isn't feasible, if I find (or don't) differences between proportions in the above mentioned experiments, it could be due to engineering changes, critters, or other unmeasured factors.

The metaphorical $64K widget comparison question. 

 Is the fact that there is no randomization a "show stopper", in other words,  should I recommend the client not prepare estimates of differences at all?  I think the answer is no.

 if I prepare comparisons, how best to explain the possible biases in the estimates in the absence of randomization. IF I do a good job of that, I think I can get the client to start using randomization.

thanks in advance.







-------------------------------------------
Chris Barker, Ph.D.
President - San Francisco Bay Area Chapter of the American Statistical Association
www,barkerstats.com

---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
-------------------------------------------

So I think the question is what can be assumed in the absence of randomization about the distribution of the number of widgets that get turned on? Wha would it take for the binomial assumption to be valid for a individual experiment?  You haven't said how these widgets get disributed among the critters.  Does each critter get 20 widgets?  Might not matter if it is reasonable to assume that there is no critter effect.  But maybe each critter has a different probability that it will turn the widget on.  I think it would be reasoanble to assume on any test that for a given critter the number of widgets that turn on is binomial since the widgets are identical.  But there is no reason to assume the p will be the same for each critter.  In that case what does it mean to compute a proportion p that combines all 100 widgets tested by the 5 critters.  Well the parameter is a linear combination of the pis for the critters with i=1,2,3,4 and 5.  But then the distribution for the sample is a mixture of binomial distrbutions.  Looks complicated but doable.Do you want to test for a critter effect?  Hopefully things will simplify and there is no need to assume a critter effect.  Then the mixture of binomials would simplify to a single binomial.  Then comparing two experiments would reduce to testing the null hypothesis that two binomials have the same p.  I think you can treat these as independent tests since there is no critter effect.  Then the only concern would be the mutliplicity of testing.
But what if there is a real critter effect.  Then the number turned on in each experiment is a linear combination of binomials.  Even if you can test for a difference in proportions between two experiments what does it mean?  The critter effect is confounded with a possible experiment effect.  Also since critters appear in more than one experiment the experiments have a correlation induced by the common critters.
I think the show stopper is the potential critter effect and the poor design rather than lack of randomization.  You could assume no critter effect and then do inference but on the surface I would no trust that approach.  since you can't tell us the details about the widgets and critters we cannot know whether or not a critter effect is possible.

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------

Interesting problem Chris, I'll give some scattered thoughts, but you're clearly in an area where there are numerous ways to go.

I would model the probability the device 'turns-on' (a success), as a function of the individual, i, and the device, d
(I think you were calling this experiment).  I would then model each using a random-effects (Bayesian Hierarchical) model:

logit(p) = beta + alpha_i + theta_d

alpha_i ~ N(0,tau^2); model tau^2 _ Gamma^-1

theta_d ~ N(0, sigma^2); model sigma^2 ~ Gamma^-1

beta ~ N(m,s^2)

I think each of the hierarchical aspects are important, certainly the mouse effects, but modeling the device effects
this way will certainly provide better estimates.

I think the randomization issue is an interesting one -- and it's more about comparisons.  
Given there is no comparison to anything going on, the lack of randomization is likely not a big deal.  If you model the 
individual effects (mouse) and dont worry about 'fatigue' issues -- getting multiple devices implanted, it doesnt seem
that bad.  Randomizing within the 5 mice doesnt do much as you still never have concurrent comparisons.  Randomization
doesnt help that.  What would be great is if device1 and device2 were compared directly, but the sequential nature of the
learning likely prevents that.  There may be increased value with keeping the previous device as a control -- so randomize 
4 new devices and 1 old device in each batch of 5 mice in the new 100.  You then could also estimate a possible "fatigue" 
aspect.  But you always have a bridging of devices across time (you may have A vs. B, B vs. C, C vs. D, etc, but this could
allow comparisons from A to D).

In humans we worry the population changes as the experiments progress, they get healthier, sicker, younger, older,
have other devices, etc, a nice aspect of mice is that they are probably very homogenous, and the question you are posing 
is whether they can evolve much, etc, which makes a control critical.  If you assume a mouse is a mouse, I'm not sure
randomization is a big deal at all.  Now, I know Bayesian don't need randomization as a basis of a sampling distribution,
but the generalizability may not be a big deal.  If mice (humans) selected treatment now we have big worries.

Depending on timing of this you also could have adaptive questions of the experiment (you may estimate after 30 devices
that this change was good/bad and thus evolve faster in the devices).

So, randomization is nice, but concurrent comparison is likely more important.  


-------------------------------------------
Scott Berry
Berry Consultants
-------------------------------------------

A mixed effects or random effects logistic regression model may be the way to go with this.  It could be done in the frequentist or the Bayesian hierarchical modeling framework (as Scott suggests).  The issue I think is whether or not you have enough data (especially repeated measurement for each critter) to separate critter effects from experiment effects.  Even if all the parameters are identifiable since the design was not planned the estimates may not be sufficiently accurate.  Scott introduced the possible complication of time dependence suggesting that with s future design a sequential approach might be beneficial.  All these are great suggestions if the presumed problems are real.  Some of these complications could imply that salvaging the poor non-randomized design may not be salvagable.
But the difficult and frustrating situation for consultants is to try to give good advice when the real problem has to be disguised and some real information must be held back.  This was the case with the recent problem that Adam presented as well as this one from Chris.  I am not criticzing anyone for presenting the problems this way.  I understand the reasons.  But good consulting requires complete understanding of the problem to know what assumptions are valid and what models are appropriate.  Otherwise we are shooting darts in the dark.  This may lead to interesting discussion and speculation which is fine if that is what is expected.  But if you are looking for good practical advice for the real problem I don't think his is the way to go about it.  We can provide clever ideas that might sound good but in reality be unnecessary or inappropriate.
Also the terminology for the individuals was critters.  Did Scott just substitute mice for critters or is it the case that the critters are mice?

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------







Original Message:
Sent: 04-07-2012 09:15
From: Scott Berry
Subject: Smalll non-randomized experiments

Interesting problem Chris, I'll give some scattered thoughts, but you're clearly in an area where there are numerous ways to go.

I would model the probability the device 'turns-on' (a success), as a function of the individual, i, and the device, d
(I think you were calling this experiment).  I would then model each using a random-effects (Bayesian Hierarchical) model:

logit(p) = beta + alpha_i + theta_d

alpha_i ~ N(0,tau^2); model tau^2 _ Gamma^-1

theta_d ~ N(0, sigma^2); model sigma^2 ~ Gamma^-1

beta ~ N(m,s^2)

I think each of the hierarchical aspects are important, certainly the mouse effects, but modeling the device effects
this way will certainly provide better estimates.

I think the randomization issue is an interesting one -- and it's more about comparisons.  
Given there is no comparison to anything going on, the lack of randomization is likely not a big deal.  If you model the 
individual effects (mouse) and dont worry about 'fatigue' issues -- getting multiple devices implanted, it doesnt seem
that bad.  Randomizing within the 5 mice doesnt do much as you still never have concurrent comparisons.  Randomization
doesnt help that.  What would be great is if device1 and device2 were compared directly, but the sequential nature of the
learning likely prevents that.  There may be increased value with keeping the previous device as a control -- so randomize 
4 new devices and 1 old device in each batch of 5 mice in the new 100.  You then could also estimate a possible "fatigue" 
aspect.  But you always have a bridging of devices across time (you may have A vs. B, B vs. C, C vs. D, etc, but this could
allow comparisons from A to D).

In humans we worry the population changes as the experiments progress, they get healthier, sicker, younger, older,
have other devices, etc, a nice aspect of mice is that they are probably very homogenous, and the question you are posing 
is whether they can evolve much, etc, which makes a control critical.  If you assume a mouse is a mouse, I'm not sure
randomization is a big deal at all.  Now, I know Bayesian don't need randomization as a basis of a sampling distribution,
but the generalizability may not be a big deal.  If mice (humans) selected treatment now we have big worries.

Depending on timing of this you also could have adaptive questions of the experiment (you may estimate after 30 devices
that this change was good/bad and thus evolve faster in the devices).

So, randomization is nice, but concurrent comparison is likely more important.  


-------------------------------------------
Scott Berry
Berry Consultants
-------------------------------------------

What about a Beta-Binomial model for this situation, Binomial within critters, and Beta for the Binomial parameters across critters?

That would be like the Greenwood-Yule model,
with Poisson for the number of accidents for each individual,
and  a Gamma  distribution over the Poisson parameters across individuals.
Integrating with respect to the Gamma yields a Negative Binomial for the marginal distribution
of the number of accidents. 

best   . . .
-- stan 

-------------------------------------------
Stanley Sclove
Professor
Univ of Illinois At Chicago
-------------------------------------------







Original Message:
Sent: 04-07-2012 10:36
From: Michael Chernick
Subject: Smalll non-randomized experiments

A mixed effects or random effects logistic regression model may be the way to go with this.  It could be done in the frequentist or the Bayesian hierarchical modeling framework (as Scott suggests).  The issue I think is whether or not you have enough data (especially repeated measurement for each critter) to separate critter effects from experiment effects.  Even if all the parameters are identifiable since the design was not planned the estimates may not be sufficiently accurate.  Scott introduced the possible complication of time dependence suggesting that with s future design a sequential approach might be beneficial.  All these are great suggestions if the presumed problems are real.  Some of these complications could imply that salvaging the poor non-randomized design may not be salvagable.
But the difficult and frustrating situation for consultants is to try to give good advice when the real problem has to be disguised and some real information must be held back.  This was the case with the recent problem that Adam presented as well as this one from Chris.  I am not criticzing anyone for presenting the problems this way.  I understand the reasons.  But good consulting requires complete understanding of the problem to know what assumptions are valid and what models are appropriate.  Otherwise we are shooting darts in the dark.  This may lead to interesting discussion and speculation which is fine if that is what is expected.  But if you are looking for good practical advice for the real problem I don't think his is the way to go about it.  We can provide clever ideas that might sound good but in reality be unnecessary or inappropriate.
Also the terminology for the individuals was critters.  Did Scott just substitute mice for critters or is it the case that the critters are mice?

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------

Thank you. A quick followup.

 I also replied privately to thank each person for a response.  I appreciate the several excellent suggestions for methodology, given the anonymization of my actual consulting problem. I now have several ways to tackle the problem from the advice here.

The absence of randomization seems less of an issue than I had thought. I often work in the world of Phase III where without randomization, "abandon all hope - ye who enter here "... :)

I think "confounding" underscores the potential problems of the lack of an experimental design and is a concept my client will understand.

The client had taken my last set of statistical suggestions (same widget, but for a different problem) and used those to write SOP's. Its always gratifying to work with a client receptive to statistics.



-------------------------------------------
Chris Barker, Ph.D.
President - San Francisco Bay Area Chapter of the American Statistical Association
www,barkerstats.com

---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
-------------------------------------------







Original Message:
Sent: 04-07-2012 10:36
From: Michael Chernick
Subject: Smalll non-randomized experiments

A mixed effects or random effects logistic regression model may be the way to go with this.  It could be done in the frequentist or the Bayesian hierarchical modeling framework (as Scott suggests).  The issue I think is whether or not you have enough data (especially repeated measurement for each critter) to separate critter effects from experiment effects.  Even if all the parameters are identifiable since the design was not planned the estimates may not be sufficiently accurate.  Scott introduced the possible complication of time dependence suggesting that with s future design a sequential approach might be beneficial.  All these are great suggestions if the presumed problems are real.  Some of these complications could imply that salvaging the poor non-randomized design may not be salvagable.
But the difficult and frustrating situation for consultants is to try to give good advice when the real problem has to be disguised and some real information must be held back.  This was the case with the recent problem that Adam presented as well as this one from Chris.  I am not criticzing anyone for presenting the problems this way.  I understand the reasons.  But good consulting requires complete understanding of the problem to know what assumptions are valid and what models are appropriate.  Otherwise we are shooting darts in the dark.  This may lead to interesting discussion and speculation which is fine if that is what is expected.  But if you are looking for good practical advice for the real problem I don't think his is the way to go about it.  We can provide clever ideas that might sound good but in reality be unnecessary or inappropriate.
Also the terminology for the individuals was critters.  Did Scott just substitute mice for critters or is it the case that the critters are mice?

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------

I am glad that inspite of having to hide aspects of the problem you were able to find our advice helpful.  I think I mostly mentioned what the difficulties were. Scott and Stan provided some good models that could apply without assuming the critters have the same p.  I work mostly in pharma and clinical trials and randomization is important there because with any medical treatment patients with the same disease will respond differently to treatments.  So there is always an effect due to patient characteristics.  Important demographic characteristics might be age, gender, race, genetic markers, comorbidities, and other risk factors such as smoking or obesity.  Randomization tends to be an automatic way of balancing these factors so that confounding is avoided.  

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------







Original Message:
Sent: 04-07-2012 16:12
From: Christopher Barker
Subject: Smalll non-randomized experiments


Thank you. A quick followup.

 I also replied privately to thank each person for a response.  I appreciate the several excellent suggestions for methodology, given the anonymization of my actual consulting problem. I now have several ways to tackle the problem from the advice here.

The absence of randomization seems less of an issue than I had thought. I often work in the world of Phase III where without randomization, "abandon all hope - ye who enter here "... :)

I think "confounding" underscores the potential problems of the lack of an experimental design and is a concept my client will understand.

The client had taken my last set of statistical suggestions (same widget, but for a different problem) and used those to write SOP's. Its always gratifying to work with a client receptive to statistics.



-------------------------------------------
Chris Barker, Ph.D.
President - San Francisco Bay Area Chapter of the American Statistical Association
www,barkerstats.com

---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
-------------------------------------------







Original Message:
Sent: 04-07-2012 10:36
From: Michael Chernick
Subject: Smalll non-randomized experiments

A mixed effects or random effects logistic regression model may be the way to go with this.  It could be done in the frequentist or the Bayesian hierarchical modeling framework (as Scott suggests).  The issue I think is whether or not you have enough data (especially repeated measurement for each critter) to separate critter effects from experiment effects.  Even if all the parameters are identifiable since the design was not planned the estimates may not be sufficiently accurate.  Scott introduced the possible complication of time dependence suggesting that with s future design a sequential approach might be beneficial.  All these are great suggestions if the presumed problems are real.  Some of these complications could imply that salvaging the poor non-randomized design may not be salvagable.
But the difficult and frustrating situation for consultants is to try to give good advice when the real problem has to be disguised and some real information must be held back.  This was the case with the recent problem that Adam presented as well as this one from Chris.  I am not criticzing anyone for presenting the problems this way.  I understand the reasons.  But good consulting requires complete understanding of the problem to know what assumptions are valid and what models are appropriate.  Otherwise we are shooting darts in the dark.  This may lead to interesting discussion and speculation which is fine if that is what is expected.  But if you are looking for good practical advice for the real problem I don't think his is the way to go about it.  We can provide clever ideas that might sound good but in reality be unnecessary or inappropriate.
Also the terminology for the individuals was critters.  Did Scott just substitute mice for critters or is it the case that the critters are mice?

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------

I'm not sure it this is relevant to the problem (since it is a hypothetical problem), but the lack of randomization that I would be concerned with has to do with the repeated measures.  An ideal repeated measures design would randomize the order of the redesigned widgets.  If the order is not randomized, it makes it difficult to test whether any differences observed are period or redesign effects.  If the critters are learning how to turn the widgets on, that may be the effect observed rather than an improved design.  Since not all critters have repeated measures, maybe you can test for a period (or learning) effect.

-------------------------------------------
Colleen Kelly
Principal Consultant
Kelly Statistical Consulting
-------------------------------------------







Original Message:
Sent: 04-07-2012 16:12
From: Christopher Barker
Subject: Smalll non-randomized experiments


Thank you. A quick followup.

 I also replied privately to thank each person for a response.  I appreciate the several excellent suggestions for methodology, given the anonymization of my actual consulting problem. I now have several ways to tackle the problem from the advice here.

The absence of randomization seems less of an issue than I had thought. I often work in the world of Phase III where without randomization, "abandon all hope - ye who enter here "... :)

I think "confounding" underscores the potential problems of the lack of an experimental design and is a concept my client will understand.

The client had taken my last set of statistical suggestions (same widget, but for a different problem) and used those to write SOP's. Its always gratifying to work with a client receptive to statistics.



-------------------------------------------
Chris Barker, Ph.D.
President - San Francisco Bay Area Chapter of the American Statistical Association
www,barkerstats.com

---
"In composition you have all the time you want to decide what to say in 15 seconds, in improvisation you have 15 seconds."
-Steve Lacy
-------------------------------------------







Original Message:
Sent: 04-07-2012 10:36
From: Michael Chernick
Subject: Smalll non-randomized experiments

A mixed effects or random effects logistic regression model may be the way to go with this.  It could be done in the frequentist or the Bayesian hierarchical modeling framework (as Scott suggests).  The issue I think is whether or not you have enough data (especially repeated measurement for each critter) to separate critter effects from experiment effects.  Even if all the parameters are identifiable since the design was not planned the estimates may not be sufficiently accurate.  Scott introduced the possible complication of time dependence suggesting that with s future design a sequential approach might be beneficial.  All these are great suggestions if the presumed problems are real.  Some of these complications could imply that salvaging the poor non-randomized design may not be salvagable.
But the difficult and frustrating situation for consultants is to try to give good advice when the real problem has to be disguised and some real information must be held back.  This was the case with the recent problem that Adam presented as well as this one from Chris.  I am not criticzing anyone for presenting the problems this way.  I understand the reasons.  But good consulting requires complete understanding of the problem to know what assumptions are valid and what models are appropriate.  Otherwise we are shooting darts in the dark.  This may lead to interesting discussion and speculation which is fine if that is what is expected.  But if you are looking for good practical advice for the real problem I don't think his is the way to go about it.  We can provide clever ideas that might sound good but in reality be unnecessary or inappropriate.
Also the terminology for the individuals was critters.  Did Scott just substitute mice for critters or is it the case that the critters are mice?

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------