Discussion: View Thread

This message has been cross posted to the following eGroups: Biopharmaceutical Section and Statistical Consulting Section .
-------------------------------------------
Dear Colleagues, 

You might want to consider a Bayes interval. ------------------------------------------- Daniel Jeske University of California Department of Statistics -------------------------------------------
Original Message: Sent: 05-25-2011 00:02 From: Christopher Barker Subject: Confidence interval for a binomial proportion. This message has been cross posted to the following eGroups: Biopharmaceutical Section and Statistical Consulting Section . ------------------------------------------- Dear Colleagues, I'm working with a client who has a widget <confidentiality forbids me from revealing> that leads to success (binomial) rates close to 100% or close to zero. My client has asked me for the optimal choice of the confidence interval. My client is very knowledgeable, and having surfed the web, know that a Wald interval may not be the best choice. I'd appreciate perspectives and opinions on the choice of the "best" (in quotes) confidence interval. In advance, to name a few "Agresti Coull, Wald, Wilson, Jeffrey's, Blyth Still Casella (I have dozens of articles and more on order). And in advance of asking, and that my client has an urgent priority, I have made my recommendations to the client. I'd appreciate more recommendations. :) - -thanks in advance ------------------------------------------- Christopher Barker, Ph.D. Statistical Planning and Analysis Services, Inc. www.barkerstats.com -------------------------------------------

I would agree to some extent; I would also refer them to literature regarding diminishing returns and type II increase as you decrease your potential for error with CI. If they want to review probable outcomes and research some components of stats/methodology I would also suggest that you, with appropriate/positive attitude, include other categorical variables in your testing/research. If that makes sense?

I guess what I am trying to say is that I frequently listen to Car Talk. I don't, however, walk up to my mechanic and tell him precisely what I want done and nothing else. I want them to assess the situation in its entirety, review all possible outcomes, and yield the best conclusion/solution for (in this case) my car.

Night,

KM

-------------------------------------------
Kevin McKenna
-------------------------------------------






Original Message:
Sent: 05-25-2011 00:17
From: Daniel Jeske
Subject: Confidence interval for a binomial proportion.

You might want to consider a Bayes interval.
-------------------------------------------
Daniel Jeske
University of California Department of Statistics
-------------------------------------------

Have you seen the review in Statistical Science by Brown, Cai and DasGupta 2001? Here's the abstract:

We revisit the problem of interval estimation of a binomial proportion. The erratic behavior of the coverage probability of the standard Wald confidence interval has previously been remarked on in the literature (Blyth and Still, Agresti and Coull, Santner and others). We begin by showing that the chaotic coverage properties of the Wald interval are far more persistent than is appreciated. Furthermore, common textbook prescriptions regarding its safety are misleading and defective in several respects and cannot be trusted.

This leads us to consideration of alternative intervals. A number of natural alternatives are presented, each with its motivation and context. Each interval is examined for its coverage probability and its length. Based on this analysis, we recommend the Wilson interval or the equal-tailed Jeffreys prior interval for small n and the interval suggested in Agresti and Coull for larger n. We also provide an additional frequentist justification for use of the Jeffreys interval.

The same authors have a more detailed treatment in the Annals of Statistics 2002.

There is also a paper by Pires and Amado 2008. in RevStat Statistical Journal: Here's the abstract:

In applied statistics it is often necessary to obtain an interval estimate for an unknown
proportion (p) based on binomial sampling. This topic is considered in almost every
introductory course. However, the usual approximation is known to be poor when
the true p is close to zero or to one. To identify alternative procedures with better
properties twenty non-iterative methods for computing a (central) two-sided interval
estimate for p were selected and compared in terms of coverage probability and ex-
pected length. From this study a clear classification of those methods has emerged.
An important conclusion is that the interval based on asymptotic normality, but af-
ter the arcsine transformation and a continuity correction, and the Add 4 method
of Agresti and Coull (1998) yield very reliable results, the choice between the two
depending on the desired degree of conservativeness.

Cheers,

Simon.
-------------------------------------------
Simon Blomberg
Lecturer and Consultant Statistician
University of Queensland School of Biological Sciences
-------------------------------------------






Original Message:
Sent: 05-25-2011 00:02
From: Christopher Barker
Subject: Confidence interval for a binomial proportion.

This message has been cross posted to the following eGroups: Biopharmaceutical Section and Statistical Consulting Section .
-------------------------------------------

Dear Colleagues, 

I'm working with a client who has a widget <confidentiality forbids me from revealing>  that leads to success (binomial) rates close to 100% or close to zero.

My client has asked me for the optimal choice of the confidence interval. My client is very knowledgeable, and having surfed the web, know that a Wald interval may not be the best choice.

I'd appreciate perspectives and opinions on the choice of the "best" (in quotes) confidence interval.

In advance, to name a few "Agresti Coull, Wald, Wilson, Jeffrey's, Blyth Still Casella (I have dozens of  articles and more on order).

• And in advance of asking, and that my client has an urgent priority, I have made my recommendations to the client. 

I'd appreciate more recommendations. :)

-

-thanks in advance
-------------------------------------------
Christopher Barker, Ph.D.
Statistical Planning and Analysis Services, Inc.

www.barkerstats.com
-------------------------------------------

At very low failure rates, a Poisson probability distribution would be better as a model than a binomial.  I would run a simulation model, incrementaly changing the population failure rates against your actual results.  That should allow you to build a confidence interval, within the 90 (or 95) percent level.

-------------------------------------------
Brian Taylor
Operations Research Analyst
Army Test & Evaluation Command
-------------------------------------------






Original Message:
Sent: 05-25-2011 00:46
From: Simon Blomberg
Subject: Confidence interval for a binomial proportion.


Have you seen the review in Statistical Science by Brown, Cai and DasGupta 2001? Here's the abstract:

We revisit the problem of interval estimation of a binomial proportion. The erratic behavior of the coverage probability of the standard Wald confidence interval has previously been remarked on in the literature (Blyth and Still, Agresti and Coull, Santner and others). We begin by showing that the chaotic coverage properties of the Wald interval are far more persistent than is appreciated. Furthermore, common textbook prescriptions regarding its safety are misleading and defective in several respects and cannot be trusted.

This leads us to consideration of alternative intervals. A number of natural alternatives are presented, each with its motivation and context. Each interval is examined for its coverage probability and its length. Based on this analysis, we recommend the Wilson interval or the equal-tailed Jeffreys prior interval for small n and the interval suggested in Agresti and Coull for larger n. We also provide an additional frequentist justification for use of the Jeffreys interval.

The same authors have a more detailed treatment in the Annals of Statistics 2002.

There is also a paper by Pires and Amado 2008. in RevStat Statistical Journal: Here's the abstract:

In applied statistics it is often necessary to obtain an interval estimate for an unknown
proportion (p) based on binomial sampling. This topic is considered in almost every
introductory course. However, the usual approximation is known to be poor when
the true p is close to zero or to one. To identify alternative procedures with better
properties twenty non-iterative methods for computing a (central) two-sided interval
estimate for p were selected and compared in terms of coverage probability and ex-
pected length. From this study a clear classification of those methods has emerged.
An important conclusion is that the interval based on asymptotic normality, but af-
ter the arcsine transformation and a continuity correction, and the Add 4 method
of Agresti and Coull (1998) yield very reliable results, the choice between the two
depending on the desired degree of conservativeness.

Cheers,

Simon.
-------------------------------------------
Simon Blomberg
Lecturer and Consultant Statistician
University of Queensland School of Biological Sciences
-------------------------------------------






Original Message:
Sent: 05-25-2011 00:02
From: Christopher Barker
Subject: Confidence interval for a binomial proportion.

This message has been cross posted to the following eGroups: Biopharmaceutical Section and Statistical Consulting Section .
-------------------------------------------

Dear Colleagues, 

I'm working with a client who has a widget <confidentiality forbids me from revealing>  that leads to success (binomial) rates close to 100% or close to zero.

My client has asked me for the optimal choice of the confidence interval. My client is very knowledgeable, and having surfed the web, know that a Wald interval may not be the best choice.

I'd appreciate perspectives and opinions on the choice of the "best" (in quotes) confidence interval.

In advance, to name a few "Agresti Coull, Wald, Wilson, Jeffrey's, Blyth Still Casella (I have dozens of  articles and more on order).

• And in advance of asking, and that my client has an urgent priority, I have made my recommendations to the client. 

I'd appreciate more recommendations. :)

-

-thanks in advance
-------------------------------------------
Christopher Barker, Ph.D.
Statistical Planning and Analysis Services, Inc.

www.barkerstats.com
-------------------------------------------

1) I think in this situation (and in many others) one-sided confidence bounds make more sense
than two sided confidence intervals.

2) If you like the idea of a "mid-p-value" (which can also be applied to confidence statements),
then you shouldn't use a continuity correction.  The two ideas come very close to canceling
each other out.

3) I respectfully disagree with Brian's suggestion.  It is true that for rare events the binomial and
Poisson models closely approximate each other, but if n is fixed and known (and the trials are
independent and the success probability is constant) then the binomial model is the correct one.
Also, it is not clear why a simulation based approach would be useful in this case.

-------------------------------------------
Regards,  Rob Kushler
-------------------------------------------






Original Message:
Sent: 05-25-2011 07:44
From: Brian Taylor
Subject: Confidence interval for a binomial proportion.


At very low failure rates, a Poisson probability distribution would be better as a model than a binomial.  I would run a simulation model, incrementaly changing the population failure rates against your actual results.  That should allow you to build a confidence interval, within the 90 (or 95) percent level.

-------------------------------------------
Brian Taylor
Operations Research Analyst
Army Test & Evaluation Command
-------------------------------------------

I was going to make the same point as Robert Kushler made about using a Poisson approximation to the binomial.  It is amazing how much literature there is on such a simple sounding problem but having done research in this area I understand why.  Unless the sample size is very large I prefer exact methods to approximate ones.

I think we are beginning to overwhelm the questioner with a lot of journal articles (I am guilty too).  I think we should really try to give simple advice which really depends on understanding the actual problem better than we do.  As was mentioned we should find out if this is really a problem that requires binomial confidence intervals.  What is the actual sample size?  Is a 1-sided confidence set acceptable?  How close to 0 or 1 does the client expect the proportion to be?  When we know all the facts the solution could be very simple and we may not need to get bogged down in the theoretical subtleties.

I think the purpose of these threads on this eGroup is to help solve real problems and too often we run off about theory.

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------






Original Message:
Sent: 05-25-2011 10:06
From: Robert Kushler
Subject: Confidence interval for a binomial proportion.


1) I think in this situation (and in many others) one-sided confidence bounds make more sense
than two sided confidence intervals.

2) If you like the idea of a "mid-p-value" (which can also be applied to confidence statements),
then you shouldn't use a continuity correction.  The two ideas come very close to canceling
each other out.

3) I respectfully disagree with Brian's suggestion.  It is true that for rare events the binomial and
Poisson models closely approximate each other, but if n is fixed and known (and the trials are
independent and the success probability is constant) then the binomial model is the correct one.
Also, it is not clear why a simulation based approach would be useful in this case.

-------------------------------------------
Regards,  Rob Kushler
-------------------------------------------






Original Message:
Sent: 05-25-2011 07:44
From: Brian Taylor
Subject: Confidence interval for a binomial proportion.


At very low failure rates, a Poisson probability distribution would be better as a model than a binomial.  I would run a simulation model, incrementaly changing the population failure rates against your actual results.  That should allow you to build a confidence interval, within the 90 (or 95) percent level.

-------------------------------------------
Brian Taylor
Operations Research Analyst
Army Test & Evaluation Command
-------------------------------------------

I've been following all the posts and appreciate the suggestions and the associated references. I was familiar with the Wikipedia article and like Michael's validation of it. 

It would be helpful if Christopher would give us an idea of the range of sample sizes to be expected.

Since this is a consulting section, I want to point out that Brian's suggestion of simulating data is an important one. Whenever I get a request for which the answer isn't clear, I simulate the data and try various methods that potentially apply. I also believe in conducting a sensitivity analysis: how sensitive are the various methods to the decision to be made. Finally, I apply the rule: we won't go to the next level of sophistication unless the business decision would be wrong otherwise.

Those guidelines have served me well over the years.

-------------------------------------------
Patrick Spagon
-------------------------------------------






Original Message:
Sent: 05-25-2011 07:44
From: Brian Taylor
Subject: Confidence interval for a binomial proportion.


At very low failure rates, a Poisson probability distribution would be better as a model than a binomial.  I would run a simulation model, incrementaly changing the population failure rates against your actual results.  That should allow you to build a confidence interval, within the 90 (or 95) percent level.

-------------------------------------------
Brian Taylor
Operations Research Analyst
Army Test & Evaluation Command
-------------------------------------------

Very good suggestions.  Simulation could be a useful tool in many cases.  But understand the problem first.  Don't simulate a Poisson process if the problem is to find a binomial confidence interval.  Also simulation should be used when direct analytic solutions are not available.  If there is an anytic solution to the problem why simulate?
-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------






Original Message:
Sent: 05-25-2011 12:46
From: Patrick Spagon
Subject: Confidence interval for a binomial proportion.

I've been following all the posts and appreciate the suggestions and the associated references. I was familiar with the Wikipedia article and like Michael's validation of it. 

It would be helpful if Christopher would give us an idea of the range of sample sizes to be expected.

Since this is a consulting section, I want to point out that Brian's suggestion of simulating data is an important one. Whenever I get a request for which the answer isn't clear, I simulate the data and try various methods that potentially apply. I also believe in conducting a sensitivity analysis: how sensitive are the various methods to the decision to be made. Finally, I apply the rule: we won't go to the next level of sophistication unless the business decision would be wrong otherwise.

Those guidelines have served me well over the years.

-------------------------------------------
Patrick Spagon
-------------------------------------------

Just another suggestion for consulting: If I don't have sufficient real data from the client, the only reasons I would simulate data when there are competing analytic solutions to the problem are:

1. It may not be clear to the client which method is appropriate and

2. To convince them which approach is best given data that matches their problem

We know the old saying: "In God we trust, all others must bring data."

Because of what I've had to deal with the over the years for many different organizations I have added the following to the saying:

"Unless your stature in the organization is such that your opinion is data!"

How often have we come across a VP or CEO who will say, "I don't care. We will do it this way."

That's when using their own data or data simulated that clearly matches theirs goes a long way in helping convince the decision-makers to take your recommended approach.

-------------------------------------------
Patrick Spagon
-------------------------------------------






Original Message:
Sent: 05-25-2011 12:53
From: Michael Chernick
Subject: Confidence interval for a binomial proportion.


Very good suggestions.  Simulation could be a useful tool in many cases.  But understand the problem first.  Don't simulate a Poisson process if the problem is to find a binomial confidence interval.  Also simulation should be used when direct analytic solutions are not available.  If there is an anytic solution to the problem why simulate?
-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------

I work in industry, so I understand where you are coming from on this.  It just amazes me how people can be in such awe of the computer that a simulation will be assumed correct without question. But if the model is wrong the data generated by the simulation is wrong.  A mathematical closed form solution is an exact answer.  Simulation only approximates the answer using a finite sample.

But it is true that for some executives it is not possible to communicate through mathematics but they can be convinced that simulation is replicating the real situation.
-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------

Original Message:
Sent: 05-25-2011 13:04
From: Patrick Spagon
Subject: Confidence interval for a binomial proportion.

Just another suggestion for consulting: If I don't have sufficient real data from the client, the only reasons I would simulate data when there are competing analytic solutions to the problem are:

1. It may not be clear to the client which method is appropriate and

2. To convince them which approach is best given data that matches their problem

We know the old saying: "In God we trust, all others must bring data."

Because of what I've had to deal with the over the years for many different organizations I have added the following to the saying:

"Unless your stature in the organization is such that your opinion is data!"

How often have we come across a VP or CEO who will say, "I don't care. We will do it this way."

That's when using their own data or data simulated that clearly matches theirs goes a long way in helping convince the decision-makers to take your recommended approach.

-------------------------------------------
Patrick Spagon
-------------------------------------------

I do not see analytic solutions vs simulation to be an "only one of a or b" problem.

Both in teaching and in other communication my experience is that it is more productive to do both.

Sometimes it helps in the process of moving from the presenting question to the underlying question to at least think about how one would simulate an analogous situation.

My undergrad background in Aristotelian philosophy drove home the value of seeking ways to slip between the horns of the dilemma.

I would need to have a better understanding of the actual problem.  How many hits might there be out of how many tries?  What about when the number of hits is zero or n?
Is this a one-time or recurring problem?  What is the cost of being wrong? Is the distinction between a hit and a non-hit itself done with the possibility of measurement error? In other words is the dichotomy actually nominal level or is a cut on a continuous construct?

-------------------------------------------
Arthur Kendall
Social Research Consultants
-------------------------------------------






Original Message:
Sent: 05-25-2011 13:04
From: Patrick Spagon
Subject: Confidence interval for a binomial proportion.

Just another suggestion for consulting: If I don't have sufficient real data from the client, the only reasons I would simulate data when there are competing analytic solutions to the problem are:

1. It may not be clear to the client which method is appropriate and

2. To convince them which approach is best given data that matches their problem

We know the old saying: "In God we trust, all others must bring data."

Because of what I've had to deal with the over the years for many different organizations I have added the following to the saying:

"Unless your stature in the organization is such that your opinion is data!"

How often have we come across a VP or CEO who will say, "I don't care. We will do it this way."

That's when using their own data or data simulated that clearly matches theirs goes a long way in helping convince the decision-makers to take your recommended approach.

-------------------------------------------
Patrick Spagon
-------------------------------------------

The discussion on binomial confidence intervals has gone about as far as it could go but the discussion turned to talking about when it is appropriate to use simulation and I think the points I made were misinterpreted,

I used simulation as a graduate student to confirm that the limit theorem I was trying to prove was correct.  I once wrote a section on Monte Carlo methods in a publication on risk analysis techniques.  Simulation is an excellent tool for teaching probability and statistics.  In fact Julian Simon used bootstrapping by Monte Carlo as a teaching tool.  On my papers on bootstrapping classification error rates the papers were entirely based on simulations to demonstrate the properties of bootstrap estimates and to compare estimates.  Of course it has also been valuable in robustness studies and in solving many other research problems in science, statistics, operations research and applied mathematics.  

In this discussion we have also touched on the value of simulation in explaining results to upper management and clients we consult with.  My points were simply these.

1. As a professional statistician when we know the answer by direct computation or closed form solution it would be silly to use Monte Carlo.  At least that is how I see it.

2. It seems ironic to me that we are successful at convincing management or clients of the correctness of our results by the aid of simulation when a simple statistical argument might suffice.  What I am thinking of is that our explanation is in the form of the output of the simulation without any discussion of how the simulation was done.  When you simulate there are really a lot of questions that should be asked.
    i. What random number generators did you use?
    ii. What is the underlying probability model used in the simulation?  Of course this would include the probability distributions associated with the input variables.
    iii.  Did your computer code execute the simulation the way you intended?
    iv.  How many Monte Carlo iterations did you use?  Of course the number of iterations affects how accurate the simulation is at approximating the answer.  When sampling from continuous distributions Monte Carlo always provides an approximate answer because it is based on samples from the assumed populations of the input variables.

What I think is ironic is that the results are accepted without asking any of these 4 questions when a perfectly correct mathematical solution would be disgarded because it is not understood.
Clients in this case would not really understand the Monte Carlo solution either but they think they do.  
-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------






Original Message:
Sent: 05-26-2011 07:20
From: Arthur Kendall
Subject: Confidence interval for a binomial proportion.

I do not see analytic solutions vs simulation to be an "only one of a or b" problem.

Both in teaching and in other communication my experience is that it is more productive to do both.

Sometimes it helps in the process of moving from the presenting question to the underlying question to at least think about how one would simulate an analogous situation.

My undergrad background in Aristotelian philosophy drove home the value of seeking ways to slip between the horns of the dilemma.

I would need to have a better understanding of the actual problem.  How many hits might there be out of how many tries?  What about when the number of hits is zero or n?
Is this a one-time or recurring problem?  What is the cost of being wrong? Is the distinction between a hit and a non-hit itself done with the possibility of measurement error? In other words is the dichotomy actually nominal level or is a cut on a continuous construct?

-------------------------------------------
Arthur Kendall
Social Research Consultants
-------------------------------------------






Original Message:
Sent: 05-25-2011 13:04
From: Patrick Spagon
Subject: Confidence interval for a binomial proportion.

Just another suggestion for consulting: If I don't have sufficient real data from the client, the only reasons I would simulate data when there are competing analytic solutions to the problem are:

1. It may not be clear to the client which method is appropriate and

2. To convince them which approach is best given data that matches their problem

We know the old saying: "In God we trust, all others must bring data."

Because of what I've had to deal with the over the years for many different organizations I have added the following to the saying:

"Unless your stature in the organization is such that your opinion is data!"

How often have we come across a VP or CEO who will say, "I don't care. We will do it this way."

That's when using their own data or data simulated that clearly matches theirs goes a long way in helping convince the decision-makers to take your recommended approach.

-------------------------------------------
Patrick Spagon
-------------------------------------------

I'm half sorry I worked so many up with the of using simulation confidence intervals.  My main point is different however.  A symmetric distribution, such as a binomial, is not a good representation of the real world when the probability of failure is very low.  This low probability is often the case for military weapon systems.  And of course, we all hope the probability of failure is low for these.  So for low failure rates (or low success rates), a poisson distribution is a better model.  One can use simulation to built confidence interval around a non-symmetric distribution.

-------------------------------------------
Brian Taylor
Operations Research Analyst
Army Test & Evaluation Command
-------------------------------------------






Original Message:
Sent: 05-26-2011 09:36
From: Michael Chernick
Subject: Confidence interval for a binomial proportion.


The discussion on binomial confidence intervals has gone about as far as it could go but the discussion turned to talking about when it is appropriate to use simulation and I think the points I made were misinterpreted,

I used simulation as a graduate student to confirm that the limit theorem I was trying to prove was correct.  I once wrote a section on Monte Carlo methods in a publication on risk analysis techniques.  Simulation is an excellent tool for teaching probability and statistics.  In fact Julian Simon used bootstrapping by Monte Carlo as a teaching tool.  On my papers on bootstrapping classification error rates the papers were entirely based on simulations to demonstrate the properties of bootstrap estimates and to compare estimates.  Of course it has also been valuable in robustness studies and in solving many other research problems in science, statistics, operations research and applied mathematics.  

In this discussion we have also touched on the value of simulation in explaining results to upper management and clients we consult with.  My points were simply these.

1. As a professional statistician when we know the answer by direct computation or closed form solution it would be silly to use Monte Carlo.  At least that is how I see it.

2. It seems ironic to me that we are successful at convincing management or clients of the correctness of our results by the aid of simulation when a simple statistical argument might suffice.  What I am thinking of is that our explanation is in the form of the output of the simulation without any discussion of how the simulation was done.  When you simulate there are really a lot of questions that should be asked.
    i. What random number generators did you use?
    ii. What is the underlying probability model used in the simulation?  Of course this would include the probability distributions associated with the input variables.
    iii.  Did your computer code execute the simulation the way you intended?
    iv.  How many Monte Carlo iterations did you use?  Of course the number of iterations affects how accurate the simulation is at approximating the answer.  When sampling from continuous distributions Monte Carlo always provides an approximate answer because it is based on samples from the assumed populations of the input variables.

What I think is ironic is that the results are accepted without asking any of these 4 questions when a perfectly correct mathematical solution would be disgarded because it is not understood.
Clients in this case would not really understand the Monte Carlo solution either but they think they do.  
-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------

Who said the binomial is symmetric?  The Poisson is just a convenience, but nothing is lost in using the actual binomial.

Jon

-------------------------------------------
Jon Shuster
University of Florida
-------------------------------------------






Original Message:
Sent: 05-26-2011 10:48
From: Brian Taylor
Subject: Confidence interval for a binomial proportion.

I'm half sorry I worked so many up with the of using simulation confidence intervals.  My main point is different however.  A symmetric distribution, such as a binomial, is not a good representation of the real world when the probability of failure is very low.  This low probability is often the case for military weapon systems.  And of course, we all hope the probability of failure is low for these.  So for low failure rates (or low success rates), a poisson distribution is a better model.  One can use simulation to built confidence interval around a non-symmetric distribution.

-------------------------------------------
Brian Taylor
Operations Research Analyst
Army Test & Evaluation Command
-------------------------------------------

This comment works us up even more than the one about the Poisson because the binomial is only symmetric when p=1/2.  Otherwise it is skewed to the left or right depending on p.
With p near 0 or 1 the binomial is highly skewed.  So if the binomial is a bad model for your weapons system tests it hos nothing to do with symmetry.  As others have said if the number of trials is fixed the Poisson is only a tool as a possibly useful approximation because it represents number of events by time t and not number of failures in n trials.

It is possible that what you meant which would be true is that when p is close to 0 or 1 the normal approximation to the binomial is not good because the normal distribution is symmetric and the binomial is not.  But what we have been discussing includes confidence intervals based on the exact binomial or a slight adjustment to that and not just the normal approximation.

By the way I worked for AMSAA at APG for 1969-1974 and my brother worked there from 1970 to now with a brief stint working for a contractor located in Aberdeen (Lockheed-Martin).  We are very familiar with Tecom.

A few years ago I gave a lecture on bootstrap at the Maryland Chapter of ASA.

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------

Original Message:
Sent: 05-26-2011 10:48
From: Brian Taylor
Subject: Confidence interval for a binomial proportion.

I'm half sorry I worked so many up with the of using simulation confidence intervals.  My main point is different however.  A symmetric distribution, such as a binomial, is not a good representation of the real world when the probability of failure is very low.  This low probability is often the case for military weapon systems.  And of course, we all hope the probability of failure is low for these.  So for low failure rates (or low success rates), a poisson distribution is a better model.  One can use simulation to built confidence interval around a non-symmetric distribution.

-------------------------------------------
Brian Taylor
Operations Research Analyst
Army Test & Evaluation Command
-------------------------------------------

This is a common problem in doing risk analysis for actuarial modeling.

There are two aspects to this problem in risk analysis  The first is the frequency of failure, divided by the exposure time  - in other words, the failure has to be taken over the period of time the weapon is actually being deployed.  This is often modeled as a Poisson process if the risk of failure is very low.

The second is severity of failure, which is the incurred loss (or cost) for the failure events.  This is often modelled as a gamma function.  This model assumes that most failures incur small costs, and a few failures incur huge costs.  You need to check the fit against your data.

If you want to study which factors are likely to lead to failure, then you need to consider whether those factors are related in a multiplicative way or are additive.  If they are multiplicative, then you also need to use a log link function in the model.

You probably should consult a risk actuary about the way to do this in more detail, particularly those who have to insure catastrophic losses. 

The procedures for doing such analysis are found in the generalized linear model software of SAS, SPSS or R.



-------------------------------------------
Elizabeth Smith
-------------------------------------------






Original Message:
Sent: 05-26-2011 11:26
From: Michael Chernick
Subject: Confidence interval for a binomial proportion.

This comment works us up even more than the one about the Poisson because the binomial is only symmetric when p=1/2.  Otherwise it is skewed to the left or right depending on p.
With p near 0 or 1 the binomial is highly skewed.  So if the binomial is a bad model for your weapons system tests it hos nothing to do with symmetry.  As others have said if the number of trials is fixed the Poisson is only a tool as a possibly useful approximation because it represents number of events by time t and not number of failures in n trials.

It is possible that what you meant which would be true is that when p is close to 0 or 1 the normal approximation to the binomial is not good because the normal distribution is symmetric and the binomial is not.  But what we have been discussing includes confidence intervals based on the exact binomial or a slight adjustment to that and not just the normal approximation.

By the way I worked for AMSAA at APG for 1969-1974 and my brother worked there from 1970 to now with a brief stint working for a contractor located in Aberdeen (Lockheed-Martin).  We are very familiar with Tecom.

A few years ago I gave a lecture on bootstrap at the Maryland Chapter of ASA.

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------

Hi, the best I can say about sample size, is that its under a 100.
To say more, I'd have to explain the "widget" and how it is used, and due to confidentiality I can't reveal that.

I do appreciate all the comments and several people have contacted me at my private email with additional suggestions.

Again, thanks to all.

-------------------------------------------
Chris Barker
Statistical Planning and Analysis Services, Inc.
-------------------------------------------






Original Message:
Sent: 05-25-2011 12:46
From: Patrick Spagon
Subject: Confidence interval for a binomial proportion.

I've been following all the posts and appreciate the suggestions and the associated references. I was familiar with the Wikipedia article and like Michael's validation of it. 

It would be helpful if Christopher would give us an idea of the range of sample sizes to be expected.

Since this is a consulting section, I want to point out that Brian's suggestion of simulating data is an important one. Whenever I get a request for which the answer isn't clear, I simulate the data and try various methods that potentially apply. I also believe in conducting a sensitivity analysis: how sensitive are the various methods to the decision to be made. Finally, I apply the rule: we won't go to the next level of sophistication unless the business decision would be wrong otherwise.

Those guidelines have served me well over the years.

-------------------------------------------
Patrick Spagon
-------------------------------------------

I am glad that we were helpful even though we could only get partial information.

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------






Original Message:
Sent: 05-25-2011 13:05
From: Christopher Barker
Subject: Confidence interval for a binomial proportion.


Hi, the best I can say about sample size, is that its under a 100.
To say more, I'd have to explain the "widget" and how it is used, and due to confidentiality I can't reveal that.

I do appreciate all the comments and several people have contacted me at my private email with additional suggestions.

Again, thanks to all.

-------------------------------------------
Chris Barker
Statistical Planning and Analysis Services, Inc.
-------------------------------------------






Original Message:
Sent: 05-25-2011 12:46
From: Patrick Spagon
Subject: Confidence interval for a binomial proportion.

I've been following all the posts and appreciate the suggestions and the associated references. I was familiar with the Wikipedia article and like Michael's validation of it. 

It would be helpful if Christopher would give us an idea of the range of sample sizes to be expected.

Since this is a consulting section, I want to point out that Brian's suggestion of simulating data is an important one. Whenever I get a request for which the answer isn't clear, I simulate the data and try various methods that potentially apply. I also believe in conducting a sensitivity analysis: how sensitive are the various methods to the decision to be made. Finally, I apply the rule: we won't go to the next level of sophistication unless the business decision would be wrong otherwise.

Those guidelines have served me well over the years.

-------------------------------------------
Patrick Spagon
-------------------------------------------

Colleagues: we have this issue frequently as we make widgets (semiconductors) which requires quality level better than 1 PPM defective. I have a favorite method which comes from Trindade & Tobias, Applied Reliability, 2nd edition, Chapman & Hall/CRC 1995, page 264 (3rd edition is due in August) which looks like this:

PU= upper 1-alpha conf limit = C/ ((n-x)+C) where C = chisq(@(x+1), 1-alpha))/2; n= sample size, x= number of failures

he references an AMD internal memo that he wrote. He calls this "Confidence Intervals For Low PPM"



-------------------------------------------
Timothy Haifley
Altera Corporation
-------------------------------------------






Original Message:
Sent: 05-25-2011 13:05
From: Christopher Barker
Subject: Confidence interval for a binomial proportion.


Hi, the best I can say about sample size, is that its under a 100.
To say more, I'd have to explain the "widget" and how it is used, and due to confidentiality I can't reveal that.

I do appreciate all the comments and several people have contacted me at my private email with additional suggestions.

Again, thanks to all.

-------------------------------------------
Chris Barker
Statistical Planning and Analysis Services, Inc.
-------------------------------------------






Original Message:
Sent: 05-25-2011 12:46
From: Patrick Spagon
Subject: Confidence interval for a binomial proportion.

I've been following all the posts and appreciate the suggestions and the associated references. I was familiar with the Wikipedia article and like Michael's validation of it. 

It would be helpful if Christopher would give us an idea of the range of sample sizes to be expected.

Since this is a consulting section, I want to point out that Brian's suggestion of simulating data is an important one. Whenever I get a request for which the answer isn't clear, I simulate the data and try various methods that potentially apply. I also believe in conducting a sensitivity analysis: how sensitive are the various methods to the decision to be made. Finally, I apply the rule: we won't go to the next level of sophistication unless the business decision would be wrong otherwise.

Those guidelines have served me well over the years.

-------------------------------------------
Patrick Spagon
-------------------------------------------

Just checking I have the calculation right.

You have been given a wide variety of good information about binomial confidence intervals.  I would just like to add to the list.  I would like to put in some additional comments and references.  The erratic behavior of exact confidence intervals is due to the saw-toothed nature of the exact binomial since sample proportions are discrete random variables.  But if the true proportion is actually close to 0 or 1 the variance of the sample estimate is small [ p(1-p)/n ].  So the 1-sided Clopper-Pearson interval could be narrow and the issue is minor if n is large.  My article with Christine Liu in the American Statistician (2002) is a relevant article.  Also see the followup Letter to the Editor by Jeff Longmate.

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------






Original Message:
Sent: 05-25-2011 00:02
From: Christopher Barker
Subject: Confidence interval for a binomial proportion.

This message has been cross posted to the following eGroups: Biopharmaceutical Section and Statistical Consulting Section .
-------------------------------------------

Dear Colleagues, 

I'm working with a client who has a widget <confidentiality forbids me from revealing>  that leads to success (binomial) rates close to 100% or close to zero.

My client has asked me for the optimal choice of the confidence interval. My client is very knowledgeable, and having surfed the web, know that a Wald interval may not be the best choice.

I'd appreciate perspectives and opinions on the choice of the "best" (in quotes) confidence interval.

In advance, to name a few "Agresti Coull, Wald, Wilson, Jeffrey's, Blyth Still Casella (I have dozens of  articles and more on order).

• And in advance of asking, and that my client has an urgent priority, I have made my recommendations to the client. 

I'd appreciate more recommendations. :)

-

-thanks in advance
-------------------------------------------
Christopher Barker, Ph.D.
Statistical Planning and Analysis Services, Inc.

www.barkerstats.com
-------------------------------------------

Several years ago, I edited the Wikipedia page on this topic, and although it has changed a lot since then, it is still a fairly non-technical introduction to the topic. Of course, it does not go into the level of detail of any of the references that others have helpfully provided.

People either love or hate Wikpedia, but if your client is not too negative on it, the Wikipedia page might be worth showing to explain some of the options. In particular, it offers a derivation of some of the intervals as an inversion of various hypothesis tests.

http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval

-------------------------------------------
Stephen Simon
Independent Statistical Consultant
P. Mean Consulting
-------------------------------------------






Original Message:
Sent: 05-25-2011 00:02
From: Christopher Barker
Subject: Confidence interval for a binomial proportion.

This message has been cross posted to the following eGroups: Biopharmaceutical Section and Statistical Consulting Section .
-------------------------------------------

Dear Colleagues, 

I'm working with a client who has a widget <confidentiality forbids me from revealing>  that leads to success (binomial) rates close to 100% or close to zero.

My client has asked me for the optimal choice of the confidence interval. My client is very knowledgeable, and having surfed the web, know that a Wald interval may not be the best choice.

I'd appreciate perspectives and opinions on the choice of the "best" (in quotes) confidence interval.

In advance, to name a few "Agresti Coull, Wald, Wilson, Jeffrey's, Blyth Still Casella (I have dozens of  articles and more on order).

• And in advance of asking, and that my client has an urgent priority, I have made my recommendations to the client. 

I'd appreciate more recommendations. :)

-

-thanks in advance
-------------------------------------------
Christopher Barker, Ph.D.
Statistical Planning and Analysis Services, Inc.

www.barkerstats.com
-------------------------------------------

I took a look at the article that Dr. Simon is referring to.  Wikipedia varies in quality depending on the knowledge of the author and that is why some people don't trust it.  This article is very well written and authoritative and provides clear and correct descriptions of the intervals and their basic properties.  This is a good source for clients whether or not they are well-versed in statistics.
-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------






Original Message:
Sent: 05-25-2011 10:54
From: Stephen Simon
Subject: Confidence interval for a binomial proportion.

Several years ago, I edited the Wikipedia page on this topic, and although it has changed a lot since then, it is still a fairly non-technical introduction to the topic. Of course, it does not go into the level of detail of any of the references that others have helpfully provided.

People either love or hate Wikpedia, but if your client is not too negative on it, the Wikipedia page might be worth showing to explain some of the options. In particular, it offers a derivation of some of the intervals as an inversion of various hypothesis tests.

http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval

-------------------------------------------
Stephen Simon
Independent Statistical Consultant
P. Mean Consulting
-------------------------------------------






Original Message:
Sent: 05-25-2011 00:02
From: Christopher Barker
Subject: Confidence interval for a binomial proportion.

This message has been cross posted to the following eGroups: Biopharmaceutical Section and Statistical Consulting Section .
-------------------------------------------

Dear Colleagues, 

I'm working with a client who has a widget <confidentiality forbids me from revealing>  that leads to success (binomial) rates close to 100% or close to zero.

My client has asked me for the optimal choice of the confidence interval. My client is very knowledgeable, and having surfed the web, know that a Wald interval may not be the best choice.

I'd appreciate perspectives and opinions on the choice of the "best" (in quotes) confidence interval.

In advance, to name a few "Agresti Coull, Wald, Wilson, Jeffrey's, Blyth Still Casella (I have dozens of  articles and more on order).

• And in advance of asking, and that my client has an urgent priority, I have made my recommendations to the client. 

I'd appreciate more recommendations. :)

-

-thanks in advance
-------------------------------------------
Christopher Barker, Ph.D.
Statistical Planning and Analysis Services, Inc.

www.barkerstats.com
-------------------------------------------

Just to stir the pot a little more; if you have zero percent success in the data, you can use the value of 3/(n+1.8) as the approximate 95% confidence upper bound on the probability of success.  This is an improvement over the old rule of 3/n; this formula is correct to within 1% of the true value [Am Statistician 2002;56(3):252]
-------------------------------------------
Richard Browne
Texas Scottish Rite Hospital for Children
-------------------------------------------






Original Message:
Sent: 05-25-2011 00:02
From: Christopher Barker
Subject: Confidence interval for a binomial proportion.

This message has been cross posted to the following eGroups: Biopharmaceutical Section and Statistical Consulting Section .
-------------------------------------------

Dear Colleagues, 

I'm working with a client who has a widget <confidentiality forbids me from revealing>  that leads to success (binomial) rates close to 100% or close to zero.

My client has asked me for the optimal choice of the confidence interval. My client is very knowledgeable, and having surfed the web, know that a Wald interval may not be the best choice.

I'd appreciate perspectives and opinions on the choice of the "best" (in quotes) confidence interval.

In advance, to name a few "Agresti Coull, Wald, Wilson, Jeffrey's, Blyth Still Casella (I have dozens of  articles and more on order).

• And in advance of asking, and that my client has an urgent priority, I have made my recommendations to the client. 

I'd appreciate more recommendations. :)

-

-thanks in advance
-------------------------------------------
Christopher Barker, Ph.D.
Statistical Planning and Analysis Services, Inc.

www.barkerstats.com
-------------------------------------------

The old rule is called the rule of three and like so many rules of thumb it is explained nicely in
Statistical Rules of Thumb (2002) bt Gerald van Belle and published by Wiley.  For the benefit of members of this section van Belle has a wonderful final chapter (Chapter 8) on statistical consulting and rules of thumb for consultants.
-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------






Original Message:
Sent: 05-25-2011 11:11
From: Richard Browne
Subject: Confidence interval for a binomial proportion.


Just to stir the pot a little more; if you have zero percent success in the data, you can use the value of 3/(n+1.8) as the approximate 95% confidence upper bound on the probability of success.  This is an improvement over the old rule of 3/n; this formula is correct to within 1% of the true value [Am Statistician 2002;56(3):252]
-------------------------------------------
Richard Browne
Texas Scottish Rite Hospital for Children
-------------------------------------------

If we want to use the rule of three mentioned by Michael Chernick to determine a sample size using precision of the 1-sided exact binomial 95% confidence limit when we expect to see zero events (or zero non-events), and if we want to cloak this rule with an air of authority & give citations for it, some people in the biomedical field call it Hanley's Rule of Three, and two citations for it follow. 

The rule's first appearance in the biomedical literature was in the Journal of the American Medical Association, where it was published as:
Hanley JA, Lippman-Hand A. If nothing goes wrong, is everything all right? Interpreting zero numerators.  JAMA. 1983 Apr 1;249(13):1743-5.  (http://www.ncbi.nlm.nih.gov/pubmed/6827763).

In 1995, Eypasch et al made Hanley's Rule of Three a central point of their British Medical Journal article:
Eypasch E, Lefering R, Kum CK, Troidl H. Probability of adverse events that have not yet occurred: a statistical reminder.  BMJ. 1995 Sep 2;311(7005):619-20.  (http://www.ncbi.nlm.nih.gov/pubmed/7663258) (free article at http://www.bmj.com/content/311/7005/619.long) 


-------------------------------------------
Eric Siegel
Boistatistician
Univ of Arkansas for Medical Sciences
-------------------------------------------



Original Message:
Sent: 05-25-2011 11:28
From: Michael Chernick
Subject: Confidence interval for a binomial proportion.


The old rule is called the rule of three and like so many rules of thumb it is explained nicely in
Statistical Rules of Thumb (2002) bt Gerald van Belle and published by Wiley.  For the benefit of members of this section van Belle has a wonderful final chapter (Chapter 8) on statistical consulting and rules of thumb for consultants.
-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------

Here's another Rule of Three related reference. Winkler, Smith, and Fryback compare results from the Rule of Three with a Bayesian approach.


http://faculty.fuqua.duke.edu/~jes9/bio/The_Role_of_Informative_Priors_in_Zero_Numerator_Problems.pdf

-------------------------------------------
Patrick Spagon
-------------------------------------------






Original Message:
Sent: 05-25-2011 14:50
From: Eric Siegel
Subject: Confidence interval for a binomial proportion.

If we want to use the rule of three mentioned by Michael Chernick to determine a sample size using precision of the 1-sided exact binomial 95% confidence limit when we expect to see zero events (or zero non-events), and if we want to cloak this rule with an air of authority & give citations for it, some people in the biomedical field call it Hanley's Rule of Three, and two citations for it follow. 

The rule's first appearance in the biomedical literature was in the Journal of the American Medical Association, where it was published as:
Hanley JA, Lippman-Hand A. If nothing goes wrong, is everything all right? Interpreting zero numerators.  JAMA. 1983 Apr 1;249(13):1743-5.  (http://www.ncbi.nlm.nih.gov/pubmed/6827763).

In 1995, Eypasch et al made Hanley's Rule of Three a central point of their British Medical Journal article:
Eypasch E, Lefering R, Kum CK, Troidl H. Probability of adverse events that have not yet occurred: a statistical reminder.  BMJ. 1995 Sep 2;311(7005):619-20.  (http://www.ncbi.nlm.nih.gov/pubmed/7663258) (free article at http://www.bmj.com/content/311/7005/619.long) 


-------------------------------------------
Eric Siegel
Boistatistician
Univ of Arkansas for Medical Sciences
-------------------------------------------

Hmmm.  And that's a very interesting Section 4 in that article.  Thank you very much for it. 

-------------------------------------------
Eric Siegel
Boistatistician
Univ of Arkansas for Medical Sciences
-------------------------------------------


Original Message:
Sent: 05-25-2011 15:36
From: Patrick Spagon
Subject: Confidence interval for a binomial proportion.

Here's another Rule of Three related reference. Winkler, Smith, and Fryback compare results from the Rule of Three with a Bayesian approach.


http://faculty.fuqua.duke.edu/~jes9/bio/The_Role_of_Informative_Priors_in_Zero_Numerator_Problems.pdf

-------------------------------------------
Patrick Spagon
-------------------------------------------

Re "Statistical Rules of Thumb" by Gerald van Belle:

An exact, single-sided, 95% upper confidence interval for x failures out of a sample size of n is easily found using the inverse beta spreadsheet function = BETAINV(0.95, x+1, n-x). For zero failures, the function is simply =BETAINV(0.95, 1, n).

As an example, for a sample size of 40, the approximate formula given by Browne below yields a 95% UCL of 0.07177 and the exact formula gives 0.072158.
-------------------------------------------
David Trindade
Fellow
Bloom Energy
-------------------------------------------






Original Message:
Sent: 05-25-2011 11:11
From: Richard Browne
Subject: Confidence interval for a binomial proportion.


Just to stir the pot a little more; if you have zero percent success in the data, you can use the value of 3/(n+1.8) as the approximate 95% confidence upper bound on the probability of success.  This is an improvement over the old rule of 3/n; this formula is correct to within 1% of the true value [Am Statistician 2002;56(3):252]
-------------------------------------------
Richard Browne
Texas Scottish Rite Hospital for Children
-------------------------------------------

Yes most of us know how to compute exact binomial confidence intervals by the Clopper-Pearson Method.  That is not the issue.  The issue is which method has the most desirable properties to please the client. Because of the nature of discrete distributions the exact binomial test can have power go down with an increase in sample size (sawtoothed power function ).  This may not be intuitively appealing to a client and difficult to explain in a way that they can understand. As a consequence the width of a confidence interval can also widen with increased sample size.
-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------






Original Message:
Sent: 05-25-2011 13:50
From: David Trindade
Subject: Confidence interval for a binomial proportion.

An exact, single-sided, 95% upper confidence interval for x failures out of a sample size of n is easily found using the inverse beta spreadsheet function = BETAINV(0.95, x+1, n-x). For zero failures, the function is simply =BETAINV(0.95, 1, n).

As an example, for a sample size of 40, the approximate formula given by Browne below yields a 95% UCL of 0.07177 and the exact formula gives 0.072158.
-------------------------------------------
David Trindade
Fellow
Bloom Energy
-------------------------------------------

While the power can go down (sawtooth function) for the exact binomial confidence test. the statement that the confidence interval can widen with increased sample size needs clarification.  For the same binomial population p value, the width of the interval will not widen with increased sample size. So, for example, if the population p value is say 10%, then 1 failure out of 10 will have a wider width than 2 failures out of 20, and so on. So, increasing the sample size to measure the same population p value will produce narrower widths, that is, more precise estimates.
-------------------------------------------
David Trindade
Fellow
Bloom Energy
-------------------------------------------






Original Message:
Sent: 05-25-2011 14:44
From: Michael Chernick
Subject: Confidence interval for a binomial proportion.


Yes most of us know how to compute exact binomial confidence intervals by the Clopper-Pearson Method.  That is not the issue.  The issue is which method has the most desirable properties to please the client. Because of the nature of discrete distributions the exact binomial test can have power go down with an increase in sample size (sawtoothed power function ).  This may not be intuitively appealing to a client and difficult to explain in a way that they can understand. As a consequence the width of a confidence interval can also widen with increased sample size.
-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------

Your argument is flawed. While a confidence interval for p when there are 2 failures in 20 may be narrower than one that has 1 failure in 10 that does not imply that as the sample size increases from 10 to 20 it always increases.  The reason the power function is sawtoothed is because it can be shown that when the sample size increases from n to n+1 for some n the power decreases.  Similarly to say that the width of a confidence interval increases means that for some n a failure (or success) on the next observation causes the width of te interval to widen.  Because the binomial distribution is discrete it is not always possible to construct a confidence interval with say exactly 95% coverage.  So the definition of a 95% confidence interval is a random interval that will in repeated samplling covered the true parameter in at least 95% of the cases.  So to see why the width of a 95% confidence interval can widen as n increases consider the fact that for some n and number of successes s an exact 95% confidence interval can be constructed but for n+1 it cannot. It may happen that if say we have a failure with the next observation that an exact 95% confidence interval for s successes out of n+1 observations is not possible.  Say to obtain at least 95% coverage you must construct an exact 98% confidence interval.  Since the required exact coverage went up 3% a wider interval could be needed.  This can be illustrated with a specific numerical example but I think you probably get the idea.
-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------






Original Message:
Sent: 05-26-2011 20:04
From: David Trindade
Subject: Confidence interval for a binomial proportion.


While the power can go down (sawtooth function) for the exact binomial confidence test. the statement that the confidence interval can widen with increased sample size needs clarification.  For the same binomial population p value, the width of the interval will not widen with increased sample size. So, for example, if the population p value is say 10%, then 1 failure out of 10 will have a wider width than 2 failures out of 20, and so on. So, increasing the sample size to measure the same population p value will produce narrower widths, that is, more precise estimates.
-------------------------------------------
David Trindade
Fellow
Bloom Energy
-------------------------------------------

Yes, but his argument is nonetheless correct under the conditions he specified, namely, where the numerator k and denominator n both increase in a way that maintains constant proportion p.

-------------------------------------------
Eric Siegel
Boistatistician
Univ of Arkansas for Medical Sciences
-------------------------------------------



Original Message:
Sent: 05-26-2011 20:57
From: Michael Chernick
Subject: Confidence interval for a binomial proportion.


Your argument is flawed. While a confidence interval for p when there are 2 failures in 20 may be narrower than one that has 1 failure in 10 that does not imply that as the sample size increases from 10 to 20 it always increases.  The reason the power function is sawtoothed is because it can be shown that when the sample size increases from n to n+1 for some n the power decreases.  Similarly to say that the width of a confidence interval increases means that for some n a failure (or success) on the next observation causes the width of te interval to widen.  Because the binomial distribution is discrete it is not always possible to construct a confidence interval with say exactly 95% coverage.  So the definition of a 95% confidence interval is a random interval that will in repeated samplling covered the true parameter in at least 95% of the cases.  So to see why the width of a 95% confidence interval can widen as n increases consider the fact that for some n and number of successes s an exact 95% confidence interval can be constructed but for n+1 it cannot. It may happen that if say we have a failure with the next observation that an exact 95% confidence interval for s successes out of n+1 observations is not possible.  Say to obtain at least 95% coverage you must construct an exact 98% confidence interval.  Since the required exact coverage went up 3% a wider interval could be needed.  This can be illustrated with a specific numerical example but I think you probably get the idea.
-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------

Okay.  I interpreted David's argument as one that was trying to refute the notion that the width of a confidence interval for a binomial proportion cannot increase with increasing sample size.  If that wasn't his intention I apologize.  The statement he made may be correct but I don't see exactly how it relates to the discussion.  I hope i clarified why binomial confidence intervals can widen with increasing sample size which is what I think he asked me to do.
-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------






Original Message:
Sent: 05-26-2011 21:35
From: Eric Siegel
Subject: Confidence interval for a binomial proportion.

Yes, but his argument is nonetheless correct under the conditions he specified, namely, where the numerator k and denominator n both increase in a way that maintains constant proportion p.

-------------------------------------------
Eric Siegel
Boistatistician
Univ of Arkansas for Medical Sciences
-------------------------------------------

It was not my intention to refute the notion that the width of a confidence interval for a binomial proportion cannot increase with increasing sample size. I understand that concept perfectly well. What I was describing is the situation in which we are trying to estimate a fixed population proportion. If the number of failures x and the  sample size n increase in such a way that the ratio p=x/n remains constant, then the width of the exact confidence interval will not widen with increasing sample size.   

-------------------------------------------
David Trindade
Fellow
Bloom Energy
-------------------------------------------






Original Message:
Sent: 05-26-2011 21:51
From: Michael Chernick
Subject: Confidence interval for a binomial proportion.


Okay.  I interpreted David's argument as one that was trying to refute the notion that the width of a confidence interval for a binomial proportion cannot increase with increasing sample size.  If that wasn't his intention I apologize.  The statement he made may be correct but I don't see exactly how it relates to the discussion.  I hope i clarified why binomial confidence intervals can widen with increasing sample size which is what I think he asked me to do.
-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------

I guess I don't understand your point.  Why would anyone be interested in how the interval changes in that way.  Also why did you say the following?

"While the power can go down (sawtooth function) for the exact binomial confidence test. the statement that the confidence interval can widen with increased sample size needs clarification. For the same binomial population p value, the width of the interval will not widen with increased sample size."

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------






Original Message:
Sent: 05-27-2011 01:22
From: David Trindade
Subject: Confidence interval for a binomial proportion.

It was not my intention to refute the notion that the width of a confidence interval for a binomial proportion cannot increase with increasing sample size. I understand that concept perfectly well. What I was describing is the situation in which we are trying to estimate a fixed population proportion. If the number of failures x and the  sample size n increase in such a way that the ratio p=x/n remains constant, then the width of the exact confidence interval will not widen with increasing sample size.   

-------------------------------------------
David Trindade
Fellow
Bloom Energy
-------------------------------------------






Original Message:
Sent: 05-26-2011 21:51
From: Michael Chernick
Subject: Confidence interval for a binomial proportion.


Okay.  I interpreted David's argument as one that was trying to refute the notion that the width of a confidence interval for a binomial proportion cannot increase with increasing sample size.  If that wasn't his intention I apologize.  The statement he made may be correct but I don't see exactly how it relates to the discussion.  I hope i clarified why binomial confidence intervals can widen with increasing sample size which is what I think he asked me to do.
-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------

Let's take a simple example.  Supppose we are interested in a single-sided binomial test of H0: p = 0.1 versus the alternative Ha: p> 0.1. We'll use a 5% significance level. For a sample size of 40, the critical number of rejects is 7 (acceptance number is 6) with actual probability of 7 or more rejects being 0.042.  For a larger sample size of 50, the critical number of rejects is 9 (acceptance number is 8) with probability of 9 or more rejects at 0.025. The power is less for the larger sample size. It is easy to show for this situation that a sample size of 50 will have less power than a sample size of 40 for all alternatives p> 0.1.  Consider now sampling from a population with p = 0.1. For a sample size of 40 or 50, the expected number of rejects will be 4 or 5, respectively. The single sided exact binomial upper confidence limit for 4 rejects out of 40 is 0.214,  The same upper confidence limit for 5 rejects out of 50 is 0.199.  Thus, even though the power is less for the larger sample size, the confidence limit is not wider. I hope this example clarifies the point I was trying to make.      
-------------------------------------------
David Trindade
Fellow
Bloom Energy
-------------------------------------------






Original Message:
Sent: 05-27-2011 06:23
From: Michael Chernick
Subject: Confidence interval for a binomial proportion.

I guess I don't understand your point.  Why would anyone be interested in how the interval changes in that way.  Also why did you say the following?

"While the power can go down (sawtooth function) for the exact binomial confidence test. the statement that the confidence interval can widen with increased sample size needs clarification. For the same binomial population p value, the width of the interval will not widen with increased sample size."

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------

If I understand you correctly this is something completely different from what you had previously said.
You are saying that there are situations where the power of the test decreases and yet the confidence interval get narrower.  But the example you give is confusing.  It looks like 0.042 and 0.025 are the respective type 1 errors (i.e. probability of reject the null hypothesis when p=0.1).  But power is 1-type II error.  Where have you computed the type II error to show that the power goes down?

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------






Original Message:
Sent: 05-27-2011 14:06
From: David Trindade
Subject: Confidence interval for a binomial proportion.


Let's take a simple example.  Supppose we are interested in a single-sided binomial test of H0: p = 0.1 versus the alternative Ha: p> 0.1. We'll use a 5% significance level. For a sample size of 40, the critical number of rejects is 7 (acceptance number is 6) with actual probability of 7 or more rejects being 0.042.  For a larger sample size of 50, the critical number of rejects is 9 (acceptance number is 8) with probability of 9 or more rejects at 0.025. The power is less for the larger sample size. It is easy to show for this situation that a sample size of 50 will have less power than a sample size of 40 for all alternatives p> 0.1.  Consider now sampling from a population with p = 0.1. For a sample size of 40 or 50, the expected number of rejects will be 4 or 5, respectively. The single sided exact binomial upper confidence limit for 4 rejects out of 40 is 0.214,  The same upper confidence limit for 5 rejects out of 50 is 0.199.  Thus, even though the power is less for the larger sample size, the confidence limit is not wider. I hope this example clarifies the point I was trying to make.      
-------------------------------------------
David Trindade
Fellow
Bloom Energy
-------------------------------------------

I'm not sure if we should take this discussion off line.  When the alternative p value is the same as the null value, the power is the same as the type I error. I can send you the spreadsheet with my calculations.  In any case, here are my power values for the example I used.

alternative p		0.1	0.11	0.12	0.13	0.14	0.15	0.16	0.17	0.18	0.19	0.2
SS=40	Power	4.2%	6.7%	10.0%	14.1%	18.9%	24.4%	30.4%	36.8%	43.4%	49.9%	56.3%
SS=50	Power	2.5%	4.3%	7.1%	10.7%	15.4%	20.9%	27.2%	34.0%	41.2%	48.5%	55.6%

-------------------------------------------
David Trindade
Fellow
Bloom Energy
-------------------------------------------






Original Message:
Sent: 05-27-2011 14:36
From: Michael Chernick
Subject: Confidence interval for a binomial proportion.

If I understand you correctly this is something completely different from what you had previously said.
You are saying that there are situations where the power of the test decreases and yet the confidence interval get narrower.  But the example you give is confusing.  It looks like 0.042 and 0.025 are the respective type 1 errors (i.e. probability of reject the null hypothesis when p=0.1).  But power is 1-type II error.  Where have you computed the type II error to show that the power goes down?

-------------------------------------------
Michael Chernick
Director of Biostatistical Services
Lankenau Institute for Medical Research
-------------------------------------------