Discussion: View Thread

Back to discussions
Expand all | Collapse all

Difference of Two Poissons

  • 1.  Difference of Two Poissons

    Posted 11-22-2011 15:23
    Let's say there are two rates for similar but not competing events.  An example would be accidents per a day on two equal lengths of different highways in the same state over a year.   They might be correlated but the accidents occur separately.    What is the t-test analog for the equality of the two means, i.e., lambda?  Also is there a theoretical distribution for a pairing such as taking the difference?  Since the difference can be negative, it wouldn't be Poisson.      

    -------------------------------------------
    Georgette Asherman
    -------------------------------------------


  • 2.  RE:Difference of Two Poissons

    Posted 11-22-2011 15:26
    Something similar came up recently either on this group e-mail list or on another one I read, and someone mentioned the Skellam distribution (eg, http://en.wikipedia.org/wiki/Skellam_distribution) is for the difference of two Poissons. Sounds like that's what you're looking for here, right?


    -------------------------------------------
    Gabriel Farkas
    -------------------------------------------








  • 3.  RE:Difference of Two Poissons

    Posted 11-22-2011 15:34

    I remember the posting about the Skellem distribution but I think that involves the difference of two independent Poissons.  If the traffic accidents are rare events that clearly follow an expoential arrive rate (Poisson process) then I think the parameteric approach should be used rather than the nonparametric approach.  But somehow the correlation must be modeled or ignored (assume independence).
    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 4.  RE:Difference of Two Poissons

    Posted 11-22-2011 17:23
    The original post mentioned that traffic accidents were just an example, not the actual problem at hand. The information we were given was "two rates for similar but not competing events" and "might be correlated but the accidents occur separately".

    Given that information, I suggested the Skellam, since for two statistically independent random Poissons with different expected values mu1 and mu2, the Skellam gives the distribution of x1 - x2. This seemed to satisfy the "similar but not competing events" stipulation.

    Also, the assumption of independence seems like: (a) the path of least resistance, and (b) probably the most logical assumption to make.

    Even if we don't want to assume independence, the Skellam also can apply for the difference of two dependent Poissons if they have a common random contribution... so even if we want to model the correlation, I think the Skellam could still be applicable.



    -------------------------------------------
    Gabriel Farkas
    -------------------------------------------






    Original Message


  • 5.  RE:Difference of Two Poissons

    Posted 11-22-2011 17:51


    -------------------------------------------
    Sheela Talwalker
    CSO, T'Walker Consulting
    -------------------------------------------
    I agree with Gabriel's message below. This is what I had suggested during the last round of similar discussion. I had also supplied the references





    Original Message


  • 6.  RE:Difference of Two Poissons

    Posted 11-22-2011 20:10

    I think I get your point but I don't think you said it right.  It would be an additive effect such as a seasonal effect that causes the correlation not the random part because they have to cancel out through subtraction.

    I got the impression that highways is the real example and if she said 'for example" it was probably just a figure of speech. 

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 7.  RE:Difference of Two Poissons

    Posted 11-23-2011 11:37

    There's another, simpler approach, if you can accept the assumption of independence and also that the traffic levels on both roads are the same.  In that case, you have two Poisson processes with parameters lambda1 and lambda2 and if we call the number of events observed in the observation period n1 and n2, then it is easy to show that n1 is distributed as Binomial(lambda1/(lambda1+lambda2), n1+n2).  Evidently this is only really useful to test hypotheses about the ratio between lambda1 and lambda2, and you really have to believe that the data generating processes are Poisson.  But it might be useful for your particular situation.
    -------------------------------------------
    Clyde Schechter
    -------------------------------------------






    Original Message


  • 8.  RE:Difference of Two Poissons

    Posted 11-23-2011 11:46

    What you are saying can't be right.  You must mean that you are conditioning on the total number of events equaling n1+n2.  You may have thought that was obvious but I had to think a while before realizing it.
    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 9.  RE:Difference of Two Poissons

    Posted 11-23-2011 12:28
      |   view attached
    This is an old problem; see attached.  If the Poisson means are equal then the then the counts from the two distributions follow a binomial with p=0.5 and N = N1+N2. 

    -------------------------------------------
    Michael Ginevan
    M.E. Ginevan & Associates
    -------------------------------------------






    Attachment(s)

    PDF
    Poisson trials paper.PDF   1.02 MB 1 version
    Original Message


  • 10.  RE:Difference of Two Poissons

    Posted 11-23-2011 13:26
    I hate to be called a nitpicker (my wife calls me that at times) but you too left out the requirement that you condition on the total being N.  In your paper when you cite the result you say that you assume the total number of events is T and then refer to your equation 8.  Also probably obvious but left unsaid is how you test to determine if the two rate parameters are equal.  Since p=1/2 when and only when the rate parameters are equal you can test whether or not the binomial proportion equals 1/2.  Rejecting p=1/2 is then equivalent to rejecting lambda1=lambda2 and not rejecting means you can't reject lambda1=lambda2.

    I think this is a rather clever way to address the problem without involving the Skellam distribution. But in the case of the Skellam distribution we had to assume the two Poisson were independent which is probably the case in your probelm where you are comparing deaths in treatment and control groups.  But that is not the case in the traffic example.  Nevertheless I think the correlation issue is finessed here because of the conditioning on the total N because that changes the relationship between N1 and N2.  Now N1 is binomial and N2=N-N1.  It is completely determined when given N and N1. So unlike the problem of drawing inference using the Skellam distribution the binomial applies here regardless of the correlation between N1 and N2 and hence solves the traffic problem without complicated modeling of the correlation structure.

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 11.  RE:Difference of Two Poissons

    Posted 11-23-2011 15:28
    Interesting-looking article.  But I don't see the journal name anywhere.  What journal was it published in?


    -----------------------------------------
    Eric Siegel
    Biostatistician
    Univ of Arkansas for Medical Sciences
    -------------------------------------------






    Original Message


  • 12.  RE:Difference of Two Poissons

    Posted 11-22-2011 15:29

    Hi Georgette

    In the example you give, I think you'd want to account for number of cars on each road; then this is likely to be testable with a t-test.

    More generally, I think you could use something like Mann Whitney U; and you could always use a permutation type test



    -------------------------------------------
    Peter Flom
    -------------------------------------------








  • 13.  RE:Difference of Two Poissons

    Posted 11-22-2011 15:42
    Well, I'd start with Poisson regression. It models the ratio rather than the difference, but you might find the ratio to be a more natural parameter. It also handles the fact that there is a relationship between the mean and the variance for this type of data.

    The data is paired, so include a random effect for day. Random effects are tricky here, so be careful. Also be careful about overdispersion.

    Someone else noted traffic volume as a critical factor. If you have information about such a variable, it can be incorporated into an offset variable.

    -------------------------------------------
    Stephen Simon
    Independent Statistical Consultant
    P. Mean Consulting
    -------------------------------------------








  • 14.  RE:Difference of Two Poissons

    Posted 11-23-2011 16:39
    Georgette,

    What you can do depends on what you have, and it's not clear to me whether you have the detailed day-to-day traffic-accident data, or whether you have only the two Poisson lambda estimates averaged across the whole year, or whether you have something intermediate between those two extremes. 

    If you have the detailed day-to-day traffic-accident data, then Poisson regression, which Stephen Simon recommended, is probably superior to pretty much everything else. The pairing between highways is a pairing on Day, which in SAS Proc Genmod can be accomodated with a Repeated statement.  

    If you don't have that level of detail, but you still have something intermediate, like maybe monthly average accident rates, then Poisson regression is still the way to go.

    But if all you have are the total accidents for the year on each stretch of highway, and that is the source of your two Poisson lambda estimates, then you can still compare them, and do so using a two-sided version of the F-test.  The trick is to recognize the following: 

    If the Average Accident Rate (AAR) for the year is computed as the total accidents N divided by the total time of observation T, and if AAR estimates the mean or lambda of a Poisson distribution, then its inverse, 1/AAR = T/N, is the Mean Time Between Accidents (MTBA), and MTBA estimates the Mean or beta parameter of an Exponential distribution.  The MTBA here is exactly analogous to the MTTF (Mean Time To Failure) from reliability testing.  In reliability testing, they compare the means of two Exponential distributions by forming the ratio R=MTTF(1)/MTTF(2), and they test H0:R=1 versus HA:R<>1 by comparing R against the F distribution with appropriate degrees of freedom, using both tails of the F distribution since it's a two-sided test.  The PASS software package uses this F test as the basis of its power & sample-size calculation for experiments that compare the means of two Exponential distributions. 

    The exact same F-test procedure can be implemented here to compare the MTBAs between Highway #1 and Highway #2.  And since MTBA=1/AAR, the F-test comparing MTBAs automatically compares Poisson lambda estimates.  

    F-test Degrees of Freedom:  Let MTBA(1) = T(1)/N(1), and let MTBA(2) = T(2)/N(2), where N(1) and N(2) are the number of accidents observed during times T(1) and T(2), respectively.  Let F = MTBA(1)/MTBA(2).  Then F follows the F-distribution with numerator Degrees of Freedom = 2N(1) and denominator Degrees of Freedom = 2N(2).     

    The first time I ever tried this F-test procedure, my Poisson lambda estimates were crude mortality ratios of two populations, and I happened to have the results of a Cox regression comparing the same two populations.  The F-test and the Cox regression gave remarkably similar results. 


    -------------------------------------------
    Eric Siegel
    Biostatistician
    Univ of Arkansas for Medical Sciences
    -------------------------------------------








  • 15.  RE:Difference of Two Poissons

    Posted 11-23-2011 17:09

    Georgette can answer for herself.  I thought that since she wants to compare two Poisson processes she has all the accidents with time of occurrence.  But given the trick with the binomial and the fact that she only wants to test the difference between the lambdas, the binomial test does it and the Poisson regression is unnecessary even though the may allow her to do it.
    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message