Discussion: View Thread

Back to discussions
Expand all | Collapse all

Multiple measures above 5% standard: overall test?

  • 1.  Multiple measures above 5% standard: overall test?

    Posted 09-07-2012 11:34
    This question comes to be from a friend; I work in marketing applications rather than medical, but I know many on this list are biostatisticians who might be able to advise as to current practice:
     
    "We have a group of 78 workers exposed to chemicals.  We have blood tests for 8 toxic metals like lead, mercury, cadmium and arsenic.  We have data for the number of workers in the top 5% of a "standard population" blood levels for the various metals- many of the workers are in the top 5% of metal levels in serum.  About 30 workers have lead levels in the top 5% of the standard for lead, 25 workers in top 5% for mercury etc.
     
    "My question is this- how can we look at the data for all 8 tests and determine if the overall  toxic metal load levels are statistically elevated relative to the "standard population"?  Should we use a multiple correlation method for like Boniferris method- to determine if the overall levels of metals are excessive.  Or are my statistics getting rusty?"
     
     


    -------------------------------------------
    Michael Kruger
    Information Resources Inc
    -------------------------------------------


  • 2.  RE:Multiple measures above 5% standard: overall test?

    Posted 09-07-2012 13:58
    I'm guessing that the fact that someone tests positive for one of these metals is highly correlated
    with that person testing positive for another.  The Bonferroni adjustment is more typically used
    to guard against seizing on a single significant test when many unrelated ones have been performed (because
    even if you have a 20-sided die, if you roll it enough times, the 20-side will come up by random chance).

    From my perspective, a more reasonble test would be based on how unusual it is for someone
    to be over a certain threshold in any (i.e., at least one) of these metals.  At the very least, that's
    30 of the workers, or 38% at or above the threshold for 5% of the "standard" population.

    -------------------------------------------
    Katherine Godfrey
    -------------------------------------------








  • 3.  RE:Multiple measures above 5% standard: overall test?

    Posted 09-07-2012 14:44
    I think Katherine's advice is good.  If you want to take the multiple testing approach there are many other options better than Bonferroni. Also FDR can be considered as the criterion rather than FWER.  There are bootstrap and permutation approaches to adjusting p-values.  As Katherine point out some of the tests are probably highly positively correlated.  If that is the case the p-value adjustment should be less severe than it would be if all the tests were completely independent of each other.  Perhaps therre is a way to model the correlation to modify the calculation of FDR or FWER.  I also think there may be a way to incorporate the correlation trhough bootstrapping.

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 4.  RE:Multiple measures above 5% standard: overall test?

    Posted 09-07-2012 14:41
    For the moment ignore the fact that tests within workers are not indepedent
    (if I'm high for lead I'm more likely to be high for mercury) and that tests
    across workers are not independent (if I'm exposed to zinc, the guy next to me
    probably is as well). There are 78 workers and 8 metals for 624 total tests.

    The crudest method you could use to get an 'overall' test of exposure would
    be to treat the tests as all independent with probability of success 0.05
    and count a person's test as a 'success' if they fell in the top 5% of the
    reference 'standard population' for a given metal. Under these assumptions
    the total number of workers in the top 5% over all tests would be distributed
    bin(624, 0.05), so you could get a single p-value from that.

    For example, 20 workers total in the top 5% over all tests would be p = 0.98, but
    >= 40 positive worker-tests would be p < 0.05; >= 60 would be p < 0.0000007, and
    >= 100 would be p < 5 x10^-25.

    -------------------------------------------
    John Dawson
    Postdoctoral Scholar
    University of Alabama at Birmingham
    -------------------------------------------








  • 5.  RE:Multiple measures above 5% standard: overall test?

    Posted 09-07-2012 14:50

    Hi, Michael,

    Do you mean Bonferroni? I don't think that applies here. This is not a question of multiple comparisons that Bonferroni addresses. You are not hypothesizing with alpha=0.05 that the measured levels are >0 or that the difference between measured scores of any two metals is > 0. You already know whether the measured values occur <=5% of the time in the "standard population" or not. That's not a hypothesis you are testing. I believe what you are asking is "Is my sample of workers consistent with a sample of the same size drawn from the "standard population?" In other words, "Do I have evidence that high metal scores are occurring in my workers more frequently than I would expect simply by chance alone?"

    If your 78 workers represented a random sample for the "standard population" then you can expect about 4 (0.05*78=3.9 workers) to have a blood score in the top 5% for any given metal. Further, if exposure to these metals in the standard population is independent, i.e. if one is exposed to mercury it is not also likely that one is exposed to arsenic, e.g., then you can expect less than one worker (0.05^2*78=0.2 workers) to have blood scores in the top 5% for any two metals and essentially none to have blood scores in the top 5% for three or more metals. This is the null hypothesis: that the workers represent a random draw from the "standard population" and that exposure to these chemicals in the "standard population" is independent.

    If the data you present are real, then the fact that 30 out of 78 workers have high lead levels when you expect only 4 to have is very strong reason to believe they do not represent a random sample from the "standard population." To observe 30 out of 78 when you expect only 4 is pretty close to impossible due to chance alone. The logical next step is, knowing that their work environment exposes them to lead, to infer that this non-random elevated sample is likely due to that exposure. You can't prove it, but it is definitely a reasonable inference given the data and your knowledge of potential exposure.

    I don't know how many of the 25 workers with high mercury levels also had high lead levels. But in many ways your numbers are already so large it doesn't much matter. You have 25 workers out of 78 in the top 5% when you expect only 4. Again, to observe 30 out of 78 when you expect only 4 out of 78 is pretty close to impossible due to chance alone.

    Hope this helps.

    -------------------------------------------
    Manuela Huso
    Research Statistician
    US Geological Survey
    -------------------------------------------








  • 6.  RE:Multiple measures above 5% standard: overall test?

    Posted 09-07-2012 14:52
    Right, this was my line of thinking as well, only that there's 78*8 tests, not just 78.

    -------------------------------------------
    John Dawson
    Postdoctoral Scholar
    University of Alabama at Birmingham
    -------------------------------------------






    Original Message


  • 7.  RE:Multiple measures above 5% standard: overall test?

    Posted 09-07-2012 14:59
    I am quite sure the other Michael did mean Bonferroni.  Multiplicity adjustment does have a place in this problem as a large number 78x8=624 tests are being done simultaneously.  But as Katherine pointed out given the correlations FWER with Bonferroni pays too high of a penalty to be useful.  But my suggestion for adjusting the p-value for FDR using a bootstrap approach might be practical and taking the correlation into account would be helpful.

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 8.  RE:Multiple measures above 5% standard: overall test?

    Posted 09-07-2012 14:59

    Actually, I beleive there are only 8 tests, one for each metal, not one for each person, and the tests are simply "Does > 5% of my worker population exhibit levels of X metal > the 95th quantile for the reference population?"  
     
    -------------------------------------------
    Manuela Huso
    Research Statistician
    US Geological Survey
    -------------------------------------------






    Original Message


  • 9.  RE:Multiple measures above 5% standard: overall test?

    Posted 09-07-2012 15:07
    Refer to this line:

    "We have data for the number of workers in the top 5% of a "standard population"
    blood levels for the various metals- many of the workers are in the top 5% of metal levels in serum."

    I read this as: For the various metals we have a standard population upper 5% level and we
    also have counted how many of our 78 workers are above this level for each metal.

    If that reading is correct, then there's 78*8 tests. Perhaps the original poster can clarify this point?

    -------------------------------------------
    John Dawson
    Postdoctoral Scholar
    University of Alabama at Birmingham
    -------------------------------------------






    Original Message


  • 10.  RE:Multiple measures above 5% standard: overall test?

    Posted 09-07-2012 15:39
    There is obviously confusion about what the number of tests are. But whatever it is it seems we all agree that there are multiple tests and some may be positively correlated. Some sort of multiplicity adjustment is warranted whether there are 8 tests or 624.  But the number may dictate the approach to mutliplicity adjustment.

    -------------------------------------------
    Michael Chernick
    Director of Biostatistical Services
    Lankenau Institute for Medical Research
    -------------------------------------------






    Original Message


  • 11.  RE:Multiple measures above 5% standard: overall test?

    Posted 09-07-2012 23:48
    I'll double-check, but I took this as 78 * 8 observations.  If they just looked at one metal (mercury, say) then one would expect 5% of the 78 observations to be above the 5% threshold (as Manuela notes) and one could use a chi-square test on this. But there are 8 tests.





    -------------------------------------------
    Michael Kruger
    Information Resources Inc
    -------------------------------------------

     




    Original Message


  • 12.  RE:Multiple measures above 5% standard: overall test?

    Posted 09-08-2012 16:54
    I have a question that's dumb in the sense that the answer should be obvious, but I'm asking it anyway to make sure:  When they assay a worker's blood sample for the concentrations of the eight metals, can they assay for all eight metals at once in the same sample?  Or are they limited to testing for only one or two metals at a time in any one sample from the worker? 

    I also was wondering if any of the 78 workers were tested more than once over time for the same metal?   

    -------------------------------------------
    Eric Siegel
    Biostatistician
    Univ of Arkansas for Medical Sciences
    -------------------------------------------



    Original Message


  • 13.  RE:Multiple measures above 5% standard: overall test?

    Posted 09-08-2012 18:41
    Hi all, I have not read the entire thread on this interesting topic, so perhaps this was mentioned before.

    I would like to suggest some visualizing of the data. Cluster analysis would be helpful. You can cluster people, with the metals as the dependent variables. Also, do principal components on the data (normalize the variables first) and do some scatterplots of pairs of components (first vs. second, first vs. third etc.) By the cluster analysis or by the principal compenets plots do you see some separation of the groups? These displays should give you a sense of how distinct (or indistinct) the groups are. 

    Best wishes,

    Nayak



    -------------------------------------------
    Nayak Polissar
    Principal Statistician
    The Mountain-Whisper-Light Statistics
    -------------------------------------------






    Original Message


  • 14.  RE:Multiple measures above 5% standard: overall test?

    Posted 09-08-2012 20:43
    Since exposures to the various metals for a single worker are likely to be correlated, an exact calculation would have to take the correlation into account.  However, I agree with Manuelo, that it doesn't take much calculation to see that the data indicate higher levels than the reference population. 

    For one of the metals alone there are only about 4 chances in 10^14 to have 25 out of 78 above the 5% point by chance.  Of course 30 out of 78 is much less likely still.

    I am assuming the reference distribution is reasonably accurate.  However, even if the 5% point were really the 20% point, the results you described would be unusual.

    -------------------------------------------
    Michael Morton
    -------------------------------------------






    Original Message


  • 15.  RE:Multiple measures above 5% standard: overall test?

    Posted 09-07-2012 15:38
    My personal opinion is, this sounds like a situation in which Type II error is much more costly than Type I error.  The regulatory consequences of Type I error could put the workers out of work unnecessarily, whereas the health consequences of Type II error could put the workers into the hospital or the grave.  Accordingly, I would actually propose the following:  Raise alpha to 10% despite the multiple testing, in order to reduce Type II error as much as is practicable.  

    -------------------------------------------
    Eric Siegel
    Biostatistician
    Univ of Arkansas for Medical Sciences
    -------------------------------------------





  • 16.  RE:Multiple measures above 5% standard: overall test?

    Posted 09-10-2012 10:59
    It seems pointless to argue over how to do a precise calculation when simple statistical common sense says that resulting p-value will be very small.  30/78 in the top 5% for lead and 25/78 in the top 5% for mercury doesn't occur by chance alone.  Perhaps these people are being exposed to more lead and mercury than the general population.  Perhaps there is a problem with the testing.  Perhaps the local population (whether your workers or not) are being exposed to significantly more lead and mercury than the general population.

    However, you might also want to ask whether their blood levels are dangerous.  If the 95th percentile is a danger level, then millions of people in the general population are in danger.  Since that seems unlikely, your friend may want to find out whether there is a published danger level and find out what that level is.

    -------------------------------------------
    Emil M Friedman, PhD
    emil.friedman@alum.mit.edu (forwards to day job)
    emilfrie@alumni.princeton.edu (home)
    http://www.statisticalconsulting.org
    -------------------------------------------








  • 17.  RE:Multiple measures above 5% standard: overall test?

    Posted 09-10-2012 16:33
    The reference ranges are univariate; the question is multivariate.
    Suppose the workers are over-exposed to lead but not others. It could still be a safety concern.
    Adjustments for multiple comparisons might lead one say that there is no "significant" over-exposure to lead.
    Ask and answer each univariate question.

    I like the idea of "dangerous" cut-offs. This would, to some extent, compensate for the inflated Type I Error.
    A summary table of # of  abnormals would be informative,


    -------------------------------------------
    David Bristol
    Statistical Consulting Services
    -------------------------------------------






    Original Message