Concept: Multiple Comparisons

 Printer friendly

Concept Description

Last Updated: 2008-04-21

    When performing κ multiple independent significance tests each at the α level, the probability of making at least one Type I error (rejecting the null hypothesis inappropriately) is 1-(1-α) κ . For example, with κ=10 and α=0.05, there is a 40% chance of at least one of the ten tests being declared significant under the null hypothesis.

    So, when you see a significant result among the ten tests, how confident can you be that it is "really" significant? There is a 40% chance that something will turn out significant, so your effective group-wise Type I error rate is actually 40% -- a far cry from the 5% you may have thought it was.
Bonferroni's Solution
    One very simple method due to Bonferroni (1936) is to divide the test-wise significance level by the number of tests:
    α β =α / κ
    In our example, α β = 0.05 / 10 = 0.005. So if we apply a significance level of 0.005 to each of the ten tests, there is now only a 5% chance that any of them will be declared significant under the null hypothesis.
    In spite of its simplicity (or perhaps because of it), the Bonferroni correction has attracted some criticism. Its biggest problem is that it is too conservative: by controlling the group-wise error rate, each individual test is held to an unreasonably high standard. This increases the probability of a Type II error, and makes it likely that legitimately significant results will fail to be detected.

    A brief discussion of the shortcomings of the method may be found in Perneger (1998).
Alternative Methods
SAS Programming
    In SAS, these methods can be performed using the MULTTEST procedure. Legendre & Legendre (1998) contains a discussion of these methods.
More Information
    Bonferroni CE (1936). Teoria statistica delle classi e calcolo delle probabilit. Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commerciali di Firenze, 8:3-62.

Related concepts 

Related terms 


  • Benjamini Y, Hochberg Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J Roy Stat Soc B 1995;57:289-300.(View)
  • Hochberg Y. A sharper Bonferroni procedure for multiple tests of significance. Biometrika 1988;75:800-803.(View)
  • Holm S. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 1979;6:65-70.(View)
  • Hommel G. A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 1988;75:383-386.(View)
  • Legendre P, Legendre L. Numerical Ecology (2nd English edition). Amsterdam, The Netherlands: Elsevier Science B.V. 1998. 0-0.(View)
  • Perneger TV. What is wrong with Bonferroni adjustments. BMJ 1998;136:1236-1238. [Abstract] (View)
  • Roberts JD, Roos LL, Poffenroth LA, Hassard TH, Bebchuk JD, Carter AL, Law B. Surveillance of vaccine-related adverse events in the first year of life: A Manitoba cohort study. J Clin Epidemiol 1996;49(1):51-58. [Abstract] (View)


  • methods
  • statistics

The MCHP data repository sends out a quarterly update identifying new research resources, data, and documentation at MCHP. If you're interested, please enter your email address in the space below. You can update your information or unsubscribe at any time.

* Email
* First Name
* Last Name
* Business
* = Required Field
This email list is privately maintained for MCHP by iContact.

Find us. Twitter and Facebook.

MCHP on Twitter MCHP on Facebook