Max Rady College of Medicine

Concept: Multiple Comparisons

 Printer friendly

Concept Description

Last Updated: 2008-04-21

Problem

    When performing κ multiple independent significance tests each at the α level, the probability of making at least one Type I error (rejecting the null hypothesis inappropriately) is 1-(1-α) κ . For example, with κ=10 and α=0.05, there is a 40% chance of at least one of the ten tests being declared significant under the null hypothesis.

    So, when you see a significant result among the ten tests, how confident can you be that it is "really" significant? There is a 40% chance that something will turn out significant, so your effective group-wise Type I error rate is actually 40% -- a far cry from the 5% you may have thought it was.

Bonferroni's Solution

    One very simple method due to Bonferroni (1936) is to divide the test-wise significance level by the number of tests:
    α β =α / κ
    In our example, α β = 0.05 / 10 = 0.005. So if we apply a significance level of 0.005 to each of the ten tests, there is now only a 5% chance that any of them will be declared significant under the null hypothesis.

Criticisms

    In spite of its simplicity (or perhaps because of it), the Bonferroni correction has attracted some criticism. Its biggest problem is that it is too conservative: by controlling the group-wise error rate, each individual test is held to an unreasonably high standard. This increases the probability of a Type II error, and makes it likely that legitimately significant results will fail to be detected.

    A brief discussion of the shortcomings of the method may be found in Perneger (1998).

Alternative Methods

SAS Programming

    In SAS, these methods can be performed using the MULTTEST procedure. Legendre & Legendre (1998) contains a discussion of these methods.

More Information

    Bonferroni CE (1936). Teoria statistica delle classi e calcolo delle probabilit. Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commerciali di Firenze, 8:3-62.

Related concepts 

Related terms 

References 

  • Benjamini Y, Hochberg Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J Roy Stat Soc B 1995;57:289-300.(View)
  • Hochberg Y. A sharper Bonferroni procedure for multiple tests of significance. Biometrika 1988;75:800-803.(View)
  • Holm S. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 1979;6:65-70.(View)
  • Hommel G. A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 1988;75:383-386.(View)
  • Legendre P, Legendre L. Numerical Ecology (2nd English edition). Amsterdam, The Netherlands: Elsevier Science B.V. 1998. 0-0.(View)
  • Perneger TV. What is wrong with Bonferroni adjustments. BMJ 1998;136:1236-1238. [Abstract] (View)
  • Roberts JD, Roos LL, Poffenroth LA, Hassard TH, Bebchuk JD, Carter AL, Law B. Surveillance of vaccine-related adverse events in the first year of life: A Manitoba cohort study. J Clin Epidemiol 1996;49(1):51-58. [Abstract] (View)

Keywords 

  • methods
  • statistics


Request information in an accessible format

If you require access to our resources in a different format, please contact us:

We strive to provide accommodations upon request in a reasonable timeframe.

Contact us

Manitoba Centre for Health Policy
Community Health Sciences, Max Rady College of Medicine,
Rady Faculty of Health Sciences,
Room 408-727 McDermot Ave.
University of Manitoba
Winnipeg, MB R3E 3P5 Canada

204-789-3819