Concept: Multiple Comparisons
Last Updated: 2008-04-21
When performing κ multiple independent significance tests each at the α level, the probability of making at least one Type I error (rejecting the null hypothesis inappropriately) is 1-(1-α)
. For example, with κ=10 and α=0.05, there is a 40% chance of at least one of the ten tests being declared significant under the null hypothesis.
So, when you see a significant result among the ten tests, how confident can you be that it is "really" significant? There is a 40% chance that something will turn out significant, so your effective group-wise Type I error rate is actually 40% -- a far cry from the 5% you may have thought it was.
One very simple method due to Bonferroni (1936) is to divide the test-wise significance level by the number of tests:
α β =α / κIn our example, α β = 0.05 / 10 = 0.005. So if we apply a significance level of 0.005 to each of the ten tests, there is now only a 5% chance that any of them will be declared significant under the null hypothesis.
In spite of its simplicity (or perhaps because of it), the Bonferroni correction has attracted some criticism. Its biggest problem is that it is too conservative: by controlling the group-wise error rate, each individual test is held to an unreasonably high standard. This increases the probability of a Type II error, and makes it likely that legitimately significant results will fail to be detected.
A brief discussion of the shortcomings of the method may be found in Perneger (1998).
Hochberg (1988), Holm (1979), Hommel (1988), and Benjamini & Hochberg (1995)
have developed methods which are generally less conservative than Bonferroni's.
In SAS, these methods can be performed using the MULTTEST procedure.
Legendre & Legendre (1998)
contains a discussion of these methods.
Bonferroni CE (1936). Teoria statistica delle classi e calcolo delle probabilit. Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commerciali di Firenze, 8:3-62.
- Benjamini Y, Hochberg Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J Roy Stat Soc B 1995;57:289-300.(View)
- Hochberg Y. A sharper Bonferroni procedure for multiple tests of significance. Biometrika 1988;75:800-803.(View)
- Holm S. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 1979;6:65-70.(View)
- Hommel G. A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 1988;75:383-386.(View)
- Legendre P, Legendre L. Numerical Ecology (2nd English edition). Amsterdam, The Netherlands: Elsevier Science B.V. 1998. 0-0.(View)
- Perneger TV. What is wrong with Bonferroni adjustments. BMJ 1998;136:1236-1238. [Abstract] (View)
- Roberts JD, Roos LL, Poffenroth LA, Hassard TH, Bebchuk JD, Carter AL, Law B. Surveillance of vaccine-related adverse events in the first year of life: A Manitoba cohort study. J Clin Epidemiol 1996;49(1):51-58. [Abstract] (View)
Manitoba Centre for Health Policy
Community Health Sciences, Max Rady College of Medicine,
Rady Faculty of Health Sciences,
Room 408-727 McDermot Ave.
University of Manitoba
Winnipeg, MB R3E 3P5 Canada