Concept: Multiple Comparisons
Concept Description
Last Updated: 2008-04-21
Problem
When performing κ multiple independent significance tests each at the α level, the probability of making at least one Type I error (rejecting the null hypothesis inappropriately) is 1-(1-α)
κ
. For example, with κ=10 and α=0.05, there is a 40% chance of at least one of the ten tests being declared significant under the null hypothesis.
So, when you see a significant result among the ten tests, how confident can you be that it is "really" significant? There is a 40% chance that something will turn out significant, so your effective group-wise Type I error rate is actually 40% -- a far cry from the 5% you may have thought it was.
Bonferroni's Solution
One very simple method due to Bonferroni (1936) is to divide the test-wise significance level by the number of tests:
α
β
=α / κ
In our example, α
β
= 0.05 / 10 = 0.005. So if we apply a significance level of 0.005 to each of the ten tests, there is now only a 5% chance that any of them will be declared significant under the null hypothesis.
Criticisms
In spite of its simplicity (or perhaps because of it), the Bonferroni correction has attracted some criticism. Its biggest problem is that it is too conservative: by controlling the group-wise error rate, each individual test is held to an unreasonably high standard. This increases the probability of a Type II error, and makes it likely that legitimately significant results will fail to be detected.
A brief discussion of the shortcomings of the method may be found in
Perneger (1998).
Alternative Methods
SAS Programming
In SAS, these methods can be performed using the MULTTEST procedure.
Legendre & Legendre (1998)
contains a discussion of these methods.
More Information
Bonferroni CE (1936). Teoria statistica delle classi e calcolo delle probabilit. Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commerciali di Firenze, 8:3-62.
Related concepts
Related terms
References
- Benjamini Y, Hochberg Y.
Controlling the false discovery rate: A practical and powerful approach to multiple testing.
J Roy Stat Soc B
1995;57:289-300.(View)
- Hochberg Y.
A sharper Bonferroni procedure for multiple tests of significance.
Biometrika
1988;75:800-803.(View)
- Holm S.
A simple sequentially rejective multiple test procedure.
Scandinavian Journal of Statistics
1979;6:65-70.(View)
- Hommel G.
A stagewise rejective multiple test procedure based on a modified Bonferroni test.
Biometrika
1988;75:383-386.(View)
- Legendre P, Legendre L.
Numerical Ecology (2nd English edition).
Amsterdam, The Netherlands:
Elsevier Science B.V.
1998.
0-0.(View)
- Perneger TV.
What is wrong with Bonferroni adjustments.
BMJ
1998;136:1236-1238. [Abstract] (View)
- Roberts JD, Roos LL, Poffenroth LA, Hassard TH, Bebchuk JD, Carter AL, Law B.
Surveillance of vaccine-related adverse events in the first year of life: A Manitoba cohort study.
J Clin Epidemiol
1996;49(1):51-58. [Abstract] (View)
Keywords