## Concept Description

Last Updated: 2008-04-21

### Problem

When performing κ multiple independent significance tests each at the α level, the probability of making at least one Type I error (rejecting the null hypothesis inappropriately) is 1-(1-α) κ . For example, with κ=10 and α=0.05, there is a 40% chance of at least one of the ten tests being declared significant under the null hypothesis.

So, when you see a significant result among the ten tests, how confident can you be that it is "really" significant? There is a 40% chance that something will turn out significant, so your effective group-wise Type I error rate is actually 40% -- a far cry from the 5% you may have thought it was.

### Bonferroni's Solution

One very simple method due to Bonferroni (1936) is to divide the test-wise significance level by the number of tests:
α β =α / κ
In our example, α β = 0.05 / 10 = 0.005. So if we apply a significance level of 0.005 to each of the ten tests, there is now only a 5% chance that any of them will be declared significant under the null hypothesis.

### Criticisms

In spite of its simplicity (or perhaps because of it), the Bonferroni correction has attracted some criticism. Its biggest problem is that it is too conservative: by controlling the group-wise error rate, each individual test is held to an unreasonably high standard. This increases the probability of a Type II error, and makes it likely that legitimately significant results will fail to be detected.

A brief discussion of the shortcomings of the method may be found in Perneger (1998).

### SAS Programming

In SAS, these methods can be performed using the MULTTEST procedure. Legendre & Legendre (1998) contains a discussion of these methods.

Bonferroni CE (1936). Teoria statistica delle classi e calcolo delle probabilit. Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commerciali di Firenze, 8:3-62.

## References

• Benjamini Y, Hochberg Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J Roy Stat Soc B 1995;57:289-300.(View)
• Hochberg Y. A sharper Bonferroni procedure for multiple tests of significance. Biometrika 1988;75:800-803.(View)
• Holm S. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 1979;6:65-70.(View)
• Hommel G. A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 1988;75:383-386.(View)
• Legendre P, Legendre L. Numerical Ecology (2nd English edition). Amsterdam, The Netherlands: Elsevier Science B.V. 1998. 0-0.(View)
• Perneger TV. What is wrong with Bonferroni adjustments. BMJ 1998;136:1236-1238. [Abstract] (View)
• Roberts JD, Roos LL, Poffenroth LA, Hassard TH, Bebchuk JD, Carter AL, Law B. Surveillance of vaccine-related adverse events in the first year of life: A Manitoba cohort study. J Clin Epidemiol 1996;49(1):51-58. [Abstract] (View)

• methods
• statistics