Multiple Comparisons - What Can Be Done

What Can Be Done

For hypothesis testing, the problem of multiple comparisons (also known as the multiple testing problem) results from the increase in type I error that occurs when statistical tests are used repeatedly. If n independent comparisons are performed, the experiment-wide significance level α, also termed FWER for familywise error rate, is given by

.

Hence, unless the tests are perfectly dependent, α increases as the number of comparisons increases. If we do not assume that the comparisons are independent, then we can still say:

which follows from Boole's inequality. Example:

There are different ways to assure that the familywise error rate is at most . The most conservative, but free of independency and distribution assumptions method, is known as the Bonferroni correction . A more sensitive correction can be obtained by solving the equation for the familywise error rate of independent comparisons for . This yields, which is known as the Šidák correction. Another procedure is the Holm–Bonferroni method which uniformly delivers more power than the simple Bonferroni correction, by testing only the most extreme p value against the strictest criterion, and the others against progressively less strict criteria. .