P-value - Interpretation

Interpretation

Traditionally, one rejects the null hypothesis if the p-value is less than or equal to the significance level, often represented by the Greek letter α (alpha). (Greek α is also used for Type I error; the connection is that a hypothesis test that rejects the null hypothesis for all samples that have a p-value less than α will have a Type I error of α.) A significance level of 0.05 would deem extraordinary any result that is within the most extreme 5% of all possible results under the null hypothesis. In this case a p-value less than 0.05 would result in the rejection of the null hypothesis at the 5% (significance) level.

When we ask whether a given coin is fair, often we are interested in the deviation of our result from the equality of numbers of heads and tails. In this case, the deviation can be in either direction, favoring either heads or tails. Thus, in this example of 14 heads and 6 tails, we may want to calculate the probability of getting a result deviating by at least 4 from parity in either direction (two-sided test). This is the probability of getting at least 14 heads or at least 14 tails. As the binomial distribution is symmetrical for a fair coin, the two-sided p-value is simply twice the above calculated single-sided p-value; i.e., the two-sided p-value is 0.115.

In the above example we thus have:

• null hypothesis (H₀): fair coin; P(heads) = 0.5
• observation O: 14 heads out of 20 flips; and
• p-value of observation O given H₀ = Prob(≥ 14 heads or ≥ 14 tails) = 2*(1-Prob(< 14)) = 0.115.

The calculated p-value exceeds 0.05, so the observation is consistent with the null hypothesis — that the observed result of 14 heads out of 20 flips can be ascribed to chance alone — as it falls within the range of what would happen 95% of the time were the coin in fact fair. In our example, we fail to reject the null hypothesis at the 5% level. Although the coin did not fall evenly, the deviation from expected outcome is small enough to be consistent with chance.

However, had one more head been obtained, the resulting p-value (two-tailed) would have been 0.0414 (4.14%). This time the null hypothesis – that the observed result of 15 heads out of 20 flips can be ascribed to chance alone – is rejected when using a 5% cut-off.