Bias of An Estimator - Examples - Sample Variance

Sample Variance

Suppose X1, ..., Xn are independent and identically distributed (i.i.d.) random variables with expectation μ and variance σ2. If the sample mean and uncorrected sample variance are defined as

then S2 is a biased estimator of σ2, because

 \begin{align} \operatorname{E} &= \operatorname{E}\left = \operatorname{E}\bigg \\ &= \operatorname{E}\bigg[ \frac{1}{n}\sum_{i=1}^n (X_i-\mu)^2 - 2(\overline{X}-\mu)\frac{1}{n}\sum_{i=1}^n (X_i-\mu) + (\overline{X}-\mu)^2 \bigg] \\ &= \operatorname{E}\bigg = \sigma^2 - \operatorname{E}\left < \sigma^2. \end{align}

In other words, the expected value of the uncorrected sample variance does not equal the population variance σ2, unless multiplied by a normalization factor. The sample mean, on the other hand, is an unbiased estimator of the population mean μ.

The reason that S2 is biased stems from the fact that the sample mean is an ordinary least squares (OLS) estimator for μ: is the number that makes the sum as small as possible. That is, when any other number is plugged into this sum, the sum can only increase. In particular, the choice gives,

 \frac{1}{n}\sum_{i=1}^n (X_i-\overline{X})^2 < \frac{1}{n}\sum_{i=1}^n (X_i-\mu)^2,

and then

 \begin{align} \operatorname{E} &= \operatorname{E}\bigg < \operatorname{E}\bigg = \sigma^2. \end{align}

Note that the usual definition of sample variance is

and this is an unbiased estimator of the population variance. This can be seen by noting the following formula, which follows from the Bienaymé formula, for the term in the inequality for the expectation of the uncorrected sample variance above:

The ratio between the biased (uncorrected) and unbiased estimates of the variance is known as Bessel's correction.

Read more about this topic:  Bias Of An Estimator, Examples

Famous quotes containing the words sample and/or variance:

    The present war having so long cut off all communication with Great-Britain, we are not able to make a fair estimate of the state of science in that country. The spirit in which she wages war is the only sample before our eyes, and that does not seem the legitimate offspring either of science or of civilization.
    Thomas Jefferson (1743–1826)

    There is an untroubled harmony in everything, a full consonance in nature; only in our illusory freedom do we feel at variance with it.
    Fyodor Tyutchev (1803–1873)