Sample Mean and Sample Covariance - Discussion

Discussion

The sample mean and the sample covariance matrix are unbiased estimates of the mean and the covariance matrix of the random vector, a row vector whose jth element (j = 1, ..., K) is one of the random variables. The sample covariance matrix has in the denominator rather than due to a variant of Bessel's correction: In short, the sample covariance relies on the difference between each observation and the sample mean, but the sample mean is slightly correlated with each observation since it's defined in terms of all observations. If the population mean is known, the analogous unbiased estimate

using the population mean, has in the denominator. This is an example of why in probability and statistics it is essential to distinguish between random variables (upper case letters) and realizations of the random variables (lower case letters).

The maximum likelihood estimate of the covariance

for the Gaussian distribution case has N in the denominator as well. The ratio of 1/N to 1/(N − 1) approaches 1 for large N, so the maximum likelihood estimate approximately equals the unbiased estimate when the sample is large.