Effect of Autocorrelation (serial Correlation)
The material above, to stress the point again, applies only to independent data. However, real-world data often does not meet this requirement; it is autocorrelated (also known as serial correlation). As one example, the successive readings of a measurement instrument that incorporates some form of “smoothing” (more correctly, “filtering”) process will be autocorrelated, since the current reading is calculated from some combination of the prior readings.
Estimates of the variance, and standard deviation, of autocorrelated data will be biased. The expected value of the sample variance is
where n is the sample size (number of measurements) and is the autocorrelation function (ACF) of the data. (Note that the expression in the brackets is simply one minus the average expected autocorrelation for the readings.) If the ACF consists of positive values then the estimate of the variance (and its square root, the standard deviation) will be biased low. That is, the actual variability of the data will be greater than that indicated by an uncorrected variance or standard deviation calculation. It is essential to recognize that, if this expression is to be used to correct for the bias, by dividing the estimate by the quantity in brackets above, then the ACF must be known analytically, not via estimation from the data. This is because the estimated ACF will itself be biased.
Read more about this topic: Unbiased Estimation Of Standard Deviation
Famous quotes containing the word effect:
“The use of symbols has a certain power of emancipation and exhilaration for all men. We seem to be touched by a wand, which makes us dance and run about happily, like children. We are like persons who come out of a cave or cellar into the open air. This is the effect on us of tropes, fables, oracles, and all poetic forms. Poets are thus liberating gods.”
—Ralph Waldo Emerson (1803–1882)