Effect of Autocorrelation (serial Correlation)
The material above, to stress the point again, applies only to independent data. However, real-world data often does not meet this requirement; it is autocorrelated (also known as serial correlation). As one example, the successive readings of a measurement instrument that incorporates some form of “smoothing” (more correctly, “filtering”) process will be autocorrelated, since the current reading is calculated from some combination of the prior readings.
Estimates of the variance, and standard deviation, of autocorrelated data will be biased. The expected value of the sample variance is
where n is the sample size (number of measurements) and is the autocorrelation function (ACF) of the data. (Note that the expression in the brackets is simply one minus the average expected autocorrelation for the readings.) If the ACF consists of positive values then the estimate of the variance (and its square root, the standard deviation) will be biased low. That is, the actual variability of the data will be greater than that indicated by an uncorrected variance or standard deviation calculation. It is essential to recognize that, if this expression is to be used to correct for the bias, by dividing the estimate by the quantity in brackets above, then the ACF must be known analytically, not via estimation from the data. This is because the estimated ACF will itself be biased.
Read more about this topic: Unbiased Estimation Of Standard Deviation
Famous quotes containing the word effect:
“We all have—to put it as nicely as I can—our lower centres and our higher centres. Our lower centres act: they act with terrible power that sometimes destroys us; but they don’t talk.... Since the war the lower centres have become vocal. And the effect is that of an earthquake. For they speak truths that have never been spoken before—truths that the makers of our domestic institutions have tried to ignore.”
—George Bernard Shaw (1856–1950)