Effect of Autocorrelation (serial Correlation)
The material above, to stress the point again, applies only to independent data. However, real-world data often does not meet this requirement; it is autocorrelated (also known as serial correlation). As one example, the successive readings of a measurement instrument that incorporates some form of “smoothing” (more correctly, “filtering”) process will be autocorrelated, since the current reading is calculated from some combination of the prior readings.
Estimates of the variance, and standard deviation, of autocorrelated data will be biased. The expected value of the sample variance is
where n is the sample size (number of measurements) and is the autocorrelation function (ACF) of the data. (Note that the expression in the brackets is simply one minus the average expected autocorrelation for the readings.) If the ACF consists of positive values then the estimate of the variance (and its square root, the standard deviation) will be biased low. That is, the actual variability of the data will be greater than that indicated by an uncorrected variance or standard deviation calculation. It is essential to recognize that, if this expression is to be used to correct for the bias, by dividing the estimate by the quantity in brackets above, then the ACF must be known analytically, not via estimation from the data. This is because the estimated ACF will itself be biased.
Read more about this topic: Unbiased Estimation Of Standard Deviation
Famous quotes containing the word effect:
“The law before us, my lords, seems to be the effect of that practice of which it is intended likewise to be the cause, and to be dictated by the liquor of which it so effectually promotes the use; for surely it never before was conceived by any man entrusted with the administration of public affairs, to raise taxes by the destruction of the people.”
—Philip Dormer Stanhope, 4th Earl Chesterfield (1694–1773)