Empirical Measure - Definition

Let be a sequence of independent identically distributed random variables with values in the state space S with probability measure P.

Definition

The empirical measure is defined for measurable subsets of S and given by

where is the indicator function and is the Dirac measure.

For a fixed measurable set A, nP_n(A) is a binomial random variable with mean nP(A) and variance nP(A)(1 − P(A)). In particular, is an unbiased estimator of P(A).

Definition

is the empirical measure indexed by, a collection of measurable subsets of S.

To generalize this notion further, observe that the empirical measure P_n maps measurable functions to their empirical mean,

In particular, the empirical measure of A is simply the empirical mean of the indicator function, .

For a fixed measurable function f, is a random variable with mean and variance .

By the strong law of large numbers, converges to P(A) almost surely for fixed A. Similarly converges to almost surely for a fixed measurable function f. The problem of uniform convergence of to P was open until Vapnik and Chervonenkis solved it in 1968.

If the class (or ) is Glivenko–Cantelli with respect to P then converges to P uniformly over (or ). In other words, with probability 1 we have