Information Theory and Measure Theory

Information Theory And Measure Theory

Many of the formulas in information theory have separate versions for continuous and discrete cases, i.e. integrals for the continuous case and sums for the discrete case. These versions can often be generalized using measure theory. For discrete random variables, probability mass functions can be considered density functions with respect to the counting measure, thus requiring only basic discrete mathematics for what can be considered, in a measure theory context, integration. Because the same integration expression is used for the continuous case, which uses basic calculus, the same concepts and expressions can be used for both discrete and continuous cases. Consider the formula for the differential entropy of a continuous random variable with probability density function :

This can usually be taken to be

where μ is the Lebesgue measure. But if instead, X is discrete, f is a probability mass function, and ν is the counting measure, we can write:

The integral expression and the general concept is identical to the continuous case; the only difference is the measure used. In both cases the probability density function f is the Radon–Nikodym derivative of the probability measure with respect to the measure against which the integral is taken.

If is the probability measure on X, then the integral can also be taken directly with respect to :

If instead of the underlying measure μ we take another probability measure, we are led to the Kullback–Leibler divergence: let and be probability measures over the same space. Then if is absolutely continuous with respect to, written the Radon–Nikodym derivative exists and the Kullback–Leibler divergence can be expressed in its full generality:

where the integral runs over the support of Note that we have dropped the negative sign: the Kullback–Leibler divergence is always non-negative due to Gibbs' inequality.