Information Theory and Measure Theory - Entropy As A "measure"

Entropy As A "measure"

There is an analogy between Shannon's basic "measures" of the information content of random variables and a measure over sets. Namely the joint entropy, conditional entropy, and mutual information can be considered as the measure of a set union, set difference, and set intersection, respectively (Reza pp. 106–108).

If we associate the existence of abstract sets and to arbitrary discrete random variables X and Y, somehow representing the information borne by X and Y, respectively, such that:

• whenever X and Y are unconditionally independent, and
• whenever X and Y are such that either one is completely determined by the other (i.e. by a bijection);

where is a signed measure over these sets, and we set:

we find that Shannon's "measure" of information content satisfies all the postulates and basic properties of a formal measure over sets, as commonly illustrated in an information diagram. This can be a handy mnemonic device in some situations, e.g.

Because the entropy, joint entropy, conditional entropy, and bivariate mutual information of discrete random variables are all nonnegative, many basic inequalities in information theory (among no more than two random variables) can be derived from this formulation by considering the measure μ to be nonnegative.