Conditional Entropy - Chain Rule

Chain Rule

Assume that the combined system determined by two random variables X and Y has entropy, that is, we need bits of information to describe its exact state. Now if we first learn the value of, we have gained bits of information. Once is known, we only need bits to describe the state of the whole system. This quantity is exactly, which gives the chain rule of conditional probability:

Formally, the chain rule indeed follows from the above definition of conditional probability:

\begin{align} H(Y|X)=&\sum_{x\in\mathcal X, y\in\mathcal Y}p(x,y)\log \frac {p(x)} {p(x,y)}\\ =&-\sum_{x\in\mathcal X, y\in\mathcal Y}p(x,y)\log\,p(x,y) + \sum_{x\in\mathcal X, y\in\mathcal Y}p(x,y)\log\,p(x) \\ =& H(X,Y) + \sum_{x \in \mathcal X} p(x)\log\,p(x) \\ =& H(X,Y) - H(X). \end{align}

Read more about this topic:  Conditional Entropy

Famous quotes containing the words chain and/or rule:

    “From Nature’s chain whatever link you strike,
    Tenth or ten thousandth, breaks the chain alike.”
    —Alexander Pope (1688–1744)

    “Therefore doth heaven divide
    The state of man in divers functions,
    Setting endeavor in continual motion,
    To which is fixed, as an aim or butt,
    Obedience; for so work the honeybees,
    Creatures that by a rule in nature teach
    The act of order to a peopled kingdom.”
    —William Shakespeare (1564–1616)