Conditional Entropy - Definition

Definition

If is the entropy of the variable conditioned on the variable taking a certain value, then is the result of averaging over all possible values that may take.

Given discrete random variable with support and with support, the conditional entropy of given is defined as:

\begin{align} H(Y|X)\ &\equiv \sum_{x\in\mathcal X}\,p(x)\,H(Y|X=x)\\ &{=}\sum_{x\in\mathcal X}p(x)\sum_{y\in\mathcal Y}\,p(y|x)\,\log\, \frac{1}{p(y|x)}\\ &=-\sum_{x\in\mathcal X}\sum_{y\in\mathcal Y}\,p(x,y)\,\log\,p(y|x)\\ &=-\sum_{x\in\mathcal X, y\in\mathcal Y}p(x,y)\log\,p(y|x)\\ &=\sum_{x\in\mathcal X, y\in\mathcal Y}p(x,y)\log \frac {p(x)} {p(x,y)}. \\ \end{align}


Note: The supports of X and Y can be replaced by their domains if it is understood that should be treated as being equal to zero.

if and only if the value of is completely determined by the value of . Conversely, if and only if and are independent random variables.

Read more about this topic:  Conditional Entropy

Famous quotes containing the word definition:

    “... if, as women, we accept a philosophy of history that asserts that women are by definition assimilated into the male universal, that we can understand our past through a male lens—if we are unaware that women even have a history—we live our lives similarly unanchored, drifting in response to a veering wind of myth and bias.”
    —Adrienne Rich (b. 1929)

    “One definition of man is “an intelligence served by organs.””
    —Ralph Waldo Emerson (1803–1882)

    “Was man made stupid to see his own stupidity?
    Is God by definition indifferent, beyond us all?
    Is the eternal truth man’s fighting soul
    Wherein the Beast ravens in its own avidity?”
    —Richard Eberhart (b. 1904)