In information theory, the cross entropy between two probability distributions measures the average number of bits needed to identify an event from a set of possibilities, if a coding scheme is used based on a given probability distribution, rather than the "true" distribution .
The cross entropy for two distributions and over the same probability space is thus defined as follows:
- ,
where is the entropy of, and is the Kullback-Leibler divergence of from (also known as the relative entropy).
For discrete and this means
The situation for continuous distributions is analogous:
NB: The notation is sometimes used for both the cross entropy as well as the joint entropy of and .
Read more about Cross Entropy: Motivation, Estimation, Cross-entropy Minimization
Famous quotes containing the words cross and/or entropy:
“It is an agreeable change to cross a lake, after you have been shut up in the woods, not only on account of the greater expanse of water, but also of sky. It is one of the surprises which Nature has in store for the traveler in the forest. To look down, in this case, over eighteen miles of water, was liberating and civilizing even.”
—Henry David Thoreau (1817–1862)
“Just as the constant increase of entropy is the basic law of the universe, so it is the basic law of life to be ever more highly structured and to struggle against entropy.”
—Václav Havel (b. 1936)