In information theory, the cross entropy between two probability distributions measures the average number of bits needed to identify an event from a set of possibilities, if a coding scheme is used based on a given probability distribution, rather than the "true" distribution .
The cross entropy for two distributions and over the same probability space is thus defined as follows:
- ,
where is the entropy of, and is the Kullback-Leibler divergence of from (also known as the relative entropy).
For discrete and this means
The situation for continuous distributions is analogous:
NB: The notation is sometimes used for both the cross entropy as well as the joint entropy of and .
Read more about Cross Entropy: Motivation, Estimation, Cross-entropy Minimization
Famous quotes containing the words cross and/or entropy:
“Men are not to be told anything they might find too painful; the secret depths of human nature, the sordid physicalities, might overwhelm or damage them. For instance, men often faint at the sight of their own blood, to which they are not accustomed. For this reason you should never stand behind one in the line at the Red Cross donor clinic.”
—Margaret Atwood (b. 1939)
“Just as the constant increase of entropy is the basic law of the universe, so it is the basic law of life to be ever more highly structured and to struggle against entropy.”
—Václav Havel (b. 1936)