Proof: Source Coding Theorem For Symbol Codes
Let denote the wordlength of each possible . Define, where C is chosen so that .
Then
where the second line follows from Gibbs' inequality and the fifth line follows from Kraft's inequality: so .
For the second inequality we may set
so that
and so
and
and so by Kraft's inequality there exists a prefix-free code having those wordlengths. Thus the minimal S satisfies
Read more about this topic: Shannon's Source Coding Theorem
Famous quotes containing the words source, theorem, symbol and/or codes:
“If all else perished, and he remained, I should still continue to be; and if all else remained, and he were annihilated, the universe would turn to a mighty stranger. I should not seem a part of it.... My love for Heathcliff resembles the eternal rocks beneatha source of little visible delight, but necessary. Nelly, I am Heathcliffhes always, always in my mindnot as a pleasure, any more than I am always a pleasure to myselfbut as my own being.”
—Emily Brontë (18181848)
“To insure the adoration of a theorem for any length of time, faith is not enough, a police force is needed as well.”
—Albert Camus (19131960)
“The truth has never been of any real value to any human beingit is a symbol for mathematicians and philosophers to pursue. In human relations kindness and lies are worth a thousand truths.”
—Graham Greene (19041991)
“We must trust infinitely to the beneficent necessity which shines through all laws. Human nature expresses itself in them as characteristically as in statues, or songs, or railroads, and an abstract of the codes of nations would be an abstract of the common conscience.”
—Ralph Waldo Emerson (18031882)