Entropy (information Theory) - Extending Discrete Entropy To The Continuous Case: Differential Entropy

Extending Discrete Entropy To The Continuous Case: Differential Entropy

The Shannon entropy is restricted to random variables taking discrete values. The corresponding formula for a continuous random variable with probability density function f(x) on the real line is defined by analogy, using the above form of the entropy as an expectation:

This formula is usually referred to as the continuous entropy, or differential entropy. A precursor of the continuous entropy is the expression for the functional in the H-theorem of Boltzmann.

Although the analogy between both functions is suggestive, the following question must be set: is the differential entropy a valid extension of the Shannon discrete entropy? Differential entropy lacks a number of properties that the Shannon discrete entropy has – it can even be negative – and thus corrections have been suggested, notably limiting density of discrete points.

To answer this question, we must establish a connection between the two functions:

We wish to obtain a generally finite measure as the bin size goes to zero. In the discrete case, the bin size is the (implicit) width of each of the n (finite or infinite) bins whose probabilities are denoted by pn. As we generalize to the continuous domain, we must make this width explicit.

To do this, start with a continuous function f discretized as shown in the figure. As the figure indicates, by the mean-value theorem there exists a value xi in each bin such that

and thus the integral of the function f can be approximated (in the Riemannian sense) by

where this limit and "bin size goes to zero" are equivalent.

We will denote

and expanding the logarithm, we have

\begin{align} H^{\Delta} &= - \sum_{i=-\infty}^{\infty} \Delta f(x_i) \log \Delta f(x_i) \\ &= - \sum_{i=-\infty}^{\infty} \Delta f(x_i) \log f(x_i) -\sum_{i=-\infty}^{\infty} f(x_i) \Delta \log \Delta. \end{align}

As, we have

and also

But note that as, therefore we need a special definition of the differential or continuous entropy:

which is, as said before, referred to as the differential entropy. This means that the differential entropy is not a limit of the Shannon entropy for . Rather, it differs from the limit of the Shannon entropy by an infinite offset.

It turns out as a result that, unlike the Shannon entropy, the differential entropy is not in general a good measure of uncertainty or information. For example, the differential entropy can be negative; also it is not invariant under continuous co-ordinate transformations.

Read more about this topic:  Entropy (information Theory)

Famous quotes containing the words extending, discrete, entropy, continuous and/or differential:

    “The radiance was that of the full, setting, and blood-red moon, which now shone vividly through that once barely- discernible fissure,... extending from the roof of the building, in a zigzag direction, to the base. While I gazed, this fissure rapidly widened.”
    —Edgar Allan Poe (1809–1849)

    “We have good reason to believe that memories of early childhood do not persist in consciousness because of the absence or fragmentary character of language covering this period. Words serve as fixatives for mental images. . . . Even at the end of the second year of life when word tags exist for a number of objects in the child’s life, these words are discrete and do not yet bind together the parts of an experience or organize them in a way that can produce a coherent memory.”
    —Selma H. Fraiberg (20th century)

    “Just as the constant increase of entropy is the basic law of the universe, so it is the basic law of life to be ever more highly structured and to struggle against entropy.”
    —Václav Havel (b. 1936)

    “I describe family values as responsibility towards others, increase of tolerance, compromise, support, flexibility. And essentially the things I call the silent song of life—the continuous process of mutual accommodation without which life is impossible.”
    —Salvador Minuchin (20th century)

    “But how is one to make a scientist understand that there is something unalterably deranged about differential calculus, quantum theory, or the obscene and so inanely liturgical ordeals of the precession of the equinoxes.”
    —Antonin Artaud (1896–1948)