Theoretical Relationship
Despite all that, there is an important difference between the two quantities. The information entropy H can be calculated for any probability distribution (if the "message" is taken to be that the event i which had probability pi occurred, out of the space of the events possible), while the thermodynamic entropy S refers to thermodynamic probabilities pi specifically.
Furthermore, the thermodynamic entropy S is dominated by different arrangements of the system, and in particular its energy, that are possible on a molecular scale. In comparison, information entropy of any macroscopic event is so small as to be completely irrelevant.
However, a connection can be made between the two. If the probabilities in question are the thermodynamic probabilities pi:, the (reduced) Gibbs entropy σ can then be seen as simply the amount of Shannon information needed to define the detailed microscopic state of the system, given its macroscopic description. Or, in the words of G. N. Lewis writing about chemical entropy in 1930, "Gain in entropy always means loss of information, and nothing more". To be more concrete, in the discrete case using base two logarithms, the reduced Gibbs entropy is equal to the minimum number of yes/no questions needed to be answered in order to fully specify the microstate, given that we know the macrostate.
Furthermore, the prescription to find the equilibrium distributions of statistical mechanics —such as the Boltzmann distribution— by maximising the Gibbs entropy subject to appropriate constraints (the Gibbs algorithm) can be seen as something not unique to thermodynamics, but as a principle of general relevance in all sorts of statistical inference, if it is desired to find a maximally uninformative probability distribution, subject to certain constraints on the behaviour of its averages. (These perspectives are explored further in the article Maximum entropy thermodynamics.)
Read more about this topic: Entropy In Thermodynamics And Information Theory
Famous quotes containing the words theoretical and/or relationship:
“Post-structuralism is among other things a kind of theoretical hangover from the failed uprising of ‘68Ma way of keeping the revolution warm at the level of language, blending the euphoric libertarianism of that moment with the stoical melancholia of its aftermath.”
—Terry Eagleton (b. 1943)
“Every relationship that does not raise us up pulls us down, and vice versa; this is why men usually sink down somewhat when they take wives while women are usually somewhat raised up. Overly spiritual men require marriage every bit as much as they resist it as bitter medicine.”
—Friedrich Nietzsche (1844–1900)