Information Gain in Decision Trees

Information Gain In Decision Trees

In information theory and machine learning, information gain is an alternative synonym for Kullback–Leibler divergence.

In particular, the information gain about a random variable X obtained from an observation that a random variable A takes the value A=a is the Kullback-Leibler divergence DKL(p(x | a) || p(x | I)) of the prior distribution p(x | I) for x from the posterior distribution p(x | a) for x given a.

The expected value of the information gain is the mutual information I(X; A) of X and A — i.e. the reduction in the entropy of X achieved by learning the state of the random variable A.

In machine learning this concept can be used to define a preferred sequence of attributes to investigate to most rapidly narrow down the state of X. Such a sequence (which depends on the outcome of the investigation of previous attributes at each stage) is called a decision tree. Usually an attribute with high information gain should be preferred to other attributes.

Read more about Information Gain In Decision Trees:  General Definition, Formal Definition, Drawbacks, Constructing A Decision Tree Using Information Gain

Famous quotes containing the words information, gain, decision and/or trees:

    “When action grows unprofitable, gather information; when information grows unprofitable, sleep.”
    —Ursula K. Le Guin (b. 1929)

    “Greatness, in order to gain recognition, must all too often consent to ape greatness.”
    —Jean Rostand (1894–1977)

    “How could a man be satisfied with a decision between such alternatives and under such circumstances? No more than he can be satisfied with his hat, which he’s chosen from among such shapes as the resources of the age offer him, wearing it at best with a resignation which is chiefly supported by comparison.”
    —George Eliot [Mary Ann (or Marian)

    “In some of those dense fir and spruce woods there is hardly room for the smoke to go up. The trees are a standing night, and every fir and spruce which you fell is a plume plucked from night’s raven wing.”
    —Henry David Thoreau (1817–1862)