Information Gain In Decision Trees
In information theory and machine learning, information gain is an alternative synonym for Kullback–Leibler divergence.
In particular, the information gain about a random variable X obtained from an observation that a random variable A takes the value A=a is the Kullback-Leibler divergence DKL(p(x | a) || p(x | I)) of the prior distribution p(x | I) for x from the posterior distribution p(x | a) for x given a.
The expected value of the information gain is the mutual information I(X; A) of X and A — i.e. the reduction in the entropy of X achieved by learning the state of the random variable A.
In machine learning this concept can be used to define a preferred sequence of attributes to investigate to most rapidly narrow down the state of X. Such a sequence (which depends on the outcome of the investigation of previous attributes at each stage) is called a decision tree. Usually an attribute with high information gain should be preferred to other attributes.
Read more about Information Gain In Decision Trees: General Definition, Formal Definition, Drawbacks, Constructing A Decision Tree Using Information Gain
Famous quotes containing the words information, gain, decision and/or trees:
“Information networks straddle the world. Nothing remains concealed. But the sheer volume of information dissolves the information. We are unable to take it all in.”
—Günther Grass (b. 1927)
“Give way to your opponent; thus will you gain the crown of victory.”
—Ovid (Publius Ovidius Naso)
“Will mankind never learn that policy is not morality,—that it never secures any moral right, but considers merely what is expedient? chooses the available candidate,—who is invariably the devil,—and what right have his constituents to be surprised, because the devil does not behave like an angel of light? What is wanted is men, not of policy, but of probity,—who recognize a higher law than the Constitution, or the decision of the majority.”
—Henry David Thoreau (1817–1862)
“Here come the line-gang pioneering by.
They throw a forest down less cut than broken.
They plant dead trees for living, and the dead
They string together with a living thread.”
—Robert Frost (1874–1963)