Information Gain In Decision Trees
In information theory and machine learning, information gain is an alternative synonym for Kullback–Leibler divergence.
In particular, the information gain about a random variable X obtained from an observation that a random variable A takes the value A=a is the Kullback-Leibler divergence DKL(p(x | a) || p(x | I)) of the prior distribution p(x | I) for x from the posterior distribution p(x | a) for x given a.
The expected value of the information gain is the mutual information I(X; A) of X and A — i.e. the reduction in the entropy of X achieved by learning the state of the random variable A.
In machine learning this concept can be used to define a preferred sequence of attributes to investigate to most rapidly narrow down the state of X. Such a sequence (which depends on the outcome of the investigation of previous attributes at each stage) is called a decision tree. Usually an attribute with high information gain should be preferred to other attributes.
Read more about Information Gain In Decision Trees: General Definition, Formal Definition, Drawbacks, Constructing A Decision Tree Using Information Gain
Famous quotes containing the words information, gain, decision and/or trees:
“As information technology restructures the work situation, it abstracts thought from action.”
—Shoshana Zuboff (b. 1951)
“Those who can lose shall gain; those who wish for gain shall lose.”
—Chinese proverb.
“Every decision is liberating, even if it leads to disaster. Otherwise, why do so many people walk upright and with open eyes into their misfortune?”
—Elias Canetti (b. 1905)
“But, where the road runs near the stream,
Oft through the trees they catch a glance
Of passing troops in the sun’s beam—
Pennon, and plume, and flashing lance!
Forth to the world those soldiers fare,
To life, to cities, and to war!”
—Matthew Arnold (1822–1888)