Information Gain In Decision Trees
In information theory and machine learning, information gain is an alternative synonym for Kullback–Leibler divergence.
In particular, the information gain about a random variable X obtained from an observation that a random variable A takes the value A=a is the Kullback-Leibler divergence DKL(p(x | a) || p(x | I)) of the prior distribution p(x | I) for x from the posterior distribution p(x | a) for x given a.
The expected value of the information gain is the mutual information I(X; A) of X and A — i.e. the reduction in the entropy of X achieved by learning the state of the random variable A.
In machine learning this concept can be used to define a preferred sequence of attributes to investigate to most rapidly narrow down the state of X. Such a sequence (which depends on the outcome of the investigation of previous attributes at each stage) is called a decision tree. Usually an attribute with high information gain should be preferred to other attributes.
Read more about Information Gain In Decision Trees: General Definition, Formal Definition, Drawbacks, Constructing A Decision Tree Using Information Gain
Famous quotes containing the words information, gain, decision and/or trees:
“If you have any information or evidence regarding the O.J. Simpson case, press 2 now. If you are an expert in fields relating to the O.J. Simpson case and would like to offer your services, press 3 now. If you would like the address where you can send a letter of support to O.J. Simpson, press 1 now. If you are seeking legal representation from the law offices of Robert L. Shapiro, press 4 now.”
—Advertisement. Aired August 8, 1994 by Tom Snyder on TV station CNBC. Chicago Sun Times, p. 11 (July 24, 1994)
“To gain that which is worth having, it may be necessary to lose everything else.”
—Bernadette Devlin Mcaliskey (b. 1947)
“I know my fate. One day my name will be tied to the memory of something monstrous—a crisis without equal on earth, the most profound collision of conscience, a decision invoked against everything that had previously been believed, demanded, sanctified. I am no man, I am dynamite!”
—Friedrich Nietzsche (1844–1900)
“Who shall describe the inexpressable tenderness and immortal life of the grim forest, where Nature, though it be midwinter, is ever in her spring, where the moss-grown and decaying trees are not old, but seem to enjoy a perpetual youth.”
—Henry David Thoreau (1817–1862)