Information Gain In Decision Trees
In information theory and machine learning, information gain is an alternative synonym for Kullback–Leibler divergence.
In particular, the information gain about a random variable X obtained from an observation that a random variable A takes the value A=a is the Kullback-Leibler divergence DKL(p(x | a) || p(x | I)) of the prior distribution p(x | I) for x from the posterior distribution p(x | a) for x given a.
The expected value of the information gain is the mutual information I(X; A) of X and A — i.e. the reduction in the entropy of X achieved by learning the state of the random variable A.
In machine learning this concept can be used to define a preferred sequence of attributes to investigate to most rapidly narrow down the state of X. Such a sequence (which depends on the outcome of the investigation of previous attributes at each stage) is called a decision tree. Usually an attribute with high information gain should be preferred to other attributes.
Read more about Information Gain In Decision Trees: General Definition, Formal Definition, Drawbacks, Constructing A Decision Tree Using Information Gain
Famous quotes containing the words information, gain, decision and/or trees:
“Computers are good at swift, accurate computation and at storing great masses of information. The brain, on the other hand, is not as efficient a number cruncher and its memory is often highly fallible; a basic inexactness is built into its design. The brain’s strong point is its flexibility. It is unsurpassed at making shrewd guesses and at grasping the total meaning of information presented to it.”
—Jeremy Campbell (b. 1931)
“I find myself ... hoping a total end of all the unhappy divisions of mankind by party-spirit, which at best is but the madness of many for the gain of a few.”
—Alexander Pope (1688–1744)
“There are many things children accept as “grown-up things” over when they have no control and for which they have no responsibility—for instance, weddings, having babies, buying houses, and driving cars. Parents who are separating really need to help their children put divorce on that grown-up list, so that children do not see themselves as the cause of their parents’ decision to live apart.”
—Fred Rogers (20th century)
“Let us have a good many maples and hickories and scarlet oaks, then, I say. Blaze away! Shall that dirty roll of bunting in the gun-house be all the colors a village can display? A village is not complete, unless it have these trees to mark the season in it. They are important, like the town clock. A village that has them not will not be found to work well. It has a screw loose, an essential part is wanting.”
—Henry David Thoreau (1817–1862)