Information Gain In Decision Trees
In information theory and machine learning, information gain is an alternative synonym for Kullback–Leibler divergence.
In particular, the information gain about a random variable X obtained from an observation that a random variable A takes the value A=a is the Kullback-Leibler divergence DKL(p(x | a) || p(x | I)) of the prior distribution p(x | I) for x from the posterior distribution p(x | a) for x given a.
The expected value of the information gain is the mutual information I(X; A) of X and A — i.e. the reduction in the entropy of X achieved by learning the state of the random variable A.
In machine learning this concept can be used to define a preferred sequence of attributes to investigate to most rapidly narrow down the state of X. Such a sequence (which depends on the outcome of the investigation of previous attributes at each stage) is called a decision tree. Usually an attribute with high information gain should be preferred to other attributes.
Read more about Information Gain In Decision Trees: General Definition, Formal Definition, Drawbacks, Constructing A Decision Tree Using Information Gain
Famous quotes containing the words information, gain, decision and/or trees:
“Many more children observe attitudes, values and ways different from or in conflict with those of their families, social networks, and institutions. Yet today’s young people are no more mature or capable of handling the increased conflicting and often stimulating information they receive than were young people of the past, who received the information and had more adult control of and advice about the information they did receive.”
—James P. Comer (20th century)
“When you sympathize with a married woman you either make two enemies or gain one wife and one friend.”
—H.L. (Henry Lewis)
“The decision to have a child is both a private and a public decision, for children are our collective future.”
—Sylvia Ann Hewitt (20th century)
“A long war like this makes you realise the society you really prefer, the home, goats chickens and dogs and casual acquaintances. I find myself not caring at all for gardens flowers or vegetables cats cows and rabbits, one gets tired of trees vines and hills, but houses, goats chickens dogs and casual acquaintances never pall.”
—Gertrude Stein (1874–1946)