Linear Discriminant Analysis - LDA For Two Classes

LDA For Two Classes

Consider a set of observations (also called features, attributes, variables or measurements) for each sample of an object or event with known class y. This set of samples is called the training set. The classification problem is then to find a good predictor for the class y of any sample of the same distribution (not necessarily from the training set) given only an observation .

LDA approaches the problem by assuming that the conditional probability density functions and are both normally distributed with mean and covariance parameters and, respectively. Under this assumption, the Bayes optimal solution is to predict points as being from the second class if the log of the likelihood ratios is below some threshold T, so that;

Without any further assumptions, the resulting classifier is referred to as QDA (quadratic discriminant analysis). LDA also makes the simplifying homoscedasticity assumption (i.e. that the class covariances are identical, so ) and that the covariances have full rank. In this case, several terms cancel and the above decision criterion becomes a threshold on the dot product

for some threshold constant c, where

This means that the criterion of an input being in a class y is purely a function of this linear combination of the known observations.

It is often useful to see this conclusion in geometrical terms: the criterion of an input being in a class y is purely a function of projection of multidimensional-space point onto direction . In other words, the observation belongs to y if corresponding is located on a certain side of a hyperplane perpendicular to . The location of the plane is defined by the threshold c.