Derivation of The Normal Equations
Define the th residual to be
- .
Then can be rewritten
S is minimized when its gradient vector is zero. (This follows by definition: if the gradient vector is not zero, there is a direction in which we can move to minimize it further - see maxima and minima.) The elements of the gradient vector are the partial derivatives of S with respect to the parameters:
The derivatives are
Substitution of the expressions for the residuals and the derivatives into the gradient equations gives
Thus if minimizes S, we have
Upon rearrangement, we obtain the normal equations:
The normal equations are written in matrix notation as
- (where XT is the matrix transpose of X)
The solution of the normal equations yields the vector of the optimal parameter values.
Read more about this topic: Linear Least Squares (mathematics)
Famous quotes containing the word normal:
“A few ideas seem to be agreed upon. Help none but those who help themselves. Educate only at schools which provide in some form for industrial education. These two points should be insisted upon. Let the normal instruction be that men must earn their own living, and that by the labor of their hands as far as may be. This is the gospel of salvation for the colored man. Let the labor not be servile, but in manly occupations like that of the carpenter, the farmer, and the blacksmith.”
—Rutherford Birchard Hayes (1822–1893)