Derivation of The Normal Equations
Define the th residual to be
- .
Then can be rewritten
S is minimized when its gradient vector is zero. (This follows by definition: if the gradient vector is not zero, there is a direction in which we can move to minimize it further - see maxima and minima.) The elements of the gradient vector are the partial derivatives of S with respect to the parameters:
The derivatives are
Substitution of the expressions for the residuals and the derivatives into the gradient equations gives
Thus if minimizes S, we have
Upon rearrangement, we obtain the normal equations:
The normal equations are written in matrix notation as
- (where XT is the matrix transpose of X)
The solution of the normal equations yields the vector of the optimal parameter values.
Read more about this topic: Linear Least Squares (mathematics)
Famous quotes containing the word normal:
“We have been weakened in our resistance to the professional anti-Communists because we know in our hearts that our so-called democracy has excluded millions of citizens from a normal life and the normal American privileges of health, housing and education.”
—Agnes E. Meyer (18871970)