Derivation of The Normal Equations
Define the th residual to be
- .
Then can be rewritten
S is minimized when its gradient vector is zero. (This follows by definition: if the gradient vector is not zero, there is a direction in which we can move to minimize it further - see maxima and minima.) The elements of the gradient vector are the partial derivatives of S with respect to the parameters:
The derivatives are
Substitution of the expressions for the residuals and the derivatives into the gradient equations gives
Thus if minimizes S, we have
Upon rearrangement, we obtain the normal equations:
The normal equations are written in matrix notation as
- (where XT is the matrix transpose of X)
The solution of the normal equations yields the vector of the optimal parameter values.
Read more about this topic: Linear Least Squares (mathematics)
Famous quotes containing the word normal:
“Like sleep disturbances, some worries at separation can be expected in the second year. If you accept this, then you will avoid reacting to this anxiety as if it’s your fault. A mother who feels guilty will appear anxious to the child, as if to affirm the child’s anxiety. By contrast, a parent who understands that separation anxiety is normal is more likely to react in a way that soothes and reassures the child.”
—Cathy Rindner Tempelsman (20th century)