Linear Least Squares (mathematics) - Derivation of The Normal Equations

Derivation of The Normal Equations

Define the th residual to be

.

Then can be rewritten

S is minimized when its gradient vector is zero. (This follows by definition: if the gradient vector is not zero, there is a direction in which we can move to minimize it further - see maxima and minima.) The elements of the gradient vector are the partial derivatives of S with respect to the parameters:

The derivatives are

Substitution of the expressions for the residuals and the derivatives into the gradient equations gives

Thus if minimizes S, we have

Upon rearrangement, we obtain the normal equations:

The normal equations are written in matrix notation as

(where XT is the matrix transpose of X)

The solution of the normal equations yields the vector of the optimal parameter values.