Orthogonality Principle - A Solution To Error Minimization Problems

A Solution To Error Minimization Problems

The following is one way to find the minimum mean square error estimator by using the orthogonality principle.

We want to be able to approximate a vector by

where

is the approximation of as a linear combination of vectors in the subspace spanned by Therefore, we want to be able to solve for the coefficients, so that we may write our approximation in known terms.

By the orthogonality theorem, the square norm of the error vector, is minimized when, for all j,

Developing this equation, we obtain

$\left\langle x,p_{j}\right\rangle =\left\langle \sum_i c_{i}p_{i},p_{j}\right\rangle =\sum_i c_{i}\left\langle p_{i},p_{j}\right\rangle.$

If there is a finite number of vectors, one can write this equation in matrix form as

$\begin{bmatrix} \left\langle x,p_{1}\right\rangle \\ \left\langle x,p_{2}\right\rangle \\ \vdots\\ \left\langle x,p_{n}\right\rangle \end{bmatrix} = \begin{bmatrix} \left\langle p_{1},p_{1}\right\rangle & \left\langle p_{2},p_{1}\right\rangle & \cdots & \left\langle p_{n},p_{1}\right\rangle \\ \left\langle p_{1},p_{2}\right\rangle & \left\langle p_{2},p_{2}\right\rangle & \cdots & \left\langle p_{n},p_{2}\right\rangle \\ \vdots & \vdots & \ddots & \vdots\\ \left\langle p_{1},p_{n}\right\rangle & \left\langle p_{2},p_{n}\right\rangle & \cdots & \left\langle p_{n},p_{n}\right\rangle \end{bmatrix} \begin{bmatrix} c_{1}\\ c_{2}\\ \vdots\\ c_{n}\end{bmatrix}.$

Assuming the are linearly independent, the Gramian matrix can be inverted to obtain

$\begin{bmatrix} c_{1}\\ c_{2}\\ \vdots\\ c_{n}\end{bmatrix} = \begin{bmatrix} \left\langle p_{1},p_{1}\right\rangle & \left\langle p_{2},p_{1}\right\rangle & \cdots & \left\langle p_{n},p_{1}\right\rangle \\ \left\langle p_{1},p_{2}\right\rangle & \left\langle p_{2},p_{2}\right\rangle & \cdots & \left\langle p_{n},p_{2}\right\rangle \\ \vdots & \vdots & \ddots & \vdots\\ \left\langle p_{1},p_{n}\right\rangle & \left\langle p_{2},p_{n}\right\rangle & \cdots & \left\langle p_{n},p_{n}\right\rangle \end{bmatrix}^{-1} \begin{bmatrix} \left\langle x,p_{1}\right\rangle \\ \left\langle x,p_{2}\right\rangle \\ \vdots\\ \left\langle x,p_{n}\right\rangle \end{bmatrix},$

thus providing an expression for the coefficients of the minimum mean square error estimator.

Read more about this topic: Orthogonality Principle

Famous quotes containing the words solution, error and/or problems:

“Any solution to a problem changes the problem.”
—R.W. (Richard William)

“We shall yet acknowledge His wisdom and our own error therein.”
—Abraham Lincoln (1809–1865)

“The problems of all of humanity can only be solved by all of humanity.”
—Friedrich Dürrenmatt (1921–1990)