Orthogonality Principle - A Solution To Error Minimization Problems

A Solution To Error Minimization Problems

The following is one way to find the minimum mean square error estimator by using the orthogonality principle.

We want to be able to approximate a vector by

where

is the approximation of as a linear combination of vectors in the subspace spanned by Therefore, we want to be able to solve for the coefficients, so that we may write our approximation in known terms.

By the orthogonality theorem, the square norm of the error vector, is minimized when, for all j,

Developing this equation, we obtain

$\left\langle x,p_{j}\right\rangle =\left\langle \sum_i c_{i}p_{i},p_{j}\right\rangle =\sum_i c_{i}\left\langle p_{i},p_{j}\right\rangle.$

If there is a finite number of vectors, one can write this equation in matrix form as

$\begin{bmatrix} \left\langle x,p_{1}\right\rangle \\ \left\langle x,p_{2}\right\rangle \\ \vdots\\ \left\langle x,p_{n}\right\rangle \end{bmatrix} = \begin{bmatrix} \left\langle p_{1},p_{1}\right\rangle & \left\langle p_{2},p_{1}\right\rangle & \cdots & \left\langle p_{n},p_{1}\right\rangle \\ \left\langle p_{1},p_{2}\right\rangle & \left\langle p_{2},p_{2}\right\rangle & \cdots & \left\langle p_{n},p_{2}\right\rangle \\ \vdots & \vdots & \ddots & \vdots\\ \left\langle p_{1},p_{n}\right\rangle & \left\langle p_{2},p_{n}\right\rangle & \cdots & \left\langle p_{n},p_{n}\right\rangle \end{bmatrix} \begin{bmatrix} c_{1}\\ c_{2}\\ \vdots\\ c_{n}\end{bmatrix}.$

Assuming the are linearly independent, the Gramian matrix can be inverted to obtain

$\begin{bmatrix} c_{1}\\ c_{2}\\ \vdots\\ c_{n}\end{bmatrix} = \begin{bmatrix} \left\langle p_{1},p_{1}\right\rangle & \left\langle p_{2},p_{1}\right\rangle & \cdots & \left\langle p_{n},p_{1}\right\rangle \\ \left\langle p_{1},p_{2}\right\rangle & \left\langle p_{2},p_{2}\right\rangle & \cdots & \left\langle p_{n},p_{2}\right\rangle \\ \vdots & \vdots & \ddots & \vdots\\ \left\langle p_{1},p_{n}\right\rangle & \left\langle p_{2},p_{n}\right\rangle & \cdots & \left\langle p_{n},p_{n}\right\rangle \end{bmatrix}^{-1} \begin{bmatrix} \left\langle x,p_{1}\right\rangle \\ \left\langle x,p_{2}\right\rangle \\ \vdots\\ \left\langle x,p_{n}\right\rangle \end{bmatrix},$

thus providing an expression for the coefficients of the minimum mean square error estimator.

Read more about this topic: Orthogonality Principle

Famous quotes containing the words solution, error and/or problems:

“I herewith commission you to carry out all preparations with regard to ... a total solution of the Jewish question in those territories of Europe which are under German influence.... I furthermore charge you to submit to me as soon as possible a draft showing the ... measures already taken for the execution of the intended final solution of the Jewish question.”
—Hermann Goering (1893–1946)

“There is a lot of difference between tempting and leading into error. God tempts but does not lead into error. To tempt is to provide opportunities for us to do certain things if we do not love God, but putting us under no necessity to do so. To lead into error is to compel a man necessarily to conclude and follow a falsehood.”
—Blaise Pascal (1623–1662)

“Those great ideas which come to you in your sleep just before you awake in morning, those solutions to the world’s problems which, in the light of day, turn out to be duds of the puniest order, couldn’t they be put to some use, after all?”
—Robert Benchley (1889–1945)