Orthogonality Principle - A Solution To Error Minimization Problems

A Solution To Error Minimization Problems

The following is one way to find the minimum mean square error estimator by using the orthogonality principle.

We want to be able to approximate a vector by

where

is the approximation of as a linear combination of vectors in the subspace spanned by Therefore, we want to be able to solve for the coefficients, so that we may write our approximation in known terms.

By the orthogonality theorem, the square norm of the error vector, is minimized when, for all j,

Developing this equation, we obtain

$\left\langle x,p_{j}\right\rangle =\left\langle \sum_i c_{i}p_{i},p_{j}\right\rangle =\sum_i c_{i}\left\langle p_{i},p_{j}\right\rangle.$

If there is a finite number of vectors, one can write this equation in matrix form as

$\begin{bmatrix} \left\langle x,p_{1}\right\rangle \\ \left\langle x,p_{2}\right\rangle \\ \vdots\\ \left\langle x,p_{n}\right\rangle \end{bmatrix} = \begin{bmatrix} \left\langle p_{1},p_{1}\right\rangle & \left\langle p_{2},p_{1}\right\rangle & \cdots & \left\langle p_{n},p_{1}\right\rangle \\ \left\langle p_{1},p_{2}\right\rangle & \left\langle p_{2},p_{2}\right\rangle & \cdots & \left\langle p_{n},p_{2}\right\rangle \\ \vdots & \vdots & \ddots & \vdots\\ \left\langle p_{1},p_{n}\right\rangle & \left\langle p_{2},p_{n}\right\rangle & \cdots & \left\langle p_{n},p_{n}\right\rangle \end{bmatrix} \begin{bmatrix} c_{1}\\ c_{2}\\ \vdots\\ c_{n}\end{bmatrix}.$

Assuming the are linearly independent, the Gramian matrix can be inverted to obtain

$\begin{bmatrix} c_{1}\\ c_{2}\\ \vdots\\ c_{n}\end{bmatrix} = \begin{bmatrix} \left\langle p_{1},p_{1}\right\rangle & \left\langle p_{2},p_{1}\right\rangle & \cdots & \left\langle p_{n},p_{1}\right\rangle \\ \left\langle p_{1},p_{2}\right\rangle & \left\langle p_{2},p_{2}\right\rangle & \cdots & \left\langle p_{n},p_{2}\right\rangle \\ \vdots & \vdots & \ddots & \vdots\\ \left\langle p_{1},p_{n}\right\rangle & \left\langle p_{2},p_{n}\right\rangle & \cdots & \left\langle p_{n},p_{n}\right\rangle \end{bmatrix}^{-1} \begin{bmatrix} \left\langle x,p_{1}\right\rangle \\ \left\langle x,p_{2}\right\rangle \\ \vdots\\ \left\langle x,p_{n}\right\rangle \end{bmatrix},$

thus providing an expression for the coefficients of the minimum mean square error estimator.

Read more about this topic: Orthogonality Principle

Famous quotes containing the words solution, error and/or problems:

“Let us begin to understand the argument.
There is a solution to everything: Science.”
—Allen Tate (1899–1979)

“Theoretically, I grant you, there is no possibility of error in necessary reasoning. But to speak thus “theoretically,” is to use language in a Pickwickian sense. In practice, and in fact, mathematics is not exempt from that liability to error that affects everything that man does.”
—Charles Sanders Peirce (1839–1914)

“The question of place and climate is most closely related to the question of nutrition. Nobody is free to live everywhere; and whoever has to solve great problems that challenge all his strength actually has a very restricted choice in this matter. The influence of climate on our metabolism, its retardation, its acceleration, goes so far that a mistaken choice of place and climate can not only estrange a man from his task but can actually keep it from him: he never gets to see it.”
—Friedrich Nietzsche (1844–1900)