Orthogonality Principle - General Formulation

General Formulation

Let be a Hilbert space of random variables with an inner product defined by . Suppose is a closed subspace of, representing the space of all possible estimators. One wishes to find a vector which will approximate a vector . More accurately, one would like to minimize the mean squared error (MSE) between and .

In the special case of linear estimators described above, the space is the set of all functions of and, while is the set of linear estimators, i.e., linear functions of only. Other settings which can be formulated in this way include the subspace of causal linear filters and the subspace of all (possibly nonlinear) estimators.

Geometrically, we can see this problem by the following simple case where is a one-dimensional subspace:

We want to find the closest approximation to the vector by a vector in the space . From the geometric interpretation, it is intuitive that the best approximation, or smallest error, occurs when the error vector, is orthogonal to vectors in the space .

More accurately, the general orthogonality principle states the following: Given a closed subspace of estimators within a Hilbert space and an element in, an element achieves minimum MSE among all elements in if and only if for all

Stated in such a manner, this principle is simply a statement of the Hilbert projection theorem. Nevertheless, the extensive use of this result in signal processing has resulted in the name "orthogonality principle."