Rayleigh Quotient - Special Case of Covariance Matrices

Special Case of Covariance Matrices

A covariance matrix M can be represented as the product . Its eigenvalues are positive:

The eigenvectors are orthogonal to one another:

(different eigenvalues, in case of multiplicity, the basis can be orthogonalized).

The Rayleigh quotient can be expressed as a function of the eigenvalues by decomposing any vector on the basis of eigenvectors:

, where is the coordinate of x orthogonally projected onto

which, by orthogonality of the eigenvectors, becomes:

R(M,x) = \frac{\sum _{i=1} ^n \alpha _i ^2 \lambda _i}{\sum _{i=1} ^n \alpha _i ^2} = \sum_{i=1}^n \lambda_i \frac{(x'v_i)^2}{ (x'x)( v_i' v_i)}

In the last representation we can see that the Rayleigh quotient is the sum of the squared cosines of the angles formed by the vector x and each eigenvector, weighted by corresponding eigenvalues.

If a vector maximizes, then any vector (for ) also maximizes it, one can reduce to the Lagrange problem of maximizing under the constraint that .

Since all the eigenvalues are non-negative, the problem is convex and the maximum occurs on the edges of the domain, namely when and (when the eigenvalues are ordered in decreasing magnitude).

Alternatively, this result can be arrived at by the method of Lagrange multipliers. The problem is to find the critical points of the function

,

subject to the constraint I.e. to find the critical points of

where is a Lagrange multiplier. The stationary points of occur at

and

Therefore, the eigenvectors of M are the critical points of the Rayleigh Quotient and their corresponding eigenvalues are the stationary values of R.

This property is the basis for principal components analysis and canonical correlation.

Read more about this topic:  Rayleigh Quotient

Famous quotes containing the words special and/or case:

    “The line that I am urging as today’s conventional wisdom is not a denial of consciousness. It is often called, with more reason, a repudiation of mind. It is indeed a repudiation of mind as a second substance, over and above body. It can be described less harshly as an identification of mind with some of the faculties, states, and activities of the body. Mental states and events are a special subclass of the states and events of the human or animal body.”
    —Willard Van Orman Quine (b. 1908)

    “Without metaphor the handling of general concepts such as culture and civilization becomes impossible, and that of disease and disorder is the obvious one for the case in point. Is not crisis itself a concept we owe to Hippocrates? In the social and cultural domain no metaphor is more apt than the pathological one.”
    —Johan Huizinga (1872–1945)