Singular Value Decomposition - Geometric Meaning

Geometric Meaning

Because U and V are unitary, we know that the columns u₁,...,u_m of U yield an orthonormal basis of Km and the columns v₁,...,v_n of V yield an orthonormal basis of Kn (with respect to the standard scalar products on these spaces).

The linear transformation T :Kn → Km that takes a vector x to Mx has a particularly simple description with respect to these orthonormal bases: we have T(v_i) = σ_i u_i, for i = 1,...,min(m,n), where σ_i is the i-th diagonal entry of Σ, and T(v_i) = 0 for i > min(m,n).

The geometric content of the SVD theorem can thus be summarized as follows: for every linear map T :Kn → Km one can find orthonormal bases of Kn and Km such that T maps the i-th basis vector of Kn to a non-negative multiple of the i-th basis vector of Km, and sends the left-over basis vectors to zero. With respect to these bases, the map T is therefore represented by a diagonal matrix with non-negative real diagonal entries.

To get a more visual flavour of singular values and SVD decomposition —at least when working on real vector spaces— consider the sphere S of radius one in Rn. The linear map T maps this sphere onto an ellipsoid in Rm. Non-zero singular values are simply the lengths of the semi-axes of this ellipsoid. Especially when n=m, and all the singular values are distinct and non-zero, the SVD of the linear map T can be easily analysed as a succession of three consecutive moves : consider the ellipsoid T(S) and specifically its axes ; then consider the directions in Rn sent by T onto these axes. These directions happen to be mutually orthogonal. Apply first an isometry v* sending these directions to the coordinate axes of Rn. On a second move, apply an endomorphism d diagonalized along the coordinate axes and stretching or shrinking in each direction, using the semi-axes lengths of T(S) as stretching coefficients. The composition d v* then sends the unit-sphere onto an ellipsoid isometric to T(S). To define the third and last move u, apply an isometry to this ellipsoid so as to carry it over T(S). As can be easily checked, the composition u d v* coincides with T.