Consider the matrix \(X:\mathbb R^m\to \mathbb R^p\), here maybe take a specific matrix \(m = 2\), \(p = 3\): \[X = \begin{bmatrix} 1 & 1 \\ 1 & 3 \\ 1 & -1\end{bmatrix}\] Its image is a plane in \(\mathbb R^3\) spanned by the columns, and it takes the unit sphere to an ellipse in this plane. The SVD decomposition of \(X\) is \[X = U\Sigma V^T\] where \(U\) is \(p\times p\) with columns an orthonormal basis for \(\mathbb R^p\) (these are the left singular vectors of \(X\)), \(V\)is \(m \times m\) with columns an orthonormal basis for \(\mathbb R^m\) these are the right singular values of \(X\)), and \(\Sigma\) is a diagonal \(p\times m\) matrix whose entries are the singular values of \(X\). In our case for our specific \(X\), we have
\[ U \approx \begin{bmatrix} 0.365 & 0.447 & -8.16 \\ -0.912 & 0 & -0.408\\ 0.182 & -.894 & 0.402 \end{bmatrix} \hspace{0.5em} \Sigma \approx \begin{bmatrix} 3.464 & 0 \\ 0 & 1.414 \\ 0 & 0 \end{bmatrix} \hspace{0.5em} V \approx \begin{bmatrix} -0.316 & 0.949 \\ -0.949 & -0.316 \end{bmatrix} \]
Let \(\vec u_i\) be the columns of \(U\) and \(\sigma_j\) the singular values of \(X\). We choose \(\vec u_1\) to be the major axis of the ellipse in the plane, \(\vec u_2\) to be the next most dominant axis of the ellipse, etc up to \(\vec u_p\)to be the most minor axis of the ellipse. The vectors of \(U\) and \(V\) satisfy the relation \[X\vec v_i = \sigma_i\vec u_i\] for all indices \(i\) which make sense.
Here’s the summary.
To summarize, and slightly generalize:
- We started with a \(n \times p\) matrix \(X\).
- The unit sphere in \(\mathbb{R}^p\) is transformed into an ellipsoid living in \(\textrm{Im}(X)\) in the codomain \(\mathbb{R}^n\).
- Let \(r = \textrm{Rank}(X)\). We found the unit vectors \(\vec{u}_1, \vec{u}_2, \dots, \vec{u}_r\) pointing in the direction of the axes of this ellipsoid. We let \(\sigma_1, \sigma_2, \dots \sigma_r\) be the distance from the origin to these vertexes.
- \(\vec{v}_1, \vec{v}_2 , \dots \vec{v}_r\) are chosen to be inverse images of \(\sigma_1 \vec{u}_1, \sigma_2\vec{u}_2, \dots, \sigma_r \vec{u}_r\). It is a small miracle that these end up orthogonal as well.
- If \(n > r\), then we also completed the \(u_j\) to form a basis of \(\mathbb{R}^n\): \(\vec{u}_{r+1}, \vec{u}_{r+2}, \dots \vec{u}_n\) are chosen to be orthonormal and span \(\textrm{Im}(X)^\perp\). (These can be whatever you want as long as its a completion to an orthonormal basis.)
- Similarly, if \(p > r\) then the null space of \(X\) will be \(\textrm{Span}(\vec{v}_1, \vec{v}_2 , \dots \vec{v}_r)^\perp\), and we choose the remaining \(v_j\) to complete an orthonormal basis of \(\mathbb{R}^p\). You could also view this as the image ellipsoid having “degenerate axes” of length \(0\), and extend the \(\sigma_j\) to be \(0\) in this case.
- At the end of this process we have orthonormal bases of both the domain and codomain with:
\[X \vec{v}_j = \sigma_j \vec{u}_j.\]