Singular Value Decomposition

$$\mathbf{A}=\mathbf{U\Sigma }{\mathbf{V}}^T$$

where $\mathbf{U}$ is an $m\times m$ orthogonal matrix; $\mathbf{\Sigma }$ is an $m\times n$ rectangularly diagonal matrix; $\mathbf{V}$ is an $n\times n$ orthogonal matrix.

Essentially, SVD factors a matrix into three distinct matrices: a rotation, a scaling + dimension adjusting, and another rotation.

Since $\mathbf{V}$ is orthogonal, it holds that ${\mathbf{V}}^T={\mathbf{V}}^{-1}$.

SVD is a generalized extension of eigendecomposition (spectral decomposition), which is

$$\mathbf{A}=\mathbf{Q\Lambda }{\mathbf{Q}}^{-1}$$

where $\mathbf{Q}$ is a $k\times k$ orthogonal matrix whose columns consist of the eigenvectors of $\mathbf{A}$, and $\mathbf{\Lambda }=\mathrm{diag}\left({\lambda }_1,{\lambda }_2,...,{\lambda }_k\right)$; i.e. it is a $k\times k$ diagonal matrix consisting of the eigenvalues of $\mathbf{A}$ on the diagonal.

Example

$$\left[\begin{array}{cc}1& 2\\ 3& 4\end{array}\right]\left[\begin{array}{cccc}1& 2& 3& 4\\ 5& 6& 7& 8\end{array}\right]\left[\begin{array}{cccc}1& 2& 3& 4\\ 5& 6& 7& 8\\ 9& 10& 11& 12\\ 13& 14& 15& 16\end{array}\right]$$