At the heart of dimensionality reduction lies a simple geometric intuition: high-dimensional data often resides on or near a lower-dimensional manifold. Imagine a cloud of points forming a flattened ellipsoid in 3D space; while we need three coordinates to describe every point, the significant variation occurs primarily along two axes, with the third axis representing negligible noise. Our goal is to identify these principal axes of variation and project the data onto them, effectively rotating our perspective to align with the data's natural structure.
To formalize this, we turn to the concept of Eigenvalues and Eigenvectors, which describe how a linear transformation stretches space along specific directions. For a square symmetric matrix $A$, an eigenvector $v$ is a non-zero vector that does not change direction when the transformation is applied, satisfying the equation $Av = \lambda v$, where $\lambda$ is the corresponding eigenvalue. In the context of data, if $A$ represents the covariance matrix, the eigenvectors point in the directions of maximum variance, and the eigenvalues quantify the magnitude of that variance.
However, real-world data matrices are rarely square or symmetric; they are typically rectangular matrices $X$ of size $m \\× n$, where $m$ is the number of samples and $n$ is the number of features. This limitation necessitates the Singular Value Decomposition (SVD), a powerful factorization that generalizes eigen-decomposition to any rectangular matrix. The SVD factors matrix $X$ into three components: $X = U \Sigma V^T$, where $U$ and $V$ are orthogonal matrices containing left and right singular vectors, and $\Sigma$ is a diagonal matrix containing the singular values.
The connection between SVD and eigen-decomposition is profound and mathematically elegant. The right singular vectors in $V$ are precisely the eigenvectors of the covariance-like matrix $X^T X$, and the squares of the singular values in $\Sigma$ correspond to the eigenvalues of $X^T X$. Specifically, if $\sigma_i$ is a singular value, then $\lambda_i = \sigma_i^2$. This relationship allows us to analyze the variance structure of non-square data matrices by leveraging the robust properties of symmetric positive semi-definite matrices derived from the data.
Dimensionality reduction, specifically Principal Component Analysis (PCA), utilizes this decomposition to compress data by truncating the SVD. By keeping only the top $k$ largest singular values in $\Sigma$ and their corresponding columns in $U$ and $V$, we construct a low-rank approximation $X_k = U_k \Sigma_k V_k^T$. According to the Eckart-Young-Mirsky theorem, this truncated reconstruction is the optimal rank-$k$ approximation of the original matrix in terms of minimizing the Frobenius norm of the reconstruction error, ensuring we lose the least amount of information possible.
Visually, this process rotates the data so that the new coordinate axes align with the directions of greatest spread, then discards the axes with the smallest spread. The first principal component corresponds to the largest singular value and captures the most significant pattern in the dataset, while subsequent components capture orthogonal patterns of decreasing importance. By projecting the original data onto the subspace spanned by the top $k$ right singular vectors, we transform the data from an $n$-dimensional space to a $k$-dimensional space where $k \ll n$.
In practice, this mathematical framework enables us to denoise images, compress large datasets for faster training, and visualize high-dimensional clusters in 2D or 3D. The singular values act as a diagnostic tool; a rapid decay in singular values suggests that the data has a low intrinsic dimension, meaning a small number of components can explain most of the variability. Conversely, a slow decay indicates that the data is truly high-dimensional and complex, resisting simple compression.
Ultimately, mastering SVD and eigenvalues provides the theoretical backbone for understanding not just PCA, but also modern techniques like Latent Semantic Analysis and deep learning initialization strategies. These tools reveal that beneath the apparent complexity of massive datasets lies a structured, often low-dimensional geometry waiting to be uncovered. By aligning our models with these fundamental structures, we build more efficient, interpretable, and robust machine learning systems.