To understand dimensionality reduction, we must first grasp the intuition of 'energy' or 'variance' in data. Imagine a cloud of data points in 3D space that mostly lies along a slanted 2D plane. While the data is technically 3D, the most critical information—the variance—is concentrated along two primary axes. Dimensionality reduction is the process of finding these axes and projecting the data onto them, effectively discarding the noise or redundant dimensions that contribute little to the overall structure.
At the heart of this process is the Eigen-decomposition of the covariance matrix. For a centered data matrix $X$, the covariance matrix is defined as $\\Sigma = rac{1}{n-1} X^T X$. By solving the characteristic equation $\\det(\\Sigma - \\lambda I) = 0$, we derive the eigenvalues $\\lambda_i$ and their corresponding eigenvectors $v_i$. The eigenvector $v_1$ associated with the largest eigenvalue $\\lambda_1$ represents the direction of maximum variance in the data. By keeping only the top $k$ eigenvectors, we project the data into a lower-dimensional subspace that captures as much information as possible.
While Eigen-decomposition works for square, positive semi-definite matrices, Singular Value Decomposition (SVD) is a more general framework applicable to any $m × n$ matrix $A$. SVD factorizes the matrix into three components: $A = U \\Sigma V^T$. Here, $U$ is an $m × m$ orthogonal matrix representing the left singular vectors, $\\Sigma$ is an $m × n$ diagonal matrix containing the singular values $\\sigma_i$, and $V^T$ is an $n × n$ orthogonal matrix representing the right singular vectors. These singular values $\\sigma_i$ are the square roots of the eigenvalues of $A^T A$.
The profound connection between SVD and dimensionality reduction lies in the Low-Rank Approximation theorem (Eckart-Young-Mirsky). If we truncate $\\Sigma$ by keeping only the largest $k$ singular values and setting the rest to zero, we create a matrix $\\Sigma_k$. The resulting approximation $A_k = U_k \\Sigma_k V_k^T$ is the best possible rank-$k$ approximation of the original matrix $A$ in terms of the Frobenius norm. This allows us to compress the dataset while minimizing the reconstruction error $\\|A - A_k\\|_F$.
Principal Component Analysis (PCA) is essentially a specific application of SVD. When we center our data (subtract the mean), the right singular vectors $V$ become the principal components of the data. The singular values $\\sigma_i$ relate directly to the variance explained by each component: $ ext{Variance}_i = rac{\\sigma_i^2}{n-1}$. By analyzing the 'Scree Plot' of these values, we can determine the optimal cutoff $k$ where adding more dimensions yields diminishing returns in captured variance.
In summary, the transition from SVD to dimensionality reduction is a journey from geometric rotation to information filtering. By decomposing a matrix into its fundamental singular components, we shift our perspective from raw coordinates to a latent coordinate system. Whether it is for image compression, noise reduction in signal processing, or latent semantic analysis in NLP, the ability to isolate the dominant eigenvalues ensures that we retain the signal while discarding the noise.