Dimensionality Reduction: PCA, SVD & Manifolds

Compress high-dimensional data while preserving maximum variance via covariance matrix eigendecomposition and SVD.

1. Principal Component Analysis (PCA)

Given mean-centered data \(X\), PCA finds orthogonal projection axes \(\mathbf{w}\) that maximize sample variance \(\mathbf{w}^T \Sigma \mathbf{w}\), where \(\Sigma = \frac{1}{n} X^T X\) is the sample covariance matrix:

\[ \Sigma \mathbf{v}_i = \lambda_i \mathbf{v}_i \]

Eigenvectors \(\mathbf{v}_i\) define the principal component directions, and eigenvalues \(\lambda_i\) indicate the variance captured along each axis.

🎯 Module Mastery Certification Quiz

+100 XP

Why must features be standardized (mean=0, variance=1) before applying PCA?

Features with large numeric scales will artificially dominate the covariance matrix and explain 99% of the variance regardless of actual information content.

PCA cannot calculate negative numbers.

To convert the matrix into a single integer.

To remove all rows from the dataset.