Regularization (Ridge, Lasso) & Bias-Variance Tradeoff

Prevent overfitting by constraining model complexity using L1 Lasso sparsity, L2 Ridge shrinkage, and ElasticNet.

1. Ridge Regression (\(L_2\) Regularization)

Adds an \(L_2\) Euclidean norm penalty on the coefficient weights, shrinking them towards zero without setting them exactly to zero:

\[ \mathcal{L}_{\text{Ridge}} = \sum_{i=1}^n (y_i - \hat{y}_i)^2 + \lambda \sum_{j=1}^p \beta_j^2 \implies \hat{\beta} = (X^T X + \lambda I)^{-1} X^T \mathbf{y} \]

2. Lasso Regression (\(L_1\) Regularization & Automatic Feature Selection)

Adds an \(L_1\) absolute value penalty: \(\mathcal{L}_{\text{Lasso}} = \sum (y_i - \hat{y}_i)^2 + \lambda \sum |\beta_j|\). Due to the diamond geometry of the \(L_1\) constraint, it forces less important coefficients to become **strictly zero**.

🎯 Module Mastery Certification Quiz

+100 XP

Why does L1 Lasso regularization perform automatic feature selection while L2 Ridge does not?

The L1 norm constraint has sharp corners on the coordinate axes, causing the optimization objective to intersect coefficients at exact zero.

Lasso deletes random columns before training.

Ridge divides all coefficients by infinity.

Lasso can only run on boolean data.