Linear & Polynomial Regression (OLS Derivations)

Derive the closed-form Ordinary Least Squares normal equation and evaluate goodness-of-fit via R² and Adjusted R².

1. The Ordinary Least Squares (OLS) Normal Equation

Given design matrix \(X \in \mathbb{R}^{n \times p}\) and target vector \(\mathbf{y} \in \mathbb{R}^n\), minimizing the sum of squared residuals \(L(\beta) = \|\mathbf{y} - X\beta\|_2^2\) yields the closed-form analytical solution:

\[ \hat{\beta} = (X^T X)^{-1} X^T \mathbf{y} \]

2. Coefficient of Determination (\(R^2\)) & Adjusted \(R^2\)

\(R^2 = 1 - \frac{\text{SS}_{\text{res}}}{\text{SS}_{\text{tot}}}\). Because adding arbitrary features always inflates \(R^2\), we use Adjusted \(R^2\) to penalize feature count \(p\):

\[ R_{\text{adj}}^2 = 1 - \left( \frac{1 - R^2}{n - p - 1} \right) (n - 1) \]

🎯 Module Mastery Certification Quiz

+100 XP

What does the Normal Equation beta = (X^T X)^(-1) X^T y require to be computationally solvable?

The matrix X^T X must be invertible (non-singular / full column rank with no perfect multicollinearity).

The target y must be integer values.

The dataset must contain fewer than 50 rows.

All features must be scaled to [0, 1].