Backpropagation Calculus & Gradient Optimizers

Master the chain rule of calculus for reverse-mode automatic differentiation and modern adaptive optimizers (AdamW).

1. Reverse-Mode Automatic Differentiation

Backpropagation computes the gradient of the scalar loss function \(L\) with respect to every weight parameter \(w_{ij}\) by applying the multivariate chain rule across the computational graph:

\[ \frac{\partial L}{\partial w_{ij}} = \frac{\partial L}{\partial a_j} \cdot \frac{\partial a_j}{\partial z_j} \cdot \frac{\partial z_j}{\partial w_{ij}} = \delta_j \cdot a_i \]

2. The AdamW Optimizer (Adaptive Moment Estimation with Decoupled Weight Decay)

Adam maintains running exponential moving averages of both the past gradients (1st moment / momentum \(m_t\)) and the uncentered variance (2nd moment \(v_t\)):

\[ m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t, \quad v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2 \] \[ \hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}, \quad \theta_{t+1} = \theta_t - \eta \left( \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} + \lambda \theta_t \right) \]

🎯 Module Mastery Certification Quiz

+100 XP

Why does AdamW decouple weight decay (L2 regularization) from the gradient update step?

To reduce GPU VRAM consumption by 50%.

Standard Adam divides L2 gradients by the second moment, causing weights with large historical gradients to be under-regularized.

To eliminate the need for a learning rate schedule.

To allow training on binary integers instead of floating point numbers.