Gradient Boosting Machines, XGBoost & LightGBM

Master sequential residual learning, 2nd-order Taylor expansions, and extreme gradient boosting optimizations.

1. The Gradient Boosting Principle

Unlike bagging (which trains trees independently in parallel), Gradient Boosting builds trees sequentially: each new tree \(f_m(x)\) fits the pseudo-residuals (negative gradient of the loss) of the previous ensemble \(F_{m-1}(x)\):

\[ r_{im} = - \left[ \frac{\partial L(y_i, F(x_i))}{\partial F(x_i)} \right]_{F = F_{m-1}} \]

2. XGBoost (Extreme Gradient Boosting — Chen & Guestrin)

XGBoost approximates the loss function using a **second-order Taylor expansion** incorporating both first-order gradients \(g_i\) and second-order Hessians \(h_i\), with explicit leaf regularization \(\gamma T + \frac{1}{2} \lambda \sum w_j^2\).

🎯 Module Mastery Certification Quiz

+100 XP

What mathematical advancement enables XGBoost to optimize arbitrary custom loss functions with high convergence speed?

Second-order Taylor expansion using both first-order gradients (g) and second-order Hessians (h).

Converting all trees into neural networks.

Running without a learning rate parameter.

Removing tree depth constraints.