Decision Trees, Information Gain & Random Forests

Understand recursive binary splitting, Gini Impurity, Shannon Entropy, and Bootstrap Aggregation (Bagging).

1. Splitting Criteria: Shannon Entropy & Gini Impurity

Decision trees evaluate candidate feature splits to maximize the reduction in node impurity:

\[ H(S) = - \sum_{k=1}^K p_k \log_2(p_k) \quad \text{(Entropy)}, \qquad I_G(S) = 1 - \sum_{k=1}^K p_k^2 \quad \text{(Gini Impurity)} \]

2. Random Forests: Bagging + Feature Sub-sampling

Random Forests train an ensemble of \(B\) decorrelated deep trees by combining **Bootstrap Aggregation** (sampling rows with replacement) with random feature sub-sampling (evaluating only \(\sqrt{p}\) features at each node split).

🎯 Module Mastery Certification Quiz

+100 XP

Why does Random Forest randomly select a subset of sqrt(p) features at each split instead of considering all p features?

To decorrelate the individual decision trees in the ensemble, preventing dominant strong features from creating identical tree structures.

To reduce the number of trees required from 100 to 2.

To eliminate the need for training data.

Because trees cannot process more than 10 features.