LLM Pretraining, Perplexity & Scaling Laws

The science of next-token prediction, cross-entropy loss, perplexity evaluation, and Chinchilla compute-optimal scaling.

1. Autoregressive Causal Language Modeling

Large Language Models are pretrained by maximizing the log-likelihood of next-token sequences across trillion-token web scale datasets:

\[ \mathcal{L}_{\text{LM}}(\theta) = - \sum_{i=1}^T \log P(x_i \mid x_1, x_2, \dots, x_{i-1}; \theta) \]

2. Perplexity Metric (PPL)

Perplexity is the exponentiated average negative log-likelihood per token, representing the effective branching factor of the model's prediction distribution:

\[ \text{PPL} = \exp\left( - \frac{1}{N} \sum_{i=1}^N \log P(x_i \mid x_{

3. Chinchilla Scaling Laws (Hoffmann et al.)

For compute-optimal training, model parameters \(N\) and training tokens \(D\) should be scaled in equal proportions: for every doubling of model parameters, the training dataset size must also be doubled (\(D \approx 20 \times N\)).

🎯 Module Mastery Certification Quiz

+100 XP

According to the Chinchilla scaling laws, what is the compute-optimal ratio between training tokens (D) and model parameters (N)?

D should be approximately 20 times N (e.g. 70B parameter model requires ~1.4 Trillion tokens).

D should be equal to N / 10.

Parameters should grow 100x faster than dataset tokens.

Dataset tokens are irrelevant once parameters exceed 100 Billion.