Attention Mechanisms & Transformer Math

The heart of modern AI: Derive the Scaled Dot-Product Attention equation and understand Query, Key, and Value matrix projections.

1. The Scaled Dot-Product Attention Equation

The landmark 2017 paper 'Attention Is All You Need' (Vaswani et al.) replaced recurrent networks with pure attention. Given input matrices \(Q\) (Queries), \(K\) (Keys), and \(V\) (Values):

\[ \text{Attention}(Q, K, V) = \text{softmax}\left( \frac{Q K^T}{\sqrt{d_k}} + M \right) V \]

\(Q K^T\): Computes pairwise dot-product compatibility scores between every query token and all key tokens.
\(\sqrt{d_k}\): Scaling factor preventing the dot products from growing excessively large in high dimensions, which would cause the softmax function to enter regions with near-zero gradients.
\(M\): Causal attention mask (for autoregressive decoder models like GPT) where upper triangular elements are set to \(-\infty\) so tokens cannot attend to future tokens.

2. Multi-Head Attention (MHA)

Multi-Head Attention projects \(Q, K, V\) into \(h\) distinct subspaces, allowing the model to simultaneously attend to information from different representation subspaces at different positions:

\[ \text{MHA}(Q,K,V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h) W^O \]

🎯 Module Mastery Certification Quiz

+100 XP

What is the purpose of dividing by sqrt(d_k) in the scaled dot-product attention equation?

To reduce matrix memory size on GPU HBM.

To prevent large magnitude dot products in high dimensions from pushing softmax into regions with vanishingly small gradients.

To enforce causal ordering from left to right.

To invert the key matrix into an orthogonal vector.