Fine-Tuning, LoRA & Parameter Efficiency (PEFT)

Adapt massive 70B+ LLMs on consumer GPUs using Low-Rank Adaptation (LoRA) matrix decomposition and 4-bit NormalFloat quantization (QLoRA).

1. Low-Rank Adaptation (LoRA) Mathematical Foundation

Full fine-tuning updates all \(d \times k\) parameters \(W_0\). Hu et al. observed that weight updates \(\Delta W\) have a low 'intrinsic rank'. LoRA freezes \(W_0\) and decomposes \(\Delta W\) into two low-rank matrices \(B \in \mathbb{R}^{d \times r}\) and \(A \in \mathbb{R}^{r \times k}\) with rank \(r \ll \min(d, k)\):

\[ h = W_0 x + \Delta W x = W_0 x + \frac{\alpha}{r} B A x \]

When \(d = 4096\) and rank \(r = 16\), the number of trainable parameters drops by **over 99.6%**, reducing VRAM requirements from 160GB to under 16GB!

2. QLoRA (Dettmers et al.)

QLoRA quantizes the frozen base model weights into **4-bit NormalFloat (NF4)** and uses Double Quantization + Paged Optimizers to prevent CUDA out-of-memory spikes during gradient backpropagation.

🎯 Module Mastery Certification Quiz

+100 XP

How does LoRA (Low-Rank Adaptation) reduce VRAM requirements by over 95% during LLM fine-tuning?

By freezing the original weight matrix W0 and learning only two small low-rank adapter matrices B and A.

By pruning 90% of the attention heads permanently.

By running the model entirely on CPU RAM.

By reducing text context window to 64 tokens.