Skip to content
Road to Intelligence

Concept · Chapter 2: The Math Toolkit

KL Divergence

Must knowUnderstand20 minDifficulty

KL divergence measures how much one probability distribution differs from another — the extra surprise you pay for using the wrong distribution.

The problem

We often need to compare two distributions — a student model and a teacher, a fine-tuned model and its starting point — with a single number.

The solution

Compute the expected log-ratio of the two distributions' probabilities; it is zero only when they match and positive otherwise.

The consequence

It shows up as a penalty or target throughout modern AI: distillation, the RLHF penalty that keeps a tuned model near its base, variational methods and more.

You should understand first

  1. Probability and Distributions
  2. Entropy
  3. Softmax
  4. Loss Functions
  5. Cross-Entropy Loss
  6. KL Divergence

Intuition

If you design a code for distribution qq but the data actually follows pp, you'll use more bits than necessary. KL divergence is exactly that extra cost. If q=pq = p there's no waste; the further apart they are, the more you pay.

Tiny numeric example (nats)

p=[0.5,0.5]p = [0.5, 0.5] (a fair coin), q=[0.9,0.1]q = [0.9, 0.1] (a model that thinks heads is very likely):

KL(p ∥ q)=0.5ln⁡0.50.9+0.5ln⁡0.50.1≈−0.29+0.80=0.51\mathrm{KL}(p \,\|\, q) = 0.5 \ln\tfrac{0.5}{0.9} + 0.5 \ln\tfrac{0.5}{0.1} \approx -0.29 + 0.80 = 0.51 KL(q ∥ p)=0.9ln⁡0.90.5+0.1ln⁡0.10.5≈0.53−0.16=0.37\mathrm{KL}(q \,\|\, p) = 0.9 \ln\tfrac{0.9}{0.5} + 0.1 \ln\tfrac{0.1}{0.5} \approx 0.53 - 0.16 = 0.37

Different numbers: direction matters.

The equation

KL(p ∥ q)=∑xp(x)log⁡p(x)q(x)=H(p,q)−H(p)\mathrm{KL}(p \,\|\, q) = \sum_x p(x) \log \frac{p(x)}{q(x)} = H(p, q) - H(p)

Where it appears in AI

  • Knowledge distillation (Chapter 11): train a small student to match a large teacher's output distribution by minimizing KL between them.
  • RLHF (Chapter 10): a KL penalty keeps the tuned model from drifting too far from its original behaviour while chasing reward.
  • DPO and other preference methods are derived from objectives with a KL constraint.

What to remember

  • KL(p ‖ q) = Σ p log(p/q) ≥ 0, and = 0 only when p = q.
  • Not symmetric: KL(p ‖ q) ≠ KL(q ‖ p) — it isn't a true distance.
  • Cross-entropy H(p, q) = H(p) + KL(p ‖ q).
  • Used to keep models close to a reference (RLHF), and to match a teacher (distillation).

Key papers

Optional

On Information and Sufficiency

S. Kullback, R. A. Leibler · 1951 · The Annals of Mathematical Statistics

Introduced the divergence now called KL divergence — used in distillation, RLHF's penalty term, variational methods and more.

~45 min readdoi:10.1214/aoms/1177729694✓ verified 2026-09-26

Watch