Skip to content
Road to Intelligence

Concept · Chapter 2: The Math Toolkit

Conditional Probability and Bayes' Theorem

Must knowKnow well30 minDifficulty

Conditional probability asks how likely something is given what you already know, and Bayes' theorem tells you how to flip it — from P(evidence | cause) to P(cause | evidence).

The problem

We usually know how likely the evidence is under each explanation (a test's accuracy, a word's frequency in spam), but we want the reverse: how likely each explanation is given the evidence.

The solution

Combine the prior (how common the explanation is) with the likelihood (how well it predicts the evidence), and normalize over all explanations.

The consequence

It gives a principled way to update beliefs, underlies classic classifiers like Naive Bayes, and explains why base rates matter when interpreting any model's alarms.

You should understand first

  1. Probability and Distributions
  2. Conditional Probability and Bayes' Theorem

Intuition: restrict your view

Conditional probability P(A∣B)P(A \mid B) means: among the cases where B happened, how often does A happen? You shrink the world to just the B cases and look again.

Bayes' theorem is the tool for reversing a condition. A medical test is described by P(positive∣sick)P(\text{positive} \mid \text{sick}), but a patient wants P(sick∣positive)P(\text{sick} \mid \text{positive}). These are very different numbers.

Tiny numeric example — counting people

  1. Start with 1,000 people

    1% have the condition: 10 sick, 990 healthy.

  2. Apply the test

    It catches 90% of the sick: 9 true positives. It also flags 9% of the healthy: about 89 false positives.

  3. Look only at the positives

    98 people test positive, of whom 9 are sick: 9/98≈9 / 98 \approx 9%.

A "90% accurate" test, and a positive result still means you're probably healthy. The rarity of the condition — the prior — dominates.

The equation

P(H∣E)=P(E∣H) P(H)P(E),P(E)=∑hP(E∣h) P(h)P(H \mid E) = \frac{P(E \mid H)\, P(H)}{P(E)}, \qquad P(E) = \sum_{h} P(E \mid h)\, P(h)

Try it

Try it

Bayes with 1,000 People

A rare condition, an imperfect test: see why a positive result can still mean you're probably fine — Bayes' theorem as counting.

Know well6 min

Where it appears in AI

  • Language models are conditional models: everything an LLM does is estimate P(next token∣previous tokens)P(\text{next token} \mid \text{previous tokens}).
  • Naive Bayes, a classic text classifier (Chapter 3), applies Bayes' theorem with a simplifying independence assumption.
  • Evaluating detectors: when you deploy a classifier for rare events, the base rate decides whether its alerts are mostly right or mostly wrong — the same arithmetic as the medical test.

Why should I care?

As a researcher

Conditioning is everywhere: P(next token | context), P(label | input), P(data | parameters). Bayesian reasoning also underlies uncertainty estimation and many generative models.

As an engineer

Any detector — fraud, toxicity, prompt-injection — faces base rates: a '99% accurate' filter on rare events can still raise mostly false alarms.

Modern systems that depend on it

  • Language modeling (P(next | previous))
  • Naive Bayes classifiers
  • Calibration and uncertainty
  • Evaluating detectors on rare events

Historical context

Before

Bayes' theorem goes back to Thomas Bayes and Pierre-Simon Laplace in the 18th century.

After

Naive Bayes spam filters in the 1990s–2000s; Bayesian machine learning; and the conditional-probability view of language models.

Used today

In every conditional model — a language model is a giant estimator of P(next token | all previous tokens) — and in reasoning about precision on rare events.

What to remember

  • P(A | B) = P(A and B) / P(B): restrict attention to the cases where B happened.
  • Bayes: P(H | E) = P(E | H) · P(H) / P(E).
  • Posterior ∝ likelihood × prior.
  • With rare events, false positives can outnumber true positives even for accurate tests.

Watch