Skip to content
Road to Intelligence

Concept · Chapter 2: The Math Toolkit

Probability of Sequences

Must knowKnow well20 minDifficulty

The probability of a whole sequence equals the product of each element's probability given everything before it — the identity that turns 'model language' into 'predict the next token'.

The problem

There are astronomically many possible sentences; we can't list a probability for each one directly.

The solution

Factor the joint probability into a chain of conditional probabilities: first word, then second given the first, then third given the first two, and so on.

The consequence

A single model that predicts the next token given the past defines a probability for every possible text — the foundation of autoregressive language models.

You should understand first

  1. Probability and Distributions
  2. Conditional Probability and Bayes' Theorem
  3. Probability of Sequences

Intuition

What's the probability of the sentence "the cat sat"? Build it one word at a time: how likely is the to start a sentence; given the, how likely is cat; given the cat, how likely is sat. Multiply those.

Tiny numeric example

  1. Each factor

    P(the)=0.05P(\text{the}) = 0.05,   P(cat∣the)=0.01\;P(\text{cat} \mid \text{the}) = 0.01,   P(sat∣the cat)=0.2\;P(\text{sat} \mid \text{the cat}) = 0.2

  2. Multiply

    0.05×0.01×0.2=0.00010.05 \times 0.01 \times 0.2 = 0.0001

  3. Use logs

    ln⁡0.05+ln⁡0.01+ln⁡0.2≈−3.0−4.6−1.6=−9.2\ln 0.05 + \ln 0.01 + \ln 0.2 \approx -3.0 - 4.6 - 1.6 = -9.2, and e−9.2≈0.0001e^{-9.2} \approx 0.0001. Adding logs avoids numbers so small a computer rounds them to zero.

The equation

P(x1,…,xn)=∏t=1nP(xt∣x1,…,xt−1)⟺log⁡P=∑t=1nlog⁡P(xt∣x<t)P(x_1, \dots, x_n) = \prod_{t=1}^{n} P(x_t \mid x_1, \dots, x_{t-1}) \qquad \Longleftrightarrow \qquad \log P = \sum_{t=1}^{n} \log P(x_t \mid x_{<t})

Where it appears in AI

This identity is why "predict the next token" is a complete recipe for modelling text. Train a network to estimate P(xt∣x<t)P(x_t \mid x_{<t}) well, and you have a model of whole documents; sample one token at a time from it, and you have a generator. The training loss for each factor is cross-entropy.

What to remember

  • P(x₁, …, xₙ) = P(x₁) · P(x₂ | x₁) · … · P(xₙ | x₁, …, xₙ₋₁).
  • This is exact — no approximation — for any sequence.
  • An LLM models each factor: P(next token | all previous tokens).
  • In practice we sum log-probabilities instead of multiplying tiny numbers.