Skip to content
Road to Intelligence

Concept · Chapter 2: The Math Toolkit

Stochastic Gradient Descent (SGD)

Must knowKnow well20 minDifficulty

Stochastic gradient descent estimates the gradient from a small random batch of examples instead of the whole dataset, trading a little noise for enormous speed.

The problem

Computing the exact gradient requires a pass over every training example — impossible to do at every step when the dataset has billions of examples.

The solution

At each step, sample a mini-batch (say 32–4,096 examples), compute the average gradient on it, and update. On average, the estimate points the right way.

The consequence

Training scales to any dataset size, the noise itself can help escape poor regions, and batch size becomes a key hyperparameter alongside the learning rate.

You should understand first

  1. Derivatives and Gradients
  2. Loss Functions
  3. Gradient Descent
  4. Probability and Distributions
  5. Expected Value and Variance
  6. Stochastic Gradient Descent (SGD)

Intuition

To learn which way the landscape slopes, you don't need to survey the whole mountain range — a few quick measurements nearby give a rough but useful direction. Take a step, measure again with a fresh random sample, step again. The path wobbles, but it heads downhill.

The equation

wt+1=wt−η⋅1B∑i∈Bt∇ℓi(wt)\mathbf{w}_{t+1} = \mathbf{w}_t - \eta \cdot \frac{1}{B} \sum_{i \in \mathcal{B}_t} \nabla \ell_i(\mathbf{w}_t)

Vocabulary you'll see everywhere

  • Batch size: examples per step. Step (or iteration): one update. Epoch: one full pass through the data.
  • Large LLMs typically see most of their training text only about once — very different from classical ML, where models loop over a small dataset for many epochs.
  • Gradient accumulation (Chapter 9): average gradients over several small batches before updating, to simulate a larger batch on limited memory.

Turn up the gradient noise slider in the Gradient Descent Playground to see SGD's wobble.

What to remember

  • Mini-batch gradient = unbiased but noisy estimate of the full gradient.
  • Noise shrinks like 1/√(batch size).
  • Epoch = one pass through the whole training set.
  • Batch size and learning rate interact; larger batches often allow larger learning rates.

Key papers

Optional

A Stochastic Approximation Method

Herbert Robbins, Sutton Monro · 1951 · The Annals of Mathematical Statistics

The mathematical ancestor of stochastic gradient descent: it showed that noisy, step-by-step updates can still converge to the right answer.

How to read it: A pure mathematics paper. Knowing it exists — and that SGD's convergence story starts here — is enough for now.

~45 min readdoi:10.1214/aoms/1177729586✓ verified 2026-09-26

Watch