Skip to content
Road to Intelligence

Concept · Chapter 2: The Math Toolkit

Gradient Descent

Must knowImplement35 minDifficulty

Gradient descent minimizes a loss by repeatedly nudging every parameter a small step in the direction that decreases the loss fastest — the negative gradient.

The problem

A model may have billions of parameters and no formula for the best values; we need a general procedure that improves any differentiable model.

The solution

Compute the gradient of the loss with respect to all parameters, move each parameter a little against it (scaled by the learning rate), and repeat.

The consequence

One simple loop trains everything from linear regression to GPT — with the learning rate as the most important knob, and non-convex landscapes meaning no guarantee of the global best.

You should understand first

  1. Derivatives and Gradients
  2. Loss Functions
  3. Gradient Descent

Intuition

Blindfolded on a hilly landscape, trying to reach the lowest point: feel which way the ground slopes, take a step downhill, repeat. You'll get lower and lower. You might end up in a small hollow rather than the deepest valley — but in the high-dimensional landscapes of neural networks, the hollows you reach are usually good enough.

Tiny numeric example

Minimize L(w)=(w−3)2L(w) = (w - 3)^2, whose gradient is 2(w−3)2(w - 3). Start at w=0w = 0 with learning rate η=0.1\eta = 0.1:

  1. Step 1

    gradient =2(0−3)=−6= 2(0 - 3) = -6;   w←0−0.1×(−6)=0.6\;w \leftarrow 0 - 0.1 \times (-6) = 0.6

  2. Step 2

    gradient =2(0.6−3)=−4.8= 2(0.6 - 3) = -4.8;   w←0.6+0.48=1.08\;w \leftarrow 0.6 + 0.48 = 1.08

  3. Keep going

    Each step shrinks the distance to 3 by 20%: 3→2.4→1.92→⋯→03 \to 2.4 \to 1.92 \to \dots \to 0. We converge to w=3w = 3.

  4. Now with η = 1.1

    Each step multiplies the distance by 1−2η=−1.21 - 2\eta = -1.2: it overshoots, flips side, and grows. Diverged.

The equation

wt+1=wt−η ∇L(wt)\mathbf{w}_{t+1} = \mathbf{w}_t - \eta\, \nabla L(\mathbf{w}_t)

Try it

Try it · toy model

Gradient Descent Playground

Drop a point on a loss landscape and watch gradient descent, momentum and Adam race to the bottom. Push the learning rate until training diverges; add noise to see stochastic gradient descent.

Know well15 min

Where it appears in AI

Every time a model "learns", this loop runs: forward pass → loss → gradient (via backpropagation) → update. Real training adds three refinements covered next: gradients from random mini-batches (SGD), momentum and per-parameter step sizes (Adam), and learning-rate schedules (warm up, then decay).

Why should I care?

As a researcher

Optimization choices — learning rate, schedule, optimizer — often matter as much as architecture, and many papers' improvements turn out to be optimization effects.

As an engineer

Diverging losses, slow training and wasted GPU hours usually trace back to learning-rate and optimizer settings.

Modern systems that depend on it

  • Training every neural network
  • SGD, momentum, Adam/AdamW
  • Learning-rate schedules and warm-up
  • Fine-tuning and LoRA

Historical context

Before

Closed-form solutions (e.g. least squares) for simple models; Cauchy described the method of steepest descent in 1847.

After

Stochastic gradient descent made it practical on huge datasets; momentum and adaptive methods like Adam made it faster and more robust.

Used today

Every modern neural network, including every LLM, is trained with a variant of gradient descent — typically AdamW on mini-batches.

What to remember

  • Update: w ← w − η · ∇L(w).
  • η (learning rate) too small → painfully slow; too large → overshoot and diverge.
  • On non-convex losses it finds a good local region, not a guaranteed global minimum.
  • Ill-conditioned (ravine-shaped) losses make plain GD zig-zag — momentum and Adam help.
  • Stop condition in practice: a fixed budget of steps, not 'reaching the minimum'.

Key papers

Important

An overview of gradient descent optimization algorithms

Sebastian Ruder · 2016

The standard readable survey of SGD, momentum, RMSprop, Adam and friends — one paper that explains the whole optimizer family tree.

How to read it: Very approachable. Read it after trying the Gradient Descent Playground.

~40 min readarXiv:1609.04747✓ verified 2026-09-26
Essential

Learning representations by back-propagating errors

David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams · 1986 · Nature

Showed that backpropagation lets multi-layer networks learn useful internal representations — the algorithm that still trains every neural network.

How to read it: Only four pages in Nature. Read it after the chain-rule concept page.

~25 min readdoi:10.1038/323533a0✓ verified 2026-09-26

Watch