Momentum and Adam

Momentum: a ball, not a hiker

A hiker takes each step based only on the current slope. A ball carries velocity: in a ravine, the side-to-side pushes cancel out while the push along the ravine keeps adding up.

$$\mathbf{v}_{t+1} = \beta\, \mathbf{v}_t + \nabla L(\mathbf{w}_t), \qquad \mathbf{w}_{t+1} = \mathbf{w}_t - \eta\, \mathbf{v}_{t+1}$$

With $\beta = 0.9$, the velocity is roughly an average of the last ~10 gradients.

Adam: momentum plus per-parameter step sizes

Some parameters get big gradients, others tiny ones; one learning rate can't suit both. Adam keeps two running averages per parameter — of the gradient ($m$) and of the squared gradient ($v$) — and steps by their ratio:

$$\mathbf{w}_{t+1} = \mathbf{w}_t - \eta \, \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$$

Why AdamW

AdamW — Adam with weight decay applied directly to the weights rather than through the gradient — is the standard optimizer for training Transformers and LLMs Established. One practical cost: Adam stores $m$ and $v$ for every parameter — two extra numbers per weight, so the optimizer state holds twice as many values as the model itself. At LLM scale that memory is a big deal (Chapter 9).