Chapter 9 · How an LLM Is Actually Built

Will It Fit?

Add up the memory a GPU needs to train a real model: weights, gradients, optimizer states and activations. Then shard and recompute until it fits in 80 GB.