Skip to content
Road to Intelligence

Concept · Chapter 2: The Math Toolkit

Loss Functions

Must knowKnow well15 minDifficulty

A loss function turns 'how wrong is the model?' into a single number, so that learning becomes the problem of making that number small.

The problem

'Make the model better' isn't something a computer can act on. We need a precise, differentiable measure of error.

The solution

Define a function of the model's predictions and the true answers that is zero (or minimal) when predictions are right and grows as they get worse — then minimize its average over the data.

The consequence

Every learning problem becomes an optimization problem; choosing the loss is choosing what the model will care about.

Intuition

A teacher grading homework gives one number per student. A loss function grades a model's prediction: 0 for perfect, larger for worse. Average it over the training set and you get a single score for the whole model — the thing training tries to push down.

Tiny numeric example: mean squared error

  1. Predictions vs targets

    A model predicts house prices [2,4][2, 4] (in 100k);thetruepricesare100k); the true prices are [3, 3].</Step><Steptitle="Errors">.</Step> <Step title="Errors">2 - 3 = -1andand4 - 3 = 1.</Step><Steptitle="Squareandaverage">.</Step> <Step title="Square and average">\big((-1)^2 + 1^2\big)/2 = 1$. Squaring makes both errors count and punishes large mistakes more.

The equations

MSE=1N∑i=1N(y^i−yi)2L(θ)=1N∑i=1Nℓ(fθ(xi), yi)\text{MSE} = \frac{1}{N} \sum_{i=1}^{N} (\hat{y}_i - y_i)^2 \qquad\qquad \mathcal{L}(\theta) = \frac{1}{N} \sum_{i=1}^{N} \ell\big(f_\theta(x_i),\, y_i\big)

Where it appears in AI

  • Regression (predicting a number): MSE or relatives.
  • Classification and next-token prediction: cross-entropy, which scores the probability given to the correct answer.
  • Post-training (Chapter 10): losses built from human preferences, such as reward-model and DPO losses.

What to remember

  • Loss = a number measuring how wrong a prediction is; training minimizes its average over the data.
  • Regression: mean squared error, the average of (prediction − target)².
  • Classification and language modeling: cross-entropy.
  • The loss must be differentiable for gradient-based training.
  • You get what you optimize — a poorly chosen loss yields a model good at the wrong thing.

Watch