Expected Value and Variance

Must knowKnow well15 minDifficulty

The expected value is the probability-weighted average outcome, and the variance measures how far outcomes typically spread around it.

The problem

A distribution is a whole list of numbers; we often need one number for 'typical value' and one for 'how uncertain'.

The solution

Summarize with the mean (weighted average) and the variance (average squared distance from the mean); its square root is the standard deviation.

The consequence

Training objectives become expected losses, noisy gradient estimates can be reasoned about, and experimental results can carry error bars.

You should understand first

Probability and Distributions
Expected Value and Variance

Tiny numeric example

A fair die: $E[X] = (1 + 2 + 3 + 4 + 5 + 6)/6 = 3.5$ — an outcome that never actually occurs, but the long-run average.

Variance: the squared distances from 3.5 are 6.25, 2.25, 0.25, 0.25, 2.25, 6.25; their average is about 2.92, so the standard deviation is about 1.71.

The equations

E[X] = \sum_x x\, P(x) \qquad \operatorname{Var}[X] = E\big[(X - E[X])^2\big] \qquad \operatorname{sd}\big(\bar{X}_n\big) = \frac{\sigma}{\sqrt{n}}

Where it appears in AI

The training objective is an expectation: the average loss over all possible training examples, which we approximate with the examples we have.
Stochastic gradient descent uses the average gradient of a small random batch as an estimate of the true gradient; its noise shrinks like $1/\sqrt{\text{batch size}}$ .
Error bars in papers are standard deviations or standard errors across seeds or samples — see sampling and uncertainty.

What to remember

E[X] = Σ x · P(x): the probability-weighted average.
Var[X] = E[(X − E[X])²]; standard deviation = √Var.
Training minimizes an expected loss over the data distribution.
Averaging n independent samples shrinks the standard deviation by √n.