Softmax

Intuition

A model scores three candidate next words: cat 2.0, dog 1.0, car −1.0. Those numbers are useful for ranking but they aren't probabilities: one is negative and they don't sum to 1. Softmax is the conversion: make everything positive, keep the ranking, and normalize.

Why exponentiate rather than just shift and divide? Because $e^z$ is always positive, it grows fast (so a clearly better score wins clearly), and it pairs naturally with logarithms in the cross-entropy loss used to train almost every classifier and language model.

Tiny numeric example

1. 1

   Exponentiate each score

   $e^{2.0} \approx 7.39$, $\;e^{1.0} \approx 2.72$, $\;e^{-1.0} \approx 0.37$

2. 2

   Add them up

   $7.39 + 2.72 + 0.37 = 10.48$

3. 3

   Divide each by the total

   cat $= 7.39/10.48 \approx 0.71$, $\;$ dog $\approx 0.26$, $\;$ car $\approx 0.04$. They sum to 1 (up to rounding).

A one-point gap in score (cat vs dog) became a ratio of $e^1 \approx 2.7$ in probability. That's the "exponentially more weight" in the one-sentence summary.

The equation

$$\operatorname{softmax}(\mathbf{z})_i = \frac{e^{z_i / T}}{\sum_{j} e^{z_j / T}}$$

+Deep diveWhy computers subtract the maximum firstShould know

$e^{1000}$ overflows a floating-point number. Because only differences between scores matter, implementations compute $\operatorname{softmax}(\mathbf{z} - \max(\mathbf{z}))$, which gives the identical result with every exponent ≤ 0. You'll see this "max trick" in every real implementation — including the one running the interactives on this site.

Where it appears in AI

• Classification: the last layer of a classifier outputs one score (a logit) per class; softmax turns them into class probabilities.
• Attention: each token's similarity scores with every other token go through softmax to become attention weights.
• Language models: the model's logits over its whole vocabulary become the next-token distribution. Sampling with a temperature is exactly the $T$ in the equation.