Skip to content
Road to Intelligence

Concept · Chapter 7: Transformers

Multi-Head Attention

Must knowKnow well25 minDifficulty

Multi-head attention runs several smaller attention operations in parallel, each with its own learned queries, keys and values, so a layer can track several kinds of relationship at once.

The problem

One attention pattern per layer forces a single weighted average per token — it can't simultaneously focus on the subject of a verb and on the previous word.

The solution

Split the model dimension into h heads, give each head its own Q/K/V projections and let it attend independently; then concatenate the heads' outputs and mix them with an output matrix.

The consequence

Heads specialize in different patterns at almost no extra cost, and 'heads' became a unit of analysis in interpretability — and a target of efficiency tricks like multi-query and grouped-query attention.

You should understand first

  1. Vectors
  2. Dot Product
  3. Embeddings
  4. Attention
  5. Probability and Distributions
  6. Softmax
  7. Self-Attention
  8. Multi-Head Attention

The limitation of one head

A single attention operation gives each token one set of weights — one weighted average of the others. But language asks several questions at once. For the word sat in "the cat sat on the mat": who sat? (→ cat) where? (→ mat) what came just before? (→ cat, again). One averaged view blurs these together.

The idea

Run several attention operations side by side, each in a smaller subspace:

headi=Attention(XWQ(i), XWK(i), XWV(i))\text{head}_i = \text{Attention}(XW_Q^{(i)},\, XW_K^{(i)},\, XW_V^{(i)}) MultiHead(X)=Concat(head1,…,headh) WO\text{MultiHead}(X) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)\, W_O

What heads end up doing

Nobody assigns roles to heads; they emerge from training. Analyses of trained models find heads with recognizable behaviour — attending to the previous token, to matching brackets, or to a syntactic relation Established — while many heads have no clean human-readable description, and removing individual heads often changes performance only slightly Interpretation.

Why should I care?

As a researcher

Attention heads are a basic unit in interpretability work (e.g. 'induction heads'), and many architecture papers change how heads share keys and values.

As an engineer

The number of key/value heads largely determines KV-cache size at inference time — the main memory cost of serving long contexts.

Modern systems that depend on it

  • Every Transformer layer
  • Multi-query and grouped-query attention
  • KV cache sizing
  • Interpretability research on heads

Historical context

Before

A single attention distribution per position, as in the Bahdanau-style attention used with RNNs.

After

Multi-query attention (all heads share one key/value) and grouped-query attention (groups of heads share) cut memory for fast inference.

Used today

Essentially every Transformer. The original model used 8 heads; GPT-2 small uses 12; large modern models use dozens.

What to remember

  • h heads, each of size d_model / h — total cost similar to one big head.
  • Each head has its own W_Q, W_K, W_V and its own attention pattern.
  • Outputs are concatenated and projected by W_O back to d_model.
  • Heads can specialize, but what each one does is learned, not assigned.

Key papers

Essential

Attention Is All You Need

Ashish Vaswani, Noam Shazeer et al. · 2017 · NeurIPS 2017

Introduced the Transformer — the architecture behind BERT, GPT and nearly every modern large language model, and later adapted to vision, audio and more.

How to read it: Section 3 is the architecture — read it with Figure 1 open. Sections 3.2.1–3.2.2 contain the attention equation. You can skim the training details on a first pass.

~1 h 15 min readarXiv:1706.03762✓ verified 2026-09-26

Watch