Self-Attention

Start from the simplest version

Forget Q, K and V for a moment. Take one sentence:

the cat sat because it was tired

To understand "it", a reader looks back and connects it to "cat". Self-attention gives the model a way to do the same thing: each word asks "which other words matter to me?" and pulls in information from them.

We already have the tools:

1. Every word is a vector (an embedding).
2. The dot product scores how aligned two vectors are.
3. Softmax turns a list of scores into weights that sum to 1.
4. A weighted sum mixes vectors according to those weights.

That's attention. Build it one step at a time:

Why queries, keys and values?

Scoring raw embeddings against each other has a flaw: a word is always most similar to itself, and "what I'm looking for" is forced to be the same as "what I am". So each token makes three different vectors, each through its own learned matrix:

Vector	Role	Analogy
Query $q = xW_Q$	what this token is looking for	the search box
Key $k = xW_K$	what this token offers to others	the item's index card
Value $v = xW_V$	the information actually passed along	the item's contents

The model learns $W_Q, W_K, W_V$ during training, so it learns what to look for and what to advertise.

The equation

$$\text{Attention}(Q, K, V) = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V$$

Causal masking

A GPT-style model predicts the next token, so while training, position 5 must not peek at positions 6, 7, 8. We set those scores to $-\infty$ before the softmax, which makes their weights exactly 0. The attention matrix becomes lower-triangular.

The price: O(n²)

Every token scores every other token, so a sequence of $n$ tokens needs an $n \times n$ table. Doubling the context length quadruples that work and memory. Much modern engineering — KV caches, FlashAttention, grouped-query attention — exists to live with this cost.