Concept · Chapter 7: Transformers
Positional Encoding
Positional encodings add information about each token's position, because attention on its own treats a sentence as an unordered set.
The problem
Self-attention computes the same result if you shuffle the tokens (apart from shuffling the output) — 'dog bites man' and 'man bites dog' would look alike.
The solution
Give each position its own vector and add it to the token embedding (sinusoidal or learned), or build relative position directly into the attention scores.
The consequence
Transformers can use word order, and the choice of scheme affects how well models handle sequences longer than those seen in training — which is why rotary embeddings (RoPE) became popular.
You should understand first
The problem
Look at the attention equation again: every token scores every other token with a dot product. Nothing in it refers to where a token is. Swap two words and their scores swap too — the model sees a bag of words. For language, order carries meaning, so position has to be supplied from outside.
Three answers
1. Sinusoidal encodings (2017). Add a fixed vector to each token embedding, built from sine and cosine waves of different frequencies:
Low dimensions oscillate quickly, high dimensions slowly — a bit like the hands of a clock. Every position gets a unique pattern, and nothing needs to be learned.
2. Learned position embeddings. Treat position like a vocabulary: a table with one learned vector per position (GPT-2, BERT). Simple, but the model has no vector for positions beyond its trained maximum.
3. Relative / rotary positions. Instead of adding position to the input, make the attention score itself depend on the distance between tokens. RoPE rotates each query and key by an angle proportional to its position, so their dot product depends on relative offset. Many recent open LLMs use it; Chapter 11 returns to it.
What to remember
- Attention is permutation-equivariant: without positions, order is invisible.
- Original Transformer: add fixed sine/cosine waves of different frequencies.
- GPT-2/BERT: learn a vector per position.
- Many modern LLMs use RoPE, which rotates queries and keys by position.
Key papers
Attention Is All You Need
Ashish Vaswani, Noam Shazeer et al. · 2017 · NeurIPS 2017
Introduced the Transformer — the architecture behind BERT, GPT and nearly every modern large language model, and later adapted to vision, audio and more.
How to read it: Section 3 is the architecture — read it with Figure 1 open. Sections 3.2.1–3.2.2 contain the attention equation. You can skim the training details on a first pass.