Concept · Chapter 6: Language Before Transformers
Embeddings
An embedding is a learned vector for an item — a word, token, document or image — positioned so that items used in similar ways end up close together.
The problem
One-hot vectors give every word its own dimension, so 'cat' and 'kitten' are exactly as different as 'cat' and 'carburettor'.
The solution
Learn a short, dense vector for each item from data, so that the geometry of the vectors reflects how the items are used.
The consequence
Meaning becomes measurable with a dot product, and every later architecture — RNNs, Transformers, retrieval systems — starts by looking up embeddings.
You should understand first
- Vectors
- Dot Product
- Embeddings
The idea
The distributional hypothesis says a word's meaning is reflected in the company it keeps: "coffee" and "tea" appear in similar sentences, so they should get similar representations.
An embedding makes that concrete. Each word (or token) gets a vector of, say, 300 or 4,096 numbers. Training nudges those numbers so that words used in similar contexts end up pointing in similar directions. Nobody decides what dimension 17 means; the structure emerges from the training objective.
Why not one-hot?
With a 50,000-word vocabulary, a one-hot vector for "cat" is 50,000 long with a single 1. The dot product of any two different one-hot vectors is 0: every word is equally unrelated to every other. Embeddings are dense (every entry carries information) and short, and their dot products are meaningful.
Where it appears in AI
- The first layer of every language model is an embedding table: token ID in, vector out.
- Semantic search and RAG embed documents and queries into the same space and retrieve by similarity.
- Recommendation systems embed users and items so that a dot product predicts affinity.
What to remember
- An embedding is a learned vector; nobody hand-assigns what its dimensions mean.
- Similar usage ⇒ nearby vectors (measured by dot product or cosine).
- An LLM's first layer is an embedding lookup table: token ID → vector.
Key papers
A Neural Probabilistic Language Model
Yoshua Bengio, Réjean Ducharme et al. · 2003 · Journal of Machine Learning Research
Learned word representations and next-word probabilities jointly, so similar words could help the model generalize to word sequences it had never counted.
How to read it: Read the abstract and Figure 1 first: the embedding lookup and the probability model are learned together.
Efficient Estimation of Word Representations in Vector Space
Tomas Mikolov, Kai Chen et al. · 2013 · ICLR 2013 (workshop)
Showed that simple, fast models trained on billions of words produce word vectors whose geometry captures meaning — the idea behind every embedding you use today.
How to read it: Read sections 1, 3 and 4. The famous 'king − man + woman ≈ queen' analogy test is in section 4.
GloVe: Global Vectors for Word Representation
Jeffrey Pennington, Richard Socher, Christopher Manning · 2014 · EMNLP 2014
Showed another path to word vectors: fit them to global word co-occurrence statistics rather than only local prediction examples.
Watch
Andrej Karpathy
Building makemore Part 2: MLP
Implements the Bengio et al. 2003 neural language model at character level: an embedding lookup, a hidden layer and a softmax over the next character.
3Blue1Brown
Transformers, the tech behind LLMs | Deep Learning Chapter 5
A visual tour of a GPT from input text to next-token probabilities — ideal before or right after Chapter 7.