Concept · Chapter 6: Language Before Transformers
Word2Vec
Word2vec trains compact word vectors with simple local-context prediction tasks rather than a full neural language model.
The problem
Earlier neural language models learned useful vectors but were expensive to train on very large text corpora.
The solution
Use CBOW to predict a word from its neighbours or skip-gram to predict neighbours from a word; train efficiently on many word-context pairs.
The consequence
Useful word geometry became practical at scale, while each word still had one context-independent vector.
You should understand first
A smaller job that scales
In "the cat sat on the mat", CBOW tries to predict sat from nearby words such as cat and on. Skip-gram reverses the direction: use sat to predict words near it. These tasks are simpler than predicting every next word from a full sentence history, but their learned vectors capture useful regularities in how words are used.
The first word2vec paper introduced these two efficient architectures. A later 2013 paper described negative sampling: teach the model to distinguish a real word-context pair from a few sampled false pairs, avoiding an expensive full-vocabulary calculation for each example.
Some offsets in these vectors support analogies, but the famous examples are selective. A single vector for bank cannot separately represent a river bank and a financial bank in context, and patterns in training text can encode social bias. Treat the geometry as a useful statistical representation, not a dictionary of fixed meanings.
What to remember
- CBOW: context → centre word. Skip-gram: centre word → nearby context words.
- Negative sampling, described in a later 2013 word2vec paper, compares real word-context pairs with sampled non-pairs.
- A word has one vector regardless of sentence, so polysemy and bias remain problems.
Key papers
Efficient Estimation of Word Representations in Vector Space
Tomas Mikolov, Kai Chen et al. · 2013 · ICLR 2013 (workshop)
Showed that simple, fast models trained on billions of words produce word vectors whose geometry captures meaning — the idea behind every embedding you use today.
How to read it: Read sections 1, 3 and 4. The famous 'king − man + woman ≈ queen' analogy test is in section 4.
Distributed Representations of Words and Phrases and their Compositionality
Tomas Mikolov, Ilya Sutskever et al. · 2013 · NeurIPS 2013
Introduced negative sampling as a faster way to train skip-gram word vectors and explicitly discussed the limits of word-only representations.