Concept · Chapter 6: Language Before Transformers
GloVe
GloVe learns word vectors from global word co-occurrence statistics, providing another route to distributional geometry.
The problem
Local word-context prediction does not explicitly fit the overall co-occurrence patterns of a corpus.
The solution
Count how often words occur together across the corpus, then learn vectors whose relationships reflect those counts.
The consequence
Word vectors can be built from a global count matrix as well as a local prediction task.
You should understand first
What does the whole corpus say?
If ice co-occurs with solid much more often than steam does, while steam co-occurs more with gas, those count ratios reveal a relationship. GloVe builds a large word-context co-occurrence table and fits vectors so their dot products account for its statistics.
It is a different training route to the same broad kind of object: a dense word vector. Neither method guarantees that every semantic relation is a neat direction in space. Both assign one vector per vocabulary word, so context-dependent meanings are still collapsed together.
What to remember
- Word2vec learns through local prediction; GloVe fits global co-occurrence statistics.
- Both produce one context-independent vector per word.
Key papers
GloVe: Global Vectors for Word Representation
Jeffrey Pennington, Richard Socher, Christopher Manning · 2014 · EMNLP 2014
Showed another path to word vectors: fit them to global word co-occurrence statistics rather than only local prediction examples.