Concept · Chapter 6: Language Before Transformers
Text as Data
A language model receives a sequence of discrete text units and must map each unit to a vocabulary ID before any neural computation can begin.
The problem
A network accepts numbers, while written language arrives as characters and words with variable boundaries.
The solution
Choose a unit, build a vocabulary, and map each observed unit to an integer ID.
The consequence
The choice of unit determines sequence length, vocabulary size, and what happens to unseen words.
Three ways to cut the same sentence
"unhappiness grows" can become two words, a longer sequence of characters, or pieces such as "un", "happi", "ness", "grows". These are examples of possible segmentations, not a universal tokenizer output. Word units give shorter sequences but struggle with rare and new words. Characters rarely have that problem but make the sequence much longer. Word pieces are a compromise used by many later models.
After splitting text, a vocabulary assigns each distinct unit an integer. ID 17 is not "more meaningful" than ID 16. The integer is an address for a lookup table; a learned vector will supply the numerical representation.
Chapter 8 builds a real tokenizer. Here the unit is usually a whole word so the early language-modeling ideas stay visible.
What to remember
- A vocabulary is a lookup table from text units to IDs; IDs have no inherent semantic order.
- Word, character and word-piece units trade shorter sequences against vocabulary size and unseen-word handling.