Concept · Chapter 7: Transformers
Encoder, Decoder & Encoder–Decoder
The same Transformer block is wired three ways: encoder-only models (BERT) read in both directions to understand text, decoder-only models (GPT) predict the next token to generate it, and encoder–decoder models (T5) map one sequence to another.
The problem
Different tasks need different information flow: classification wants full two-way context; generation must not see the future; translation has a separate input and output.
The solution
Choose the attention mask and training objective: bidirectional attention with masked-token prediction, causal attention with next-token prediction, or an encoder plus a causal decoder joined by cross-attention.
The consequence
Encoders dominated NLP benchmarks around 2018–2020; decoder-only models, trained on next-token prediction at scale, became the basis of today's general-purpose LLMs.
You should understand first
Same block, different wiring
Trained toNext-token prediction: each position predicts the following token, seeing only the past.
Good forGeneration. One simple objective that scales well; it became the dominant design for LLMs.
Squares show which tokens (columns) each token (rows) may attend to. Cross-attention lets every decoder position look at every encoder position.
Why decoder-only won for LLMs
A common explanation is simplicity and data efficiency: next-token prediction needs no labels, uses every token of every document as a training signal, and makes the same model both the learner and the generator Interpretation. Chapter 8 follows what happened when this design was scaled up.
Why should I care?
As a researcher
Model papers describe themselves in these terms, and the choice determines what the model can be trained on and evaluated for.
As an engineer
Embedding and reranking models are often encoders; chat models are decoders; many speech and translation systems are encoder–decoders. Picking the wrong family for a task is a common mistake.
Modern systems that depend on it
- BERT and sentence-embedding models
- GPT-style chat models
- T5, Whisper and translation models
Historical context
Before
Task-specific architectures, often RNN-based encoder–decoders with attention.
After
Decoder-only scaling (GPT-3 and successors) became the dominant path to general-purpose models (Chapter 8).
Used today
All three are in use: encoders for embeddings and classification, decoders for chat and code, encoder–decoders for translation and speech recognition.
What to remember
- Encoder-only (BERT): bidirectional attention, masked-token objective; understanding and embeddings.
- Decoder-only (GPT): causal attention, next-token objective; generation.
- Encoder–decoder (original Transformer, T5): encoder reads input, decoder generates with cross-attention.
- The blocks are nearly identical — the mask and the objective make the difference.
Key papers
Attention Is All You Need
Ashish Vaswani, Noam Shazeer et al. · 2017 · NeurIPS 2017
Introduced the Transformer — the architecture behind BERT, GPT and nearly every modern large language model, and later adapted to vision, audio and more.
How to read it: Section 3 is the architecture — read it with Figure 1 open. Sections 3.2.1–3.2.2 contain the attention equation. You can skim the training details on a first pass.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang et al. · 2018 · NAACL 2019
Made 'pretrain once, fine-tune everywhere' the default in NLP, using an encoder-only Transformer that reads context in both directions.
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Colin Raffel, Noam Shazeer et al. · 2019 · JMLR 2020
Framed every NLP task as text in, text out, using an encoder–decoder Transformer — and ran a huge, careful set of ablations that is still a model of empirical method.
How to read it: Long (67 pages). Read the introduction and Section 3.2's architecture comparison; treat the rest as a reference.