Concept · Chapter 4: Neural Networks
From MLPs to Transformers: The Architecture Story
Neural network architectures evolved by building the structure of the data into the network — convolutions for images, recurrence for sequences — until attention offered a more general way to connect everything.
The problem
A plain MLP treats every input as unrelated to every other: it ignores that pixels have neighbours and words come in order, so it needs vast data and parameters.
The solution
Design architectures with built-in assumptions (inductive biases): CNNs share local filters across an image; RNNs and LSTMs process sequences step by step with memory; attention lets any element directly look at any other.
The consequence
Each architecture solved its predecessor's key limitation, and the Transformer — built on attention — became the general-purpose architecture behind modern AI.
You should understand first
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Vectors
- Features, Labels and Tasks
- Loss Functions
- Derivatives and Gradients
- Gradient Descent
- Linear Regression
- Probability and Distributions
- Entropy
- Softmax
- Cross-Entropy Loss
- Logistic Regression
- Hand-Crafted Features vs Learned Features
- Dot Product
- The Perceptron
- Activation Functions
- The Artificial Neuron
- Matrix Multiplication
- Multilayer Perceptron (MLP)
- Representation Learning
- From MLPs to Transformers: The Architecture Story
Each architecture fixes the last one's problem
| Architecture | Key idea | Problem it solved | Limitation that remained |
|---|---|---|---|
| MLP | fully connected layers | learning features at all | ignores structure; parameters explode for images |
| CNN | small filters slid across the input, shared weights | images: locality and translation | fixed-size local view; not for sequences of varying length |
| RNN | a hidden state updated token by token | sequences of any length | information fades over long distances; slow, sequential training |
| LSTM / GRU | learned gates controlling memory | longer-range memory | still sequential; still struggles with very long dependencies |
| Attention → Transformer | every position looks directly at every other | long-range dependencies, parallel training | cost grows with the square of sequence length |
The table is the road from here to Chapter 7. Chapter 5 covers CNNs in vision; Chapter 6 covers word embeddings, RNNs, LSTMs and the attention mechanism that was first added to them; Chapter 7 is the Transformer.
What to remember
- MLP: fully connected, no built-in structure.
- CNN: local filters shared across positions — images (LeNet 1998, AlexNet 2012).
- RNN: processes sequences one step at a time with a hidden state.
- LSTM/GRU: gates that preserve information over longer sequences (1997).
- Transformer: attention connects every position directly (2017).
Key papers
Attention Is All You Need
Ashish Vaswani, Noam Shazeer et al. · 2017 · NeurIPS 2017
Introduced the Transformer — the architecture behind BERT, GPT and nearly every modern large language model, and later adapted to vision, audio and more.
How to read it: Section 3 is the architecture — read it with Figure 1 open. Sections 3.2.1–3.2.2 contain the attention equation. You can skim the training details on a first pass.
Long Short-Term Memory
Sepp Hochreiter, Jürgen Schmidhuber · 1997 · Neural Computation
LSTM added gated memory cells so recurrent networks could keep information over long sequences; it dominated sequence modelling until Transformers.
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou et al. · 1998 · Proceedings of the IEEE
The LeNet paper: convolutional networks trained end-to-end with gradient descent for handwriting recognition, deployed commercially for reading cheques.
How to read it: Long (46 pages). Sections I–II explain why learned features beat hand-designed ones — the heart of Chapter 4.