Concept · Chapter 5: Vision, Speech & Reinforcement Learning
ImageNet, AlexNet and ResNet
A large labelled dataset (ImageNet), GPU training (AlexNet) and residual connections (ResNet) turned CNNs from a niche method into the default for vision between 2009 and 2015.
The problem
CNNs worked on digits but had not clearly beaten hand-engineered pipelines on large, varied, real-world photos.
The solution
Train much deeper CNNs on over a million labelled images using GPUs, ReLU and dropout; then add skip connections to go deeper still.
The consequence
Deep learning's breakout moment; the recipe (big data + compute + a learnable architecture) set the pattern later followed by language models.
You should understand first
- Vectors
- Tensors and Shapes
- Images as Tensors
- Dot Product
- Convolution
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Features, Labels and Tasks
- Loss Functions
- Derivatives and Gradients
- Gradient Descent
- Linear Regression
- Probability and Distributions
- Entropy
- Softmax
- Cross-Entropy Loss
- Logistic Regression
- The Perceptron
- Activation Functions
- The Artificial Neuron
- Matrix Multiplication
- Multilayer Perceptron (MLP)
- The Chain Rule
- The Forward Pass
- Computational Graphs and Autodiff
- Backpropagation
- Convolutional Neural Networks
- ImageNet, AlexNet and ResNet
Three ingredients arrive together
Data. ImageNet, introduced at CVPR 2009, organised millions of labelled images under WordNet categories , and its yearly challenge (from 2010) used 1,000 classes. For the first time, a model with tens of millions of parameters had enough examples to learn from.
Compute. In 2012 AlexNet, a deep CNN trained on two GPUs with ReLU activations and dropout, reached 15.3% top-5 error on the challenge, more than 10 points ahead of the next entry . "Top-5" means the right label was among the model's five best guesses.
Depth. Networks kept deepening (VGG in 2014 used 16–19 layers of small 3×3 filters), until simply adding layers began to make training error worse. ResNet (2015) added a shortcut around every pair of layers, output = x + f(x), and trained networks with 152 layers . The same residual connection sits around every sublayer of a Transformer.
Transfer learning
A CNN trained on ImageNet learns general visual features. Chop off its final classifier, attach a new one, and fine-tune on a few thousand images of your own task: this usually beats training from scratch. "Pretrain on a big generic dataset, adapt to a small specific one" is the pattern that language models later made central (Chapter 8).
What to remember
- ImageNet (2009): millions of labelled images; its challenge became the benchmark.
- AlexNet (2012): GPUs, ReLU, dropout; 15.3% top-5 error, more than 10 points ahead of the runner-up.
- ResNet (2015): residual connections made 152-layer networks trainable.
- Pretrained ImageNet CNNs became reusable feature extractors: transfer learning.
Key papers
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang et al. · 2015 · CVPR 2016
Residual (skip) connections made very deep networks trainable. Every Transformer block relies on the same trick.
ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · 2012 · NeurIPS 2012
AlexNet won ImageNet 2012 by a wide margin and triggered the deep-learning era: big data plus GPUs plus deep networks.
ImageNet: A large-scale hierarchical image database
Jia Deng, Wei Dong et al. · 2009 · CVPR 2009
A dataset, not a model: millions of labelled images organised by WordNet categories. Its yearly challenge became the benchmark on which deep CNNs proved themselves in 2012.