Skip to content
Road to Intelligence

Part I · Foundations

Chapter 4

Neural Networks

Stacking simple units until they learn their own features.

3 h 30 min core path13 concepts2 interactives

In one sentenceNeural networks compose many simple weighted units, and backpropagation tells every weight how to change to reduce the error.

The building block

One neuron

Chapter 3 ended with a limit: classical models learn only the final step, from hand-made features to an answer. Neural networks learn the features too. They're built from one simple unit, repeated millions of times.

The perceptron (1958) was the first learning neuron: a weighted sum of inputs, and a threshold. It can learn any pattern separable by a straight line — and nothing else. XOR defeats it.

ConceptThe PerceptronKnow wellMust know

The perceptron (1958) is a single artificial neuron that outputs 1 if a weighted sum of its inputs exceeds a threshold, with a simple rule for learning the weights from mistakes.

Open the concept page →

The modern artificial neuron keeps the weighted sum and replaces the hard threshold with a smooth activation function, so the unit can be trained with gradients:

a=ϕ(w⊤x+b)a = \phi(\mathbf{w}^\top\mathbf{x} + b)

With a sigmoid as ϕ\phi, that's exactly the logistic regression from Chapter 3. The magic isn't in one neuron; it's in what happens when you connect many.

ConceptThe Artificial NeuronImplementMust know

An artificial neuron computes a weighted sum of its inputs, adds a bias, and passes the result through a nonlinear activation function — a dot product plus a bend.

Open the concept page →
ConceptActivation FunctionsKnow wellMust know

An activation function is the nonlinearity applied after each neuron's weighted sum; without it, any stack of layers would collapse into a single linear map.

Open the concept page →

Architecture

Layers

Put a layer of neurons between the input and the output and each hidden unit can learn its own boundary; the output unit combines them. That's enough to solve XOR, and — with enough hidden units — to approximate essentially any function. A whole layer is computed at once as a matrix multiplication followed by an activation, and the forward pass is just that, repeated layer by layer.

h=ϕ(W1x+b1),p=σ(w2⊤h+b2)\mathbf{h} = \phi(W_1\mathbf{x} + \mathbf{b}_1), \qquad p = \sigma(\mathbf{w}_2^\top\mathbf{h} + b_2)
ConceptMultilayer Perceptron (MLP)Know wellMust know

A multilayer perceptron stacks layers of neurons — input, one or more hidden layers, output — so that each layer builds new features out of the previous layer's outputs.

Open the concept page →
ConceptThe Forward PassImplementMust know

The forward pass is computing a network's output from its input: layer by layer, multiply by weights, add biases, apply activations.

Open the concept page →

Learning

Learning: backpropagation

To train the network with gradient descent we need the gradient of the loss with respect to every weight — including weights deep inside, far from the output. Backpropagation computes all of them in one backward sweep: start from the output error, and pass it back through the network, multiplying by each layer's weights and activation slopes along the way. It's the chain rule from Chapter 2, organized so no work is repeated.

This is the chapter's centrepiece. Train the network below on XOR; set hidden units to 0 and watch a single neuron fail; pick a point and step through the four stages of backpropagation to see its error reach every weight; click any edge to change a weight yourself.

Try it · toy model

Neural Network Lab

A real two-layer network you can train, edit and dissect: watch activations flow forward, the decision boundary bend, and backpropagation send each example's error back to every weight.

Implement15 min
ConceptBackpropagationImplementMust know

Backpropagation computes how much every weight in a network contributed to the error, by passing the error backward from the output layer by layer using the chain rule.

Open the concept page →

Modern frameworks never make you do this by hand. They record every operation in a computational graph during the forward pass and apply the chain rule backward automatically — loss.backward().

ConceptComputational Graphs and AutodiffKnow wellMust know

A computational graph records every elementary operation of a calculation, so that software can compute exact derivatives of the output with respect to every input automatically.

Open the concept page →

The obstacle

Why deep networks were hard to train

Backprop was popularized in 1986, yet deep networks didn't take off until the 2010s. One major reason is in the arithmetic: the gradient reaching an early layer is a product of many per-layer factors. If those factors are a little below 1, it vanishes; a little above 1, it explodes. With sigmoid activations — whose slope is never more than 0.25 — twenty layers leave the first layer with essentially no learning signal.

Try it

Vanishing and Exploding Gradients

Send a gradient backwards through up to 40 layers and see it shrink to nothing or blow up — and how ReLU and good initialization fix it.

Know well6 min
ConceptVanishing and Exploding GradientsKnow wellMust know

In a deep network the gradient reaching early layers is a product of many per-layer factors, so it tends to shrink toward zero or blow up exponentially with depth — making early layers learn far too slowly or unstably.

Open the concept page →

The fixes came one by one: ReLU activations, initialization scaled to each layer's width, normalization layers, residual connections, and gated units for sequences.

ConceptWeight InitializationUnderstandShould know

Initialization sets the random starting weights at a scale that keeps signals and gradients roughly the same size from layer to layer, so deep networks can start learning at all.

Open the concept page →

Practice

Keeping training stable

Two more tools you'll see everywhere. Batch normalization keeps each layer's activations at a stable scale, allowing faster training; Transformers use its cousin, layer normalization. Dropout randomly switches off units during training so the network can't rely on any one of them — a regularizer, like those in Chapter 3.

ConceptBatch NormalizationUnderstandShould know

Batch normalization rescales each layer's activations to zero mean and unit variance using statistics from the current mini-batch, making deep networks train faster and more stably.

Open the concept page →
ConceptDropoutKnow wellShould know

Dropout randomly switches off a fraction of neurons at each training step, so the network can't rely on any single unit and must learn redundant, more general features.

Open the concept page →

The payoff

What depth buys: learned representations

Here is why all this effort was worth it. A deep network's hidden layers are feature detectors that nobody designed. Trained end to end on raw pixels, audio or text, early layers learn simple patterns and later layers combine them into more abstract ones. The feature-engineering bottleneck of Chapter 3 disappears: the same recipe — layers, backprop, data — works in every domain, and gets better with more data and compute.

ConceptRepresentation LearningKnow wellMust know

Representation learning means the network learns its own features: each layer transforms the data into a new representation, from simple patterns in early layers to abstract concepts in later ones.

Open the concept page →

In 2012 a deep convolutional network, trained on GPUs with ReLUs and dropout, won the ImageNet challenge by more than ten percentage points. The deep-learning era began.

What came next

From MLPs to Transformers

A plain MLP ignores the structure of its input: that pixels have neighbours, that words come in order. The next decades of architecture design built that structure in — and each design fixed its predecessor's main limitation.

ConceptFrom MLPs to Transformers: The Architecture StoryUnderstandMust know

Neural network architectures evolved by building the structure of the data into the network — convolutions for images, recurrence for sequences — until attention offered a more general way to connect everything.

Open the concept page →

The road to Chapter 7

  1. Multilayer Perceptron (MLP)
  2. CNNs (Chapter 5)
  3. RNNs and LSTMs (Chapter 6)
  4. Attention
  5. The Transformer Block

Why it matters

Why it matters

Every model from here on is a neural network trained this way. A Transformer is a particular arrangement of matrix multiplications, activations and normalizations; an LLM is a very large one; training it is backpropagation plus a variant of gradient descent, billions of times over.

The design vocabulary is set. Residual connections, normalization, careful initialization and the choice of activation, which you'll meet inside every Transformer block, exist because of the gradient-flow problems in this chapter.

Concepts in this chapter

Mark each one as you go. Must-know concepts are the core path.

What do I actually need to remember?

  • A neuron: a = φ(w·x + b) — a dot product, a bias and a nonlinearity.
  • One neuron draws one linear boundary; it can't learn XOR. Hidden layers fix that.
  • Without nonlinear activations, any number of layers collapses into one linear map.
  • A layer is a matrix multiplication plus an activation; the forward pass repeats it.
  • Backpropagation sends the output error backward with the chain rule to get every weight's gradient.
  • Weight gradient = (error at the layer's output) × (the layer's input).
  • Deep gradients are products of many factors → they vanish or explode; ReLU, careful initialization, normalization and residuals fix this.
  • Dropout and weight decay regularize; batch/layer norm stabilize training.
  • Deep networks learn their own features — representation learning — which is why they overtook hand-crafted pipelines.
  • Architectures evolved MLP → CNN → RNN/LSTM → attention → Transformer, each fixing the last one's limitation.

You do not need to memorize everything else. This list is the revision sheet.

Key papers

Essential

The perceptron: A probabilistic model for information storage and organization in the brain.

F. Rosenblatt · 1958 · Psychological Review

Introduced the perceptron — a neuron model that learns its weights from examples. Every neural network descends from it.

Problem
McCulloch–Pitts neurons could compute, but their connections had to be designed by hand.
What was new
A learning rule that adjusts connection strengths from experience, so the system improves at recognizing patterns.
~1 h readdoi:10.1037/h0042519✓ verified 2026-09-26
Essential

Learning representations by back-propagating errors

David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams · 1986 · Nature

Showed that backpropagation lets multi-layer networks learn useful internal representations — the algorithm that still trains every neural network.

Problem
Single-layer perceptrons can't learn many functions, and no practical method was widely known for training the hidden layers of deeper networks.
What was new
Propagate error gradients backward through the network with the chain rule, and demonstrate that hidden units learn meaningful features.

How to read it: Only four pages in Nature. Read it after the chain-rule concept page.

~25 min readdoi:10.1038/323533a0✓ verified 2026-09-26
Important

Multilayer feedforward networks are universal approximators

Kurt Hornik, Maxwell Stinchcombe, Halbert White · 1989 · Neural Networks

Proved that a network with a single hidden layer can approximate essentially any continuous function, given enough units — the 'universal approximation' result.

Problem
Were multi-layer networks fundamentally limited, as single-layer perceptrons had been shown to be?
What was new
A proof that standard feedforward networks with one hidden layer are universal approximators.

How to read it: A theoretical result: it says a good network exists, not that training will find it or how big it must be.

~40 min readdoi:10.1016/0893-6080(89)90020-8✓ verified 2026-09-26
Optional

Approximation by superpositions of a sigmoidal function

G. Cybenko · 1989 · Mathematics of Control, Signals, and Systems

An independent universal-approximation proof for networks with sigmoid hidden units.

Problem
Can sums of sigmoid functions approximate arbitrary continuous functions?
What was new
Yes — finite superpositions of sigmoids are dense in the space of continuous functions on a bounded domain.
~30 min readdoi:10.1007/BF02551274✓ verified 2026-09-26
Important

Learning long-term dependencies with gradient descent is difficult

Yoshua Bengio, Patrice Simard, Paolo Frasconi · 1994 · IEEE Transactions on Neural Networks

Showed why gradients vanish or explode when trained across many steps — the core obstacle for deep and recurrent networks.

Problem
Recurrent networks failed to learn dependencies spanning long time gaps.
What was new
Analysis showing a trade-off between storing information robustly and propagating useful gradients, so gradients shrink exponentially with distance.
~50 min readdoi:10.1109/72.279181✓ verified 2026-09-26
Essential

Gradient-based learning applied to document recognition

Yann LeCun, Léon Bottou et al. · 1998 · Proceedings of the IEEE

The LeNet paper: convolutional networks trained end-to-end with gradient descent for handwriting recognition, deployed commercially for reading cheques.

Problem
Handwriting recognition relied on hand-designed feature extractors plus a trainable classifier.
What was new
Learn the features too: convolutional networks trained end to end, plus whole systems trained with gradients.

How to read it: Long (46 pages). Sections I–II explain why learned features beat hand-designed ones — the heart of Chapter 4.

~1 h 30 min readdoi:10.1109/5.726791✓ verified 2026-09-26
Important

Understanding the difficulty of training deep feedforward neural networks

Xavier Glorot, Yoshua Bengio · 2010 · AISTATS 2010

Explained why deep networks with sigmoid units and naive initialization trained poorly, and introduced 'Xavier' initialization.

Problem
Deep networks trained with standard gradient descent from random initialization got stuck or trained very slowly.
What was new
An analysis of how activations and gradients change across layers, and an initialization that keeps their variance roughly constant.
~40 min read✓ verified 2026-09-26
Essential

ImageNet Classification with Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · 2012 · NeurIPS 2012

AlexNet won ImageNet 2012 by a wide margin and triggered the deep-learning era: big data plus GPUs plus deep networks.

Problem
Image recognition relied on hand-engineered features and had plateaued on large, varied datasets.
What was new
A deep convolutional network trained on GPUs with ReLUs and dropout on 1.2 million images, cutting top-5 error dramatically.
~40 min read✓ verified 2026-09-26
Important

Dropout: A Simple Way to Prevent Neural Networks from Overfitting

Nitish Srivastava, Geoffrey Hinton et al. · 2014 · Journal of Machine Learning Research

Dropout — randomly switching off units during training — became a standard, simple regularizer for neural networks.

Problem
Large neural networks overfit, and averaging many separately trained networks is expensive.
What was new
Randomly drop units during training, effectively training an ensemble of thinned networks that share weights; use the full network at test time.
~50 min read✓ verified 2026-09-26
Important

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Sergey Ioffe, Christian Szegedy · 2015 · ICML 2015

Batch normalization made deep networks train faster and more reliably with higher learning rates; it became standard in convolutional networks.

Problem
Training deep networks was slow and sensitive to initialization and learning rate.
What was new
Normalize each layer's activations using the mean and variance of the current mini-batch, with learned scale and shift.

How to read it: The paper's explanation ('internal covariate shift') has been questioned since; the technique's usefulness has not.

~40 min readarXiv:1502.03167✓ verified 2026-09-26
Important

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Kaiming He, Xiangyu Zhang et al. · 2015 · ICCV 2015

Introduced the initialization ('He' or 'Kaiming' initialization) suited to ReLU networks, plus the PReLU activation.

Problem
Initializations designed for sigmoid-like units made very deep ReLU networks fail to train.
What was new
Scale initial weights by √(2 / fan-in) to keep activation variance stable through ReLU layers.
~40 min readarXiv:1502.01852✓ verified 2026-09-26
Optional

Gaussian Error Linear Units (GELUs)

Dan Hendrycks, Kevin Gimpel · 2016

The GELU activation, a smooth relative of ReLU used in BERT, GPT-2 and many later Transformers.

Problem
ReLU's hard cut-off at zero isn't smooth and ignores the size of negative inputs.
What was new
Weight each input by the probability a standard Gaussian falls below it: x·Φ(x).
~30 min readarXiv:1606.08415✓ verified 2026-09-26

Watch

What came next?

Chapter 5

Vision, Speech & Reinforcement Learning

Not all data is text, and not all learning comes with correct answers attached.

This chapter is being written.