Skip to content
Road to Intelligence

Concept · Chapter 4: Neural Networks

Backpropagation

Must knowImplement45 minDifficulty

Backpropagation computes how much every weight in a network contributed to the error, by passing the error backward from the output layer by layer using the chain rule.

The problem

To train with gradient descent we need the derivative of the loss with respect to every weight — including weights in hidden layers that are far from the output.

The solution

Run the forward pass, compute the output error, then move backward: each layer multiplies the incoming error by its local derivatives (weights and activation slopes) to get the error for the layer below and the gradients for its own weights.

The consequence

Gradients for all weights cost about as much as one forward pass, making it practical to train networks with millions — and now trillions — of parameters. It is how every modern neural network learns.

You should understand first

  1. Derivatives and Gradients
  2. The Chain Rule
  3. Vectors
  4. Dot Product
  5. The Turing Test
  6. Symbolic AI
  7. Logic and Rules
  8. Expert Systems
  9. Knowledge Representation
  10. The Knowledge-Acquisition Bottleneck
  11. From Rules to Learning
  12. Supervised, Unsupervised and Self-Supervised Learning
  13. Features, Labels and Tasks
  14. Loss Functions
  15. Gradient Descent
  16. Linear Regression
  17. Probability and Distributions
  18. Entropy
  19. Softmax
  20. Cross-Entropy Loss
  21. Logistic Regression
  22. The Perceptron
  23. Activation Functions
  24. The Artificial Neuron
  25. Matrix Multiplication
  26. Multilayer Perceptron (MLP)
  27. The Forward Pass
  28. Computational Graphs and Autodiff
  29. Backpropagation

Intuition: assigning blame

The network predicted 0.9; the answer was 0. Who's responsible? The output neuron, a bit. Each hidden neuron, in proportion to how strongly it's connected to the output and how active it was. Each input weight, in proportion to how much its hidden neuron was blamed and how large its input was. Backpropagation computes exactly this blame, working backward, and gradient descent then nudges every weight to reduce it.

The four steps (for one example)

  1. Forward pass

    Compute and store every layer's z and a, and the output p.

  2. Loss

    L = −ln p if the label is 1, −ln(1 − p) if it's 0.

  3. Output error

    For a sigmoid output with cross-entropy, δ_out = p − y. Gradient for each output weight: δ_out × (the hidden activation it multiplies).

  4. Hidden error

    δ_hidden = δ_out × (weight to the output) × φ′(z). Gradient for each input weight: δ_hidden × (the input it multiplies). With more layers, repeat.

The equations

δ(L)=y^−y,δ(l)=(W(l+1)⊤δ(l+1))⊙ϕ′(z(l)),∂L∂W(l)=δ(l) a(l−1)⊤\delta^{(L)} = \hat{\mathbf{y}} - \mathbf{y}, \qquad \delta^{(l)} = \big(W^{(l+1)\top}\delta^{(l+1)}\big) \odot \phi'\big(\mathbf{z}^{(l)}\big), \qquad \frac{\partial L}{\partial W^{(l)}} = \delta^{(l)}\,\mathbf{a}^{(l-1)\top}

Try it

Pick a point, then step through Forward pass → Loss → Output error → Hidden error. Dashed lines show each weight's gradient for that one example; Train averages them over all points and takes a step.

Try it · toy model

Neural Network Lab

A real two-layer network you can train, edit and dissect: watch activations flow forward, the decision boundary bend, and backpropagation send each example's error back to every weight.

Implement15 min

The method was popularized for neural networks by Rumelhart, Hinton and Williams in 1986 Established; reverse-mode differentiation and related ideas appeared earlier, e.g. in work by Linnainmaa (1970) and Werbos (1974) Established.

Why should I care?

As a researcher

Architectural choices — activations, normalization, residual connections, attention — are largely judged by how well gradients flow backward through them.

As an engineer

Exploding losses, NaNs, dead units, and the memory cost of training (storing activations for the backward pass) are all backprop phenomena.

Modern systems that depend on it

  • Training every neural network
  • Automatic differentiation frameworks
  • Gradient-based interpretability (saliency)

Historical context

Before

Perceptron learning could train only a single layer; reverse-mode differentiation existed in other fields but wasn't the standard way to train networks.

After

Became the universal training algorithm after 1986; paired with GPUs and large datasets in the 2010s it powered deep learning.

Used today

Every training step of every neural network — including every LLM — runs a forward pass and then backpropagation.

What to remember

  • Forward: compute and store activations. Backward: propagate error from output to input.
  • Output error for sigmoid/softmax + cross-entropy: δ = p − y.
  • Hidden error: δ_hidden = (Wᵀ δ_next) ⊙ φ′(z).
  • Weight gradient = (error at the layer) × (input to the layer).
  • It's the chain rule, organized to reuse shared work.

Key papers

Essential

Learning representations by back-propagating errors

David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams · 1986 · Nature

Showed that backpropagation lets multi-layer networks learn useful internal representations — the algorithm that still trains every neural network.

How to read it: Only four pages in Nature. Read it after the chain-rule concept page.

~25 min readdoi:10.1038/323533a0✓ verified 2026-09-26

Watch