Skip to content
Road to Intelligence

Concept · Chapter 4: Neural Networks

Vanishing and Exploding Gradients

Must knowKnow well25 minDifficulty

In a deep network the gradient reaching early layers is a product of many per-layer factors, so it tends to shrink toward zero or blow up exponentially with depth — making early layers learn far too slowly or unstably.

The problem

Backpropagation multiplies by each layer's weights and activation slopes; over dozens of layers, factors a little below or above 1 compound exponentially.

The solution

Keep each layer's factor near 1: activations with slope 1 (ReLU), careful initialization (Xavier, He), normalization layers, residual connections, gated recurrent units (LSTM), and gradient clipping.

The consequence

Solving it is what made 'deep' learning possible — and residual connections and normalization, invented largely for this reason, are in every Transformer block.

You should understand first

  1. Derivatives and Gradients
  2. The Chain Rule
  3. Vectors
  4. Dot Product
  5. The Turing Test
  6. Symbolic AI
  7. Logic and Rules
  8. Expert Systems
  9. Knowledge Representation
  10. The Knowledge-Acquisition Bottleneck
  11. From Rules to Learning
  12. Supervised, Unsupervised and Self-Supervised Learning
  13. Features, Labels and Tasks
  14. Loss Functions
  15. Gradient Descent
  16. Linear Regression
  17. Probability and Distributions
  18. Entropy
  19. Softmax
  20. Cross-Entropy Loss
  21. Logistic Regression
  22. The Perceptron
  23. Activation Functions
  24. The Artificial Neuron
  25. Matrix Multiplication
  26. Multilayer Perceptron (MLP)
  27. The Forward Pass
  28. Computational Graphs and Autodiff
  29. Backpropagation
  30. Vanishing and Exploding Gradients

The arithmetic

From the chain rule, the gradient at the first layer of an LL-layer network is roughly a product of LL factors, each involving a weight matrix and an activation slope:

∂L∂a(1)  ≈  ∏l=2LW(l)⊤ diag(ϕ′(z(l)))\frac{\partial L}{\partial \mathbf{a}^{(1)}} \;\approx\; \prod_{l=2}^{L} W^{(l)\top}\,\mathrm{diag}\big(\phi'(\mathbf{z}^{(l)})\big)

If each factor shrinks vectors by half, twenty layers shrink the gradient by about a million. If each grows them by half, twenty layers grow it about three-thousand-fold. Nothing in between happens by accident — you have to design for it.

Try it

Try it

Vanishing and Exploding Gradients

Send a gradient backwards through up to 40 layers and see it shrink to nothing or blow up — and how ReLU and good initialization fix it.

Know well6 min

With sigmoid activations the slope never exceeds 0.25, so twenty layers leave the first layer with a gradient around 10−1410^{-14} — it effectively doesn't learn. Switch to ReLU with He initialization and the signal arrives intact. Make the initialization too large and it explodes instead.

The fixes you'll meet again

  • ReLU (slope exactly 1 where active) and careful initialization (scale weights by 1/n1/\sqrt{n} or 2/n\sqrt{2/n}) — next concept.
  • Normalization — batch norm, layer norm.
  • Residual connections — add the input back (x+f(x)x + f(x)) so there's always a path with factor 1 (Chapter 7).
  • LSTM gates for recurrent networks (Chapter 6), and gradient clipping for occasional explosions.

Why should I care?

As a researcher

Much of architecture design since 2010 — ReLU, initialization schemes, batch/layer norm, ResNets, LSTMs, pre-LN Transformers — is about keeping gradients well-scaled through depth or time.

As an engineer

Loss that won't budge, or that suddenly becomes NaN, is usually this. Gradient-norm monitoring and clipping are standard in training pipelines for that reason.

Modern systems that depend on it

  • Residual connections
  • Normalization layers
  • LSTM gating
  • Initialization schemes
  • Gradient clipping

Historical context

Before

Deep networks and long recurrent networks were known to be very hard to train, with little understanding why.

After

Analyses in the 1990s–2010s (Bengio et al. 1994; Glorot & Bengio 2010) and fixes — LSTM, ReLU, careful initialization, batch norm, ResNet — enabled networks with hundreds of layers.

Used today

Every deep architecture is designed around it; training dashboards track gradient norms, and optimizers clip gradients.

What to remember

  • Gradient at layer 1 ≈ product of ~L per-layer factors.
  • Factors < 1 → vanishing (early layers stop learning); > 1 → exploding (NaNs, divergence).
  • Sigmoid's slope ≤ 0.25 makes vanishing almost certain in deep stacks.
  • Fixes: ReLU, He/Xavier init, normalization, residual connections, LSTM gates, clipping.

Key papers

Essential

Deep Residual Learning for Image Recognition

Kaiming He, Xiangyu Zhang et al. · 2015 · CVPR 2016

Residual (skip) connections made very deep networks trainable. Every Transformer block relies on the same trick.

~45 min readarXiv:1512.03385✓ verified 2026-09-26
Important

Learning long-term dependencies with gradient descent is difficult

Yoshua Bengio, Patrice Simard, Paolo Frasconi · 1994 · IEEE Transactions on Neural Networks

Showed why gradients vanish or explode when trained across many steps — the core obstacle for deep and recurrent networks.

~50 min readdoi:10.1109/72.279181✓ verified 2026-09-26