Concept · Chapter 7: Transformers
Residual Connections
A residual connection adds a layer's input to its output (x + f(x)), so each layer learns a correction instead of a complete replacement.
The problem
Very deep networks were hard to train: signals and gradients degrade as they pass through many layers, and adding layers could make results worse.
The solution
Route the input around each sublayer and add it back to the output, creating a direct path through the whole network.
The consequence
Networks with dozens or hundreds of layers became trainable. In Transformers, the running sum is called the residual stream: every sublayer reads from it and writes back to it.
You should understand first
- Vectors
- Residual Connections
The idea
Instead of , compute
If a layer has nothing useful to add, it can learn and pass its input through unchanged. Learning "a small correction to what's already there" turns out to be far easier than learning a full transformation from scratch.
Why it helps training
During backpropagation the gradient of with respect to includes a term of exactly 1 — a direct path that doesn't shrink however many layers deep you are. This was the key idea of ResNet (2015), which trained image networks with over a hundred layers.
The residual stream
In a Transformer, every attention and feed-forward sublayer is wrapped in a residual connection. A useful mental model: each token has a residual stream — a vector that flows up through the network. Each sublayer reads the current stream, computes something, and adds its result back. Interpretability research uses this view heavily.
What to remember
- output = x + sublayer(x).
- Gives gradients a shortcut back through deep stacks.
- Introduced for image networks (ResNet, 2015) and used around every Transformer sublayer.
- The 'residual stream' view: layers read from and add to a shared running vector.
Key papers
Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang et al. · 2015 · CVPR 2016
Residual (skip) connections made very deep networks trainable. Every Transformer block relies on the same trick.