Concept · Chapter 7: Transformers
Layer Normalization
Layer normalization rescales each token's vector to zero mean and unit variance (then applies a learned scale and shift), keeping activations in a stable range.
The problem
As vectors pass through many layers, their scale can drift up or down, making training unstable and sensitive to the learning rate.
The solution
Normalize each vector across its own features — independently of other examples in the batch — before (or after) each sublayer.
The consequence
Deep Transformers train stably. Where the normalization sits (before each sublayer, 'pre-LN') and its exact form (e.g. RMSNorm) are small choices that matter at scale.
You should understand first
- Vectors
- Layer Normalization
What it computes
For one token's vector with features:
where and are the mean and variance of 's own entries. After normalization every token vector has a predictable scale; and are learned so the model can undo the normalization where useful.
Pre-LN vs post-LN
The original Transformer normalized after each residual addition. GPT-2 and most later models normalize the input to each sublayer instead, leaving the residual path untouched — which trains more stably in deep models.
What to remember
- Normalize each token's vector: subtract its mean, divide by its standard deviation.
- Learned γ (scale) and β (shift) restore flexibility.
- Pre-LN (before each sublayer) is the common modern placement.
Key papers
Layer Normalization
Jimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton · 2016
The normalization used inside Transformers; it keeps activations at a stable scale regardless of batch size.
On Layer Normalization in the Transformer Architecture
Ruibin Xiong, Yunchang Yang et al. · 2020 · ICML 2020
Explains why modern Transformers put layer normalization before each sublayer ('pre-LN') rather than after it.