Concept · Chapter 4: Neural Networks
Batch Normalization
Batch normalization rescales each layer's activations to zero mean and unit variance using statistics from the current mini-batch, making deep networks train faster and more stably.
The problem
As training changes earlier layers, the scale of the activations feeding later layers drifts, making training slow and sensitive to learning rate and initialization.
The solution
Normalize each feature over the mini-batch, then apply a learned scale and shift; at inference, use running averages of the statistics.
The consequence
It enabled higher learning rates and much deeper convolutional networks; Transformers instead use layer normalization, which doesn't depend on the batch.
You should understand first
- Derivatives and Gradients
- The Chain Rule
- Vectors
- Dot Product
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Features, Labels and Tasks
- Loss Functions
- Gradient Descent
- Linear Regression
- Probability and Distributions
- Entropy
- Softmax
- Cross-Entropy Loss
- Logistic Regression
- The Perceptron
- Activation Functions
- The Artificial Neuron
- Matrix Multiplication
- Multilayer Perceptron (MLP)
- The Forward Pass
- Computational Graphs and Autodiff
- Backpropagation
- Vanishing and Exploding Gradients
- Batch Normalization
What it does
For each feature, across the examples in a mini-batch, subtract the mean and divide by the standard deviation, then multiply by a learned and add a learned . Activations stay in a predictable range however the earlier layers change.
Batch norm lets networks train faster and with higher learning rates Established. The original explanation — reducing "internal covariate shift" — has been questioned; later work argues it helps mainly by making the optimization landscape smoother Interpretation.
Why Transformers use layer norm instead
Batch norm's statistics depend on the other examples in the batch, which is awkward for variable-length sequences, small batches and generation one token at a time. Layer normalization normalizes each example across its own features instead — same goal, no batch dependence.
What to remember
- Normalize each feature using the mini-batch's mean and variance.
- Learned scale (γ) and shift (β) restore flexibility.
- Behaves differently in training (batch stats) vs inference (running stats).
- Standard in CNNs; Transformers use layer norm instead.
Key papers
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Sergey Ioffe, Christian Szegedy · 2015 · ICML 2015
Batch normalization made deep networks train faster and more reliably with higher learning rates; it became standard in convolutional networks.
How to read it: The paper's explanation ('internal covariate shift') has been questioned since; the technique's usefulness has not.