Concept · Chapter 4: Neural Networks
Activation Functions
An activation function is the nonlinearity applied after each neuron's weighted sum; without it, any stack of layers would collapse into a single linear map.
The problem
Composing linear functions only ever gives another linear function, so a deep network of purely linear layers is no more powerful than one layer.
The solution
Apply a simple nonlinear function to each neuron's output — historically sigmoid or tanh, now mostly ReLU and smooth variants like GELU.
The consequence
Nonlinearity is what lets depth add expressive power — and the choice of activation matters for training: sigmoid's small slope fuels vanishing gradients, while ReLU's slope of 1 lets gradients pass.
You should understand first
- Vectors
- Dot Product
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Features, Labels and Tasks
- Loss Functions
- Derivatives and Gradients
- Gradient Descent
- Linear Regression
- Probability and Distributions
- Entropy
- Softmax
- Cross-Entropy Loss
- Logistic Regression
- The Perceptron
- Activation Functions
Why nonlinearity is essential
Two linear layers in a row, , equal one linear layer . A hundred linear layers still draw a straight decision boundary. Insert a nonlinearity between them — — and the network can bend, fold and combine regions. The ReLU toggle in the Matrix as Transformation lab shows the fold directly.
The common choices
| Function | Formula | Notes |
|---|---|---|
| Sigmoid | output in (0, 1); slope at most 0.25; now used mainly for probabilities and gates | |
| tanh | output in (−1, 1); zero-centred; saturates for large $ | |
| ReLU | slope exactly 1 for — gradients pass unshrunk; became standard around 2010–2012 | |
| GELU | smooth ReLU-like curve; used in BERT and GPT-2 |
Many recent LLMs use gated variants such as SwiGLU in their feed-forward layers Established (Chapter 11). The key property to remember isn't the exact curve — it's the slope, because during backpropagation the gradient is multiplied by it at every layer.
Try it
Vanishing and Exploding Gradients
Send a gradient backwards through up to 40 layers and see it shrink to nothing or blow up — and how ReLU and good initialization fix it.
What to remember
- Without nonlinearity, depth adds nothing: W₂(W₁x) = (W₂W₁)x.
- Sigmoid: (0, 1), slope ≤ 0.25 — saturates.
- tanh: (−1, 1), zero-centred, still saturates.
- ReLU: max(0, z) — cheap, slope 1 for z > 0; the default for years.
- GELU / SwiGLU: smooth variants used in modern Transformers.
Key papers
Gaussian Error Linear Units (GELUs)
Dan Hendrycks, Kevin Gimpel · 2016
The GELU activation, a smooth relative of ReLU used in BERT, GPT-2 and many later Transformers.