Concept · Chapter 4: Neural Networks
Multilayer Perceptron (MLP)
A multilayer perceptron stacks layers of neurons — input, one or more hidden layers, output — so that each layer builds new features out of the previous layer's outputs.
The problem
A single neuron draws one straight boundary; real problems need curved, complicated ones.
The solution
Feed inputs into a hidden layer of neurons, feed their outputs into further layers, and train all the weights together.
The consequence
With even one hidden layer of enough units, networks can approximate essentially any continuous function; in practice, deeper networks learn good features more efficiently.
You should understand first
- Vectors
- Dot Product
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Features, Labels and Tasks
- Loss Functions
- Derivatives and Gradients
- Gradient Descent
- Linear Regression
- Probability and Distributions
- Entropy
- Softmax
- Cross-Entropy Loss
- Logistic Regression
- The Perceptron
- Activation Functions
- The Artificial Neuron
- Matrix Multiplication
- Multilayer Perceptron (MLP)
How a hidden layer solves XOR
XOR needs two boundaries, not one. A hidden layer provides them: one hidden unit can learn "x₁ and x₂ both positive", another "both negative", and the output unit combines them. Each hidden unit draws its own line; the output draws a shape out of those lines. In the Neural Network Lab, watch the decision map bend as training progresses.
Universal approximation — and what it doesn't say
A network with a single hidden layer and enough units can approximate any continuous function on a bounded domain to any desired accuracy (Cybenko 1989; Hornik et al. 1989) Established. That's a statement about what exists, not about what training will find, or how many units you need — which can be astronomically many. Depth tends to be far more efficient: composing layers lets networks reuse intermediate features Interpretation (representation learning).
What to remember
- Layers: input → hidden (one or more) → output.
- Each layer: h = φ(Wx + b).
- Hidden units learn intermediate features; the output layer combines them.
- Universal approximation: one wide hidden layer can approximate any continuous function.
- 'Deep' = many hidden layers.
Key papers
Multilayer feedforward networks are universal approximators
Kurt Hornik, Maxwell Stinchcombe, Halbert White · 1989 · Neural Networks
Proved that a network with a single hidden layer can approximate essentially any continuous function, given enough units — the 'universal approximation' result.
How to read it: A theoretical result: it says a good network exists, not that training will find it or how big it must be.
Approximation by superpositions of a sigmoidal function
G. Cybenko · 1989 · Mathematics of Control, Signals, and Systems
An independent universal-approximation proof for networks with sigmoid hidden units.
Watch
3Blue1Brown
But what is a neural network? | Deep learning chapter 1
The clearest visual introduction to what a neural network actually computes, layer by layer.