Concept · Chapter 3: Machine Learning
Supervised, Unsupervised and Self-Supervised Learning
Learning paradigms differ in where the training signal comes from: human-provided labels (supervised), structure in the data alone (unsupervised), or labels manufactured from the data itself (self-supervised).
The problem
Labelled examples are expensive, yet most of the world's data — text, images, logs — comes without labels.
The solution
Match the method to the signal available: learn input→label mappings when labels exist, find structure (clusters, compressions) when they don't, and invent a prediction task from the data itself when there is plenty of raw data.
The consequence
Self-supervised learning — predict the missing or next piece of the data — unlocked training on internet-scale unlabelled text and images, and it is how every LLM is pretrained.
You should understand first
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
Four ways to get a learning signal
| Paradigm | What you have | Example | Typical task |
|---|---|---|---|
| Supervised | inputs with correct answers | emails labelled spam / not spam | classification, regression |
| Unsupervised | inputs only | customer purchase histories | clustering, dimensionality reduction |
| Semi-supervised | a few labels, many unlabelled inputs | 1,000 labelled scans, 1M unlabelled | classification with scarce labels |
| Self-supervised | inputs only — but you make labels from them | "the cat sat on the ___" → "mat" | pretraining representations |
Why self-supervised learning changed everything
Supervised learning needs a person to provide each answer, which caps the dataset at what you can afford to label. Self-supervised learning removes the cap: hide part of the data and ask the model to predict it. Every sentence on the internet becomes thousands of free training examples — predict each next word from the words before it.
This is exactly how GPT-style language models are pretrained, and masked-word prediction is how BERT was Established. The "labels" are just the text itself, shifted by one position — which is why the chain rule of probability and cross-entropy from Chapter 2 are the whole training recipe.
What to remember
- Supervised: learn from (input, correct output) pairs.
- Unsupervised: no labels — find clusters, low-dimensional structure, or density.
- Semi-supervised: a few labels plus lots of unlabelled data.
- Self-supervised: create labels from the data itself (e.g. predict the next word).
- Reinforcement learning (Chapter 5) learns from rewards instead of labels.