Concept · Chapter 3: Machine Learning
Support Vector Machines
A support vector machine chooses the separating boundary with the widest possible margin to the nearest points, and with the kernel trick it can draw curved boundaries by implicitly working in a higher-dimensional feature space.
The problem
Many boundaries separate the training data; which one is most likely to generalize — and how can a linear method separate non-linear data?
The solution
Maximize the margin (distance to the closest training points, the 'support vectors'), allow some violations with a penalty, and replace dot products with a kernel function to get non-linear boundaries.
The consequence
SVMs were the leading classifiers of the late 1990s–2000s; the kernel idea — similarity as an implicit dot product in feature space — is a useful lens on representations generally.
You should understand first
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Vectors
- Features, Labels and Tasks
- Loss Functions
- Derivatives and Gradients
- Gradient Descent
- Linear Regression
- Probability and Distributions
- Entropy
- Softmax
- Cross-Entropy Loss
- Logistic Regression
- Dot Product
- Support Vector Machines
The idea
Among all lines that separate two classes, pick the one that stays as far as possible from both — the widest "street". Only the points on the edges of the street, the support vectors, matter; move any other point and the boundary doesn't change. The soft-margin version (Cortes & Vapnik, 1995) tolerates some points inside the street, trading margin for errors.
The kernel trick
To separate data that isn't linearly separable, map inputs into a richer feature space (e.g. add , , …). The SVM's math only ever needs dot products between examples — so replace with a kernel that equals a dot product in the richer space, without ever computing that space explicitly.
What to remember
- Maximize the margin between the classes.
- Only the closest points (support vectors) determine the boundary.
- Kernel trick: compute dot products in a high-dimensional feature space without constructing it.
- Dominant before deep learning; still useful on small, clean datasets.
Key papers
Support-vector networks
Corinna Cortes, Vladimir Vapnik · 1995 · Machine Learning
Introduced the soft-margin support vector machine, the dominant classifier of the late 1990s and 2000s.