Machine Learning

The idea

What does it mean to learn?

Chapter 1 ended with a turn: instead of writing rules, collect examples and let the machine find the pattern. This chapter makes that precise. Machine learning means choosing a family of functions with adjustable parameters, then using data to pick the parameters that make the function fit — in a way that keeps working on examples it hasn't seen.

Everything depends on where the training signal comes from. With labelled examples, you learn a mapping from inputs to answers (supervised). Without labels, you look for structure (unsupervised). And with only raw data, you can invent labels from the data itself — hide a word and predict it — which is self-supervised learning, the way every LLM is pretrained.

ConceptSupervised, Unsupervised and Self-Supervised LearningKnow wellMust know

Learning paradigms differ in where the training signal comes from: human-provided labels (supervised), structure in the data alone (unsupervised), or labels manufactured from the data itself (self-supervised).

Open the concept page →

For supervised learning the data is a table you'll recognize from data engineering: rows are examples, columns are features, one column is the label. If the label is a number it's regression; if it's a category, classification.

ConceptFeatures, Labels and TasksKnow wellMust know

A supervised learning problem is a table: each row is an example described by features, and the label is what the model must predict — a category (classification) or a number (regression).

Open the concept page →

How it works

Your first models: lines and curves

Every model in this site — up to GPT — follows the same three-part recipe you met in Chapter 2: a model with parameters, a loss that scores it, and an optimizer that lowers the loss. The simplest instance is linear regression: predict a number as a weighted sum of the features, and choose the weights that minimize the squared error.

ConceptLinear RegressionImplementMust know

Linear regression predicts a number as a weighted sum of the features plus a constant, choosing the weights that minimize the average squared error on the training data.

Open the concept page →

Swap the output for a probability — pass the weighted sum through a sigmoid — and swap squared error for cross-entropy, and you have logistic regression, the basic classifier. Train one below and watch the boundary settle; then try the XOR data and watch it fail.

ConceptLogistic RegressionImplementMust know

Logistic regression is a linear classifier: it computes a weighted sum of the features and squashes it through a sigmoid to get a probability, trained by minimizing cross-entropy.

Open the concept page →

The central problem

The central problem: generalization

Here is the most important idea in machine learning. A model can always fit its training data better by becoming more flexible — but fitting the training data isn't the goal. Predicting new data is. Move the slider and watch the two errors part ways.

At degree 11, the curve passes exactly through all twelve training points — zero training error — and is wildly wrong everywhere else. That is overfitting: memorizing noise instead of learning the pattern. At degree 0 the model is too simple to capture anything — underfitting. The job is to find the middle, and the only honest way to find it is to measure on data the model never trained on.

ConceptGeneralization, Overfitting and UnderfittingKnow wellMust know

The goal of learning is generalization — good performance on data the model has never seen — and a model that memorizes its training data (overfits) or is too simple to capture the pattern (underfits) fails at it.

Open the concept page →

When you can't simplify the model, you can penalize complexity instead. Turn on regularization in the lab at degree 11 and the test error drops back near the best simple model.

ConceptRegularizationKnow wellMust know

Regularization is anything that discourages a model from fitting the training data too closely — most commonly a penalty on large weights — so that it generalizes better.

Open the concept page →

Failure modes

How ML goes wrong

Two failures account for a large share of models that look great in evaluation and disappoint in production — and both are, at heart, data pipeline problems.

Leakage: the training data contains information that won't exist at prediction time — a feature computed from the future, a preprocessing step fitted on the test set, the same customer in both splits. The model learns the leak, not the task.

ConceptData LeakageKnow wellMust know

Data leakage is when training or evaluation data contains information that won't be available when the model is actually used — making results look far better than they really are.

Open the concept page →

Distribution shift: the world the model is deployed into differs from the one it was trained on. A test-set score is a promise only about data like the test set.

ConceptDistribution ShiftUnderstandMust know

Distribution shift is when the data a model meets in use differs from the data it was trained on, so its measured performance no longer applies.

Open the concept page →

Measurement

Measuring a classifier

Accuracy is the obvious metric and often the wrong one. When one class is rare — fraud, disease, spam — a model that always says "no" scores high accuracy while being useless. The fix is to count which kind of mistakes the model makes, and to choose where on the precision–recall trade-off you want to be.

ConceptEvaluation Metrics for ClassifiersKnow wellMust know

Accuracy alone can mislead; precision (how many flagged items were right), recall (how many true items were caught) and the confusion matrix show what kind of mistakes a classifier makes.

Open the concept page →

The toolbox

The classical toolbox

You don't need to memorize algorithms — but you should recognize the main families and what each is good for. Each is a different answer to the same question: what shape of function should the model be allowed to learn?

ConceptDecision Trees and Random ForestsUnderstandShould know

A decision tree predicts by asking a sequence of yes/no questions about the features; a random forest averages many randomized trees to get a much more accurate and stable model.

Open the concept page →

ConceptSupport Vector MachinesUnderstandShould know

A support vector machine chooses the separating boundary with the widest possible margin to the nearest points, and with the kernel trick it can draw curved boundaries by implicitly working in a higher-dimensional feature space.

Open the concept page →

ConceptNaive BayesUnderstandShould know

Naive Bayes classifies by applying Bayes' theorem with the simplifying ('naive') assumption that features are independent given the class — crude, but fast and often surprisingly effective for text.

Open the concept page →

Conceptk-Means ClusteringKnow wellShould know

k-means groups unlabelled points into k clusters by alternating two steps: assign each point to its nearest centre, then move each centre to the mean of its points.

Open the concept page →

ConceptPrincipal Component Analysis (PCA)UnderstandShould know

PCA finds the few directions along which data varies the most, so high-dimensional data can be summarized, compressed or plotted with little loss.

Open the concept page →

The limit

From hand-crafted features to learned ones

Every model in this chapter learns only the last step: from features to answer. Someone still has to design the features — and for images, audio and text that was years of expert work per domain, the knowledge-acquisition bottleneck from Chapter 1, one level down. The XOR example in the logistic regression lab is this problem in miniature: a linear model fails until someone invents the right feature.

ConceptHand-Crafted Features vs Learned FeaturesKnow wellMust know

Classical ML learns only the final mapping from features to labels — people design the features — while deep learning learns the features too, directly from raw data.

Open the concept page →

The next step is to learn the features too. That is what neural networks do.

Why it matters

Why it matters

Everything later is this chapter at scale. An LLM is a model with parameters, trained by gradient descent on a loss (cross-entropy) over data, and evaluated on held-out benchmarks. Overfitting, leakage (called contamination for LLMs), distribution shift and misleading metrics are exactly the failure modes you'll meet in Chapter 16.

For your day job, much of this is directly usable. On tabular data — the kind data engineers own — well-featured linear models and tree ensembles remain strong, fast and explainable. Knowing when that's enough is as valuable as knowing how a Transformer works.