Skip to content
Road to Intelligence

Concept · Chapter 3: Machine Learning

Decision Trees and Random Forests

Should knowUnderstand15 minDifficulty

A decision tree predicts by asking a sequence of yes/no questions about the features; a random forest averages many randomized trees to get a much more accurate and stable model.

The problem

Linear models can't capture interactions and thresholds ('high income AND young') without hand-built features; single trees capture them but overfit and are unstable.

The solution

Grow a tree by repeatedly choosing the split that best separates the labels (e.g. largest entropy reduction); then combine many trees — averaging (random forests) or adding them sequentially to fix errors (gradient boosting).

The consequence

Tree ensembles remain among the strongest and most widely used methods for tabular data, where deep learning has not dominated the way it has for images and text.

You should understand first

  1. The Turing Test
  2. Symbolic AI
  3. Logic and Rules
  4. Expert Systems
  5. Knowledge Representation
  6. The Knowledge-Acquisition Bottleneck
  7. From Rules to Learning
  8. Supervised, Unsupervised and Self-Supervised Learning
  9. Vectors
  10. Features, Labels and Tasks
  11. Probability and Distributions
  12. Entropy
  13. Decision Trees and Random Forests

A learned rule system

A decision tree is what the expert systems of Chapter 1 wanted to be — a set of if–then rules — except the rules are learned from data: at each node, pick the feature and threshold that best separate the labels, measured by how much the split reduces entropy (or the similar Gini impurity).

Ensembles

A deep tree memorizes its training data. Two ensemble ideas fix this:

  • Random forests (Breiman, 2001): train hundreds of trees, each on a bootstrap sample of the rows and considering a random subset of features at each split; average their votes. Individual trees overfit in different directions and the errors cancel.
  • Gradient boosting: add small trees one at a time, each fitted to the errors of the ensemble so far.

For tabular data of the kind data engineers handle every day, gradient-boosted trees are often competitive with or better than deep networks Interpretation, and they're fast to train and easy to deploy.

What to remember

  • A tree = nested if–then splits learned from data.
  • Splits chosen to reduce impurity (entropy / Gini) the most.
  • Single trees overfit; ensembles fix that.
  • Random forest: many trees on bootstrap samples with random feature subsets, averaged.
  • Gradient boosting: add trees one at a time, each correcting the last.

Key papers

Important

Random Forests

Leo Breiman · 2001 · Machine Learning

Random forests — many decision trees trained on random subsets of data and features, then averaged — remain one of the strongest methods for tabular data.

~50 min readdoi:10.1023/A:1010933404324✓ verified 2026-09-26

Watch