Skip to content
Road to Intelligence

Concept · Chapter 3: Machine Learning

Evaluation Metrics for Classifiers

Must knowKnow well25 minDifficulty

Accuracy alone can mislead; precision (how many flagged items were right), recall (how many true items were caught) and the confusion matrix show what kind of mistakes a classifier makes.

The problem

On imbalanced problems — fraud, disease, spam — a model that predicts 'no' for everything can score high accuracy while being useless.

The solution

Count the four outcomes (true/false positives/negatives) and summarize them with the metric that matches the cost of each mistake; sweep the decision threshold to see the full trade-off (ROC curve, AUC).

The consequence

Choosing the metric is choosing what the model is for — and a reported number is only meaningful alongside the base rate and the threshold.

You should understand first

  1. The Turing Test
  2. Symbolic AI
  3. Logic and Rules
  4. Expert Systems
  5. Knowledge Representation
  6. The Knowledge-Acquisition Bottleneck
  7. From Rules to Learning
  8. Supervised, Unsupervised and Self-Supervised Learning
  9. Vectors
  10. Features, Labels and Tasks
  11. Loss Functions
  12. Derivatives and Gradients
  13. Gradient Descent
  14. Linear Regression
  15. Probability and Distributions
  16. Entropy
  17. Softmax
  18. Cross-Entropy Loss
  19. Logistic Regression
  20. Evaluation Metrics for Classifiers

The four outcomes

actually positiveactually negative
predicted positivetrue positive (TP)false positive (FP)
predicted negativefalse negative (FN)true negative (TN)
precision=TPTP+FPrecall=TPTP+FNF1=2⋅precision⋅recallprecision+recall\text{precision} = \frac{TP}{TP + FP} \qquad \text{recall} = \frac{TP}{TP + FN} \qquad F_1 = \frac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}

Try it

Try it · toy model

Precision, Recall and the Threshold

A fraud detector on imbalanced data: move the threshold and watch precision, recall, the confusion matrix and the ROC curve respond.

Know well8 min

Notice two things. With only 10% fraud, "never flag anything" scores 90% accuracy — a weak model can score lower than doing nothing. And moving the threshold never improves precision and recall together; you choose where on the trade-off to sit, based on what a missed fraud costs versus a false alarm. This is the same base-rate logic as Bayes' theorem.

ROC and AUC

The ROC curve plots recall (true-positive rate) against false-positive rate for every threshold at once; the area under it (AUC) is threshold-free: 0.5 is guessing, 1.0 is perfect separation. On heavily imbalanced problems, precision–recall curves are often more informative, because ROC can look good while precision is poor.

What to remember

  • Confusion matrix: TP, FP, FN, TN.
  • Precision = TP / (TP + FP): of what I flagged, how much was right?
  • Recall = TP / (TP + FN): of what was really there, how much did I catch?
  • Raising the threshold usually raises precision and lowers recall.
  • Always compare against the trivial baseline (e.g. 'always predict the majority class').

Watch