Naive Bayes

Should knowUnderstand10 minDifficulty

Naive Bayes classifies by applying Bayes' theorem with the simplifying ('naive') assumption that features are independent given the class — crude, but fast and often surprisingly effective for text.

The problem

Estimating how all features jointly depend on the class needs far more data than we have.

The solution

Assume each feature depends only on the class, so the probability of an example factorizes into a product of simple per-feature probabilities, each easy to estimate by counting.

The consequence

It powered early practical spam filters and remains a strong baseline for text classification — and it's the learner you saw beat hand-written rules in Chapter 1.

You should understand first

The equation

P(\text{spam} \mid w_1, \dots, w_n) \;\propto\; P(\text{spam}) \prod_{i=1}^{n} P(w_i \mid \text{spam})

Each $P(w_i \mid \text{spam})$ is just "how often does this word appear in spam?" — estimated by counting, with a small smoothing constant so unseen words don't zero everything out. Compare the score for spam and not-spam (in logs, to avoid tiny numbers) and pick the larger.

The independence assumption is plainly false — "free" and "prize" co-occur — yet the classifier often ranks classes correctly anyway. You watched this model learn a spam filter from 40 examples in Write the Rules.

What to remember

P(class | features) ∝ P(class) · Π P(feature | class).
'Naive': assumes features are independent given the class.
Train by counting; predict by summing log-probabilities.
Fast, needs little data, a strong text baseline.