Bayes' theorem

Bayes' theorem reverses a conditional probability. We have $P(A \mid E_i)$ , the probability of $A$ given each hypothesis $E_i$ . We want $P(E_i \mid A)$ , the probability of hypothesis $E_i$ given that we have observed $A$ . Bayes is the formula that does this reversal cleanly.

It is the cornerstone of statistical inference, medical diagnostics, machine learning, and any form of rational belief-updating.

Statement

Let $E_1, E_2, \ldots, E_n$ be a partition of the sample space, and let $A$ be an event with $P(A) > 0$ . Then for each $i$ : $P(E_i \mid A) = \frac{P(A \mid E_i)\,P(E_i)}{P(A)} = \frac{P(A \mid E_i)\,P(E_i)}{\sum_{j=1}^n P(A \mid E_j)\,P(E_j)}.$

The denominator is just $P(A)$ from the law of total probability.

Derivation

Start from the multiplication rule: $P(A \cap E_i) = P(E_i \mid A)\,P(A) = P(A \mid E_i)\,P(E_i)$ . Solve for $P(E_i \mid A)$ : $P(E_i \mid A) = \frac{P(A \mid E_i)\,P(E_i)}{P(A)}.$ Replace $P(A)$ by the total-probability expansion to get the full Bayes formula.

Terminology

In Bayesian language:

$P(E_i)$ is the prior probability of $E_i$ (before observing $A$ ).
$P(A \mid E_i)$ is the likelihood of $A$ under $E_i$ .
$P(E_i \mid A)$ is the posterior probability of $E_i$ (after observing $A$ ).
$P(A)$ is the marginal or normalising constant.

When to use it

Bayes is the right tool whenever you have a "cause and effect" structure: the $E_i$ 's are possible causes (hypotheses), $A$ is the observed effect, and you want to know which cause is most likely given the effect.

Classic settings:

Medical diagnosis: $E_i$ is having (or not having) the disease; $A$ is the test result.
Quality control: $E_i$ is which machine produced the item; $A$ is whether it is defective.
Spam filtering: $E_i$ is "spam" or "not spam"; $A$ is the appearance of certain words.

Worked examples

Example 1. A factory makes light bulbs on three machines $M_1, M_2, M_3$ , producing $25\%, 35\%, 40\%$ , with defect rates $5\%, 4\%, 2\%$ . A random bulb is found defective. What is the probability it came from $M_1$ ?

Priors: $P(M_1) = 0.25$ , $P(M_2) = 0.35$ , $P(M_3) = 0.40$ . Likelihoods: $P(D|M_1) = 0.05$ , $P(D|M_2) = 0.04$ , $P(D|M_3) = 0.02$ .

Marginal: $P(D) = 0.25 \cdot 0.05 + 0.35 \cdot 0.04 + 0.40 \cdot 0.02 = 0.0125 + 0.014 + 0.008 = 0.0345$ .

$P(M_1|D) = \dfrac{0.25 \cdot 0.05}{0.0345} = \dfrac{0.0125}{0.0345} \approx 0.362$ .

Example 2. The classic medical test. Disease affects 1% of population. Test: $95\%$ true positive rate, $90\%$ true negative rate (so $10\%$ false positive). A person tests positive. What is the probability they have the disease?

Priors: $P(D) = 0.01$ , $P(D^c) = 0.99$ . Likelihoods: $P(+|D) = 0.95$ , $P(+|D^c) = 0.10$ .

Marginal: $P(+) = 0.01 \cdot 0.95 + 0.99 \cdot 0.10 = 0.0095 + 0.099 = 0.1085$ .

$P(D|+) = \dfrac{0.0095}{0.1085} \approx 0.0876$ , i.e. about $8.8\%$ .

The surprising conclusion: despite a "95% accurate" test, the conditional probability of disease given a positive result is only ~9%. This is because the disease is rare and false positives dominate.

Example 3. Two urns: U1 has 3W, 2B; U2 has 1W, 4B. An urn is picked at random and a ball drawn , turns out to be white. Probability it came from U1?

Priors: $P(U_1) = P(U_2) = 1/2$ . Likelihoods: $P(W|U_1) = 3/5$ , $P(W|U_2) = 1/5$ .

Marginal: $P(W) = (1/2)(3/5) + (1/2)(1/5) = 2/5$ .

$P(U_1|W) = \dfrac{(1/2)(3/5)}{2/5} = \dfrac{3/10}{2/5} = \dfrac{3/10}{4/10} = 3/4$ .

Example 4. A coin is selected from a jar: $P(\text{fair}) = 0.6$ , $P(\text{two-headed}) = 0.4$ . The selected coin is flipped and shows heads. Probability it is the two-headed coin?

$P(H | \text{fair}) = 1/2$ , $P(H | \text{two-headed}) = 1$ .

$P(H) = 0.6 \cdot 0.5 + 0.4 \cdot 1 = 0.7$ .

$P(\text{two-headed} | H) = \dfrac{0.4}{0.7} = 4/7 \approx 0.571$ .

Example 5. Three drivers (call them D1, D2, D3) split a fleet's mileage: 40%, 35%, 25%. Their accident rates are 0.02, 0.04, 0.08 per year. An accident occurs; probability it was driver D3?

Priors: $0.40, 0.35, 0.25$ . Likelihoods: $0.02, 0.04, 0.08$ .

Marginal: $0.40 \cdot 0.02 + 0.35 \cdot 0.04 + 0.25 \cdot 0.08 = 0.008 + 0.014 + 0.020 = 0.042$ .

$P(D_3 | \text{accident}) = 0.020 / 0.042 \approx 0.476$ .

Example 6. Spam filter: $30\%$ of emails are spam. The word "free" appears in $40\%$ of spam, $5\%$ of non-spam. An email contains "free" , probability it is spam?

$P(\text{spam} | \text{free}) = \dfrac{0.30 \cdot 0.40}{0.30 \cdot 0.40 + 0.70 \cdot 0.05} = \dfrac{0.12}{0.155} \approx 0.774$ .

Try it yourself

Box A: 4R/6B; box B: 7R/3B. Choose box uniformly; draw red. $P(\text{from A})$ .
Three machines: 30%, 30%, 40%; defective rates 1%, 2%, 3%. A defective item , $P(\text{from machine 3})$ .
Two coins, fair and biased ( $P(H) = 0.7$ ). Choose at random and flip , heads. $P(\text{biased})$ .
A test: $P(+ | D) = 0.99$ , $P(+ | D^c) = 0.05$ . Prevalence $P(D) = 0.001$ . Find $P(D | +)$ .
A student answers correctly. Probability she knew = $0.7$ . If guesses, $P(\text{correct}) = 1/4$ . Find $P(\text{knew} | \text{correct})$ .
Bag A: 3W/5B. Bag B: 6W/2B. Choose with probability $1/3, 2/3$ ; draw white. $P(\text{from A})$ .
Two factories produce in ratio 3:2. Defect rates 4% and 2%. A defective , $P(\text{from factory 1})$ .
Three urns, all equally likely: U1 has 2W/3B, U2 has 4W/1B, U3 has 3W/2B. Draw a white. $P(\text{from U2})$ .
A communication channel: probability of bit being 1 is 0.6. Probability of correct transmission is 0.95. Receive 1 , $P(\text{sent 1})$ .
Two batches of fruits. Batch 1: 95% good, contributes 60% of total. Batch 2: 80% good. Pick a good fruit , $P(\text{from batch 1})$ .
Three coins: $P(H) = 0.5, 0.7, 0.9$ with prior probabilities $0.4, 0.4, 0.2$ . Flip and get H. Posterior?
Bag with 3 red, 2 white, then add another set of 5 red, 5 white. Choose first bag w.p. $0.4$ . Draw red , $P(\text{from second bag})$ .
A taxi cab: 85% Yellow, 15% Green. Witness identifies Green with $80\%$ accuracy when actually Green, and Green falsely with $20\%$ . Identify Green , actual probability it was Green?
Insurance: 70% of policies are low risk, accident rate $1\%$ . 30% are high risk, accident rate $10\%$ . Given an accident, $P(\text{policy was high risk})$ ?

Pitfalls and tricks

Identify priors, likelihoods, and the marginal. Lay out the data clearly before plugging in.
Bayes formula has a clear structure: posterior $\propto$ prior $\times$ likelihood. Don't lose sight of the structure.
Mediasource fallacy: never confuse $P(A|B)$ with $P(B|A)$ . The whole point of Bayes is that they are usually different.
Rare diseases trap. Even a highly accurate test for a rare condition has surprisingly low PPV (positive predictive value).
Always normalise. The denominator $P(A)$ is the law of total probability , sum of prior × likelihood over all $E_i$ .