Law of total probability

Often we cannot compute $P(A)$ directly, but we know $P(A \mid B_i)$ for various $B_i$'s and the $P(B_i)$'s. The law of total probability combines these into $P(A)$.

Statement

Let $E_1, E_2, \ldots, E_n$ be mutually exclusive and exhaustive events , i.e. they partition the sample space, with $P(E_i) > 0$ for each $i$. Then for any event $A$: $P(A) = \sum_{i=1}^n P(A \mid E_i)\,P(E_i).$

Why it works

Since the $E_i$'s partition the sample space, $A$ can be decomposed: $A = (A \cap E_1) \cup (A \cap E_2) \cup \cdots \cup (A \cap E_n)$, with the pieces disjoint. So $P(A) = \sum_i P(A \cap E_i) = \sum_i P(A \mid E_i) P(E_i)$, using the multiplication rule.

When to use it

The law applies whenever you can identify a natural partition of "how $A$ might come about." Each $E_i$ is one "way." Typical settings:

• Several urns/boxes: pick a box, then draw from it.
• Different defective rates: items from machine $i$ with defect rate $p_i$, fraction $w_i$ from each.
• Medical tests: disease present or not (the partition), test positive or not (the event $A$).
• Communication: signal sent (partition: which message), signal received (event $A$).

Worked examples

Example 1. Box 1 has 3 red, 7 blue. Box 2 has 5 red, 5 blue. A box is chosen at random, then a ball is drawn from it. Find $P(\text{red})$.

Partition: $E_1 =$ box 1, $E_2 =$ box 2. $P(E_1) = P(E_2) = 1/2$.

$P(\text{red} \mid E_1) = 3/10$. $P(\text{red} \mid E_2) = 5/10 = 1/2$.

$P(\text{red}) = (3/10)(1/2) + (1/2)(1/2) = 3/20 + 5/20 = 8/20 = 2/5$.

Example 2. A factory has three machines $M_1, M_2, M_3$ producing $30\%, 50\%, 20\%$ of total output, with defect rates $1\%, 2\%, 3\%$. A random item is picked. Find $P(\text{defective})$.

$P = 0.3 \cdot 0.01 + 0.5 \cdot 0.02 + 0.2 \cdot 0.03 = 0.003 + 0.010 + 0.006 = 0.019$.

So $1.9\%$ of items are defective.

Example 3. A disease affects $1\%$ of a population. A test detects it correctly $95\%$ of the time when present and falsely positives $2\%$ of the time when absent. Find $P(\text{test positive})$.

Partition: $D =$ has disease, $D^c =$ doesn't. $P(D) = 0.01$, $P(D^c) = 0.99$.

$P(\text{pos} \mid D) = 0.95$. $P(\text{pos} \mid D^c) = 0.02$.

$P(\text{pos}) = 0.95 \cdot 0.01 + 0.02 \cdot 0.99 = 0.0095 + 0.0198 = 0.0293$.

Example 4. A coin is selected at random from a jar of 5 coins: 2 are fair, 3 are biased with $P(H) = 0.8$. The selected coin is flipped. Find $P(H)$.

$P(\text{fair}) = 2/5$, $P(\text{biased}) = 3/5$. $P(H \mid \text{fair}) = 1/2$, $P(H \mid \text{biased}) = 0.8 = 4/5$.

$P(H) = (2/5)(1/2) + (3/5)(4/5) = 1/5 + 12/25 = 5/25 + 12/25 = 17/25$.

Example 5. A bag has 4 red, 6 blue balls. Two balls are drawn one after another without replacement. Find the probability that the second ball is red.

Partition by colour of first ball: $E_1 =$ 1st red, $E_2 =$ 1st blue. $P(E_1) = 4/10 = 2/5$. $P(E_2) = 6/10 = 3/5$.

$P(\text{2nd red} \mid E_1) = 3/9 = 1/3$. $P(\text{2nd red} \mid E_2) = 4/9$.

$P(\text{2nd red}) = (2/5)(1/3) + (3/5)(4/9) = 2/15 + 12/45 = 6/45 + 12/45 = 18/45 = 2/5$.

(Same as $P(\text{1st red})$ , a deep symmetry of sampling without replacement.)

Example 6. A student answers an MCQ. With probability $0.7$ she knows the answer; otherwise she guesses uniformly among 4 options. Find $P(\text{correct})$.

$P(\text{know}) = 0.7$. $P(\text{correct} \mid \text{know}) = 1$. $P(\text{correct} \mid \text{guess}) = 1/4$.

$P(\text{correct}) = 0.7 \cdot 1 + 0.3 \cdot 0.25 = 0.7 + 0.075 = 0.775$.

Try it yourself

1. Two urns: U1 has 3R/2B, U2 has 4R/4B. Choose urn uniformly, then a ball. Find $P(R)$.
2. Three urns: U1 has 1R/4B, U2 has 3R/2B, U3 has 2R/3B. Pick urn uniformly, then a ball. $P(R)$.
3. Coin probability of heads is $1/2, 1/3$, or $1/4$ with equal probabilities. Flip the chosen coin. Find $P(H)$.
4. A factory makes light bulbs at three machines (30%, 45%, 25%) with defect rates (3%, 1%, 2%). $P(\text{defective})$.
5. A car insurance company classifies drivers as low, medium, high risk: $60\%, 30\%, 10\%$. Probability of accident in a year: $0.01, 0.05, 0.20$. Probability a random driver has an accident.
6. A box has 5 white, 6 black balls. Two drawn without replacement. Find $P(\text{2nd is white})$.
7. Two coins: one fair, one with $P(H) = 0.6$. Choose at random and flip twice. Probability both flips are heads.
8. Toss a die. If even, draw from urn A (3 red/2 blue); if odd, urn B (1 red/4 blue). $P(\text{red})$.
9. A student answers a question. Probability she knows: $0.6$. Probability of correct if guesses: $1/3$. Find $P(\text{correct})$.
10. 5 cards, 3 face cards 2 number cards. Choose without replacement two cards. $P(\text{2nd is face})$.
11. A bag contains 5 W and 4 B. A ball is drawn and not replaced; then a 2nd is drawn. $P(\text{2nd is white})$.
12. A test for a rare condition affecting $0.5\%$ of population: sensitivity $98\%$, specificity $95\%$. $P(\text{test positive})$.
13. Three boxes: contains coins with respective probabilities of being authentic $0.9, 0.85, 0.7$. Choose box uniformly; draw a coin. Find $P(\text{authentic})$.
14. Two coins: a fair one and a coin with $P(H) = p$. Pick at random and flip. Probability of heads is observed to be $0.6$. Find $p$.

Pitfalls and tricks

• Verify the partition. The $E_i$'s must be mutually exclusive AND exhaustive.
• Don't forget any $E_i$. Missing a partition piece gives the wrong total.
• Each conditional $P(A \mid E_i)$ must be computed in the world where $E_i$ has occurred.
• The formula reads: prior × likelihood, summed. $P(E_i)$ is the "prior", $P(A \mid E_i)$ is the "likelihood."
• Tree diagrams are excellent for two-level partitions.