6  Bayes’ rule

Author

Karl Gregory

Bayes’ rule is a useful re-expression of the definition of conditional probability. Before stating Bayes’ rule, note that for any events \(A\) and \(B\), we have \[ P(A) = P(A \cap B) + P(A \cap B^c). \] Draw a Venn Diagram to convince yourself! We can also write \[ P(A) = P(A | B)P(B) + P(A |B^c)P(B^c) \] using the definition of conditional probability. This allows us to write

\[ P(B|A) = \frac{P(A \cap B)}{P(A)} = \frac{P(A | B)P(B)}{ P(A | B)P(B) + P(A |B^c)P(B^c) }, \tag{6.1}\]

which is an instance of Bayes’ rule. More generally, Bayes’ rule is the following:

Proposition 6.1 (Bayes’ rule) For an event \(A\) in a sample space \(S\) and a set of events \(B_1,\dots,B_K\) forming a partition of \(S\), we have \[ P(B_k|A) = \frac{P(A|B_k)P(B_k)}{P(A|B_1)P(B_1) + \dots + P(A|B_K)P(B_K)} \] for each \(k = 1,\dots,K\).

By the definition of conditional probability we have \[ P(B_k |A) = \frac{P(B_k \cap A)}{P(A)}. \] From here the result follows by writing \(P(B_k \cap A) = P(A|B_k)P(B_k)\) and \[ \begin{align*} P(A) &= P(A \cap B_1) + \dots + P(A \cap B_K) \\ &= P(A|B_1)P(B_1) + \dots + P(A|B_K)P(B_k) \end{align*} \] by the law of total probability and by using again the definition of conditional probability to write intersection probabilities as conditional times marginal probabilities.

We get the instance of Bayes’ rule in Equation 6.1 by noting that \(B\) and \(B^c\) form a partition of the sample space.

Following are a few examples in

Example 6.1 (Positive given a positive test) The National Cancer Institute estimates that \(3.65\%\) of women in their sixties develop breast cancer. Suppose that if a woman has breast cancer, a mammogram will detect it \(85\%\) of the time and if a woman does not have breast cancer, the mammogram still results in a positive test for breast cancer \(5\%\) of the time.

  1. Suppose we want to find the probability that a woman has breast cancer given that her mammogram is positive. If \(M\) is the event that a woman’s mammogram is positive for breast cancer and \(C\) is the event that she truly has breast cancer, we wish to find th conditional probablity \(P(C| M)\). We know from Bayes’ rule that \[ \begin{align*} P(C | M ) & = \frac{P( M| C)P(C)}{ P( M| C)P(C) + P(M| C^c)P(C^c) } \\ & = \frac{ 0.85 \times 0.0365}{ 0.85 \times 0.0365 + 0.05 \times (1-0.0365)} \\ & = 0.319 \end{align*} \]

  2. Suppose we wish to find the probability that a randomly selected woman tests positive for breast cancer. We have computed this probability already; it is \[ \begin{align*} P(M) &= P(M\cap C) + P(M\cap C^c)\\ &=P(M|C)P(C) + P(M|C^c)P(C^c)\\ &= 0.85 \times 0.0365 + 0.05 \times (1-0.0365) \\ &= 0.0792. \end{align*} \]

Exercise 6.1 (Scam or no scam) Suppose \(1.0\%\) of all Craigslist advertisements for housing sublets are posted by scammers. In addition, suppose \(90\%\) of scammers request that payment be sent to a P.O. box instead of a residential address, and only \(20\%\) of non-scammers, that is legitimate advertisers, request that payment be sent to a P.O. box instead of a residential address.

  1. What is the probability that a randomly selected Craigslist advertiser for a housing sublet will request that payment be sent to a P.O. box? 1

  2. If you make an inquiry into a housing sublet post and the advertiser requests that payment be sent to a P.O. box, what is the probability that it is a scam? 2

Example 6.2 (Game with bags) Consider the following game involving three bags, each containing four bills as follows: \[ \begin{align*} \text{First bag: }& \text{$3$ five dollar bills and $1$ twenty dollar bill} \\ \text{Second bag: }& \text{$2$ five dollar bills and $2$ twenty dollar bills} \\ \text{Third bag: }& \text{$1$ five dollar bill and $3$ twenty dollar bills} \end{align*} \] The player rolls a die and draws a bill from the first bag if the roll is \(1\),\(2\), or \(3\), from the second bag if the roll is \(3\) or \(4\), and from the third bag if the roll is \(6\).

  1. Suppose we want to know the probability of winning \(\$20\) from the game. Letting \(B_1\), \(B_2\), and \(B_3\) represent the event of drawing from the first, second, and third bag, respectively, and letting \(W\) be the event of winning \(\$20\) from the game, we can use the law of total probability to obtain \[ \begin{align*} P(W) &= P(W \cap B_1) + P(W \cap B_2) + P(W \cap B_3)\\ &= P(W | B_1)P(B_1) + P(W | B_2)P(B_2) + P(W | B_3)P(B_3)\\ &= (1/4)(3/6) + (2/4)(2/6) + (3/4)(1/6)\\ &= 5/12. \end{align*} \]
    This is an example in which it can be useful to draw tree diagram like the one below:

flowchart LR
  %% root
  Root[" "]

  %% bags
  Bag1["Bag 1"]
  Bag2["Bag 2"]
  Bag3["Bag 3"]
  
  %% root -> bags (edge labels)
  Root -->|3/6| Bag1
  Root -->|2/6| Bag2
  Root -->|1/6| Bag3
  
  %% Bag 3 leaves
  Bag3_20["$20"]
  Bag3_5["$5"]
  Bag3 -->|1/4| Bag3_5
  Bag3 -->|3/4| Bag3_20

  %% Bag 2 leaves
  Bag2_20["$20"]
  Bag2_5["$5"]
  Bag2 -->|2/4| Bag2_5
  Bag2 -->|2/4| Bag2_20

  %% Bag 1 leaves
  Bag1_20["$20"]
  Bag1_5["$5"]
  Bag1 -->|3/4| Bag1_5
  Bag1 -->|1/4| Bag1_20

This diagram nicely summarizes all the information we are given. To find the probability of an outcome, we trace the tree diagram starting from the beginning until we reach that outcome, and multiplying the probabilities we pass along the way. For example, \(P(W \cap B_1) = 3/6 \times 1/4\).

  1. Given that a player got twenty dollars, what is the probability that the player drew from the first bag? We can answer this question using Bayes’ rule: \[ \begin{align*} P(B_1| W) &= \frac{P(W | B_1)P(B_1)}{P(W | B_1)P(B_1) + P(W | B_2)P(B_2) + P(W | B_3)P(B_3)}\\ & = \frac{(1/4)(3/6)}{5/12} \\ &=3/10. \end{align*} \]

  1. Letting \(O\) be the event that the advertizer requests payment to a P.O. box and \(S\) the event that the advertizement is a scam (ignoring the fact that we usually use \(S\) to represent the sample space), we have \[ \begin{align*} P(O) &= P(O \cap S) + P(O \cap S^c) \\ &=P(O | S)P(S) + P(O | S^c)P(S^c) \\ & = 0.90 \times 0.01 + 0.20 \times (1 - 0.01)\\ & = 0.207. \end{align*} \]↩︎

  2. We have \[ \begin{align*} P(S|O) &= \frac{P(S\cap O)}{P(O)} \\ &= \frac{P(O|S)P(S)}{P(O)} \\ &= \frac{0.90 \times 0.01}{0.207} \\ &= 0.0435, \end{align*} \] where we have used Bayes’ formula: \[ P(S|O) =\frac{P(O|S)P(S)}{P(O | S)P(S) + P(O | S^c)P(S^c)}. \]↩︎