1.2 Conditional probability

Conditional probability is a crucial concept in probability theory, allowing us to update our understanding of events based on new information. It measures the likelihood of an event occurring given that another event has already taken place, providing a powerful tool for analyzing complex scenarios.

This topic explores the definition and calculation of conditional probabilities, independence, the multiplication rule, and Bayes' theorem. We'll also delve into conditional distributions, expectations, and real-world applications in fields like medicine, machine learning, and Bayesian inference.

Definition of conditional probability

• Conditional probability measures the probability of an event occurring given that another event has already occurred
• Allows for updating probabilities based on new information or known conditions
• Fundamental concept in probability theory with wide-ranging applications

Probability of event A given event B

• Denoted as $P(A|B)$, read as "the probability of A given B"
• Represents the probability of event A occurring, assuming that event B has already taken place
• Focuses on the subset of outcomes where event B occurs and calculates the probability of A within that subset

Notation for conditional probability

• $P(A|B) = \frac{P(A \cap B)}{P(B)}$, where $P(B) > 0$
• $P(A \cap B)$ represents the probability of both events A and B occurring (intersection)
• $P(B)$ is the probability of event B occurring, acting as a normalizing factor

Conditional probability formula

• The conditional probability formula is derived from the definition of conditional probability
• Relates the conditional probability to the joint probability and the probability of the conditioning event

Deriving the formula

• Start with the definition: $P(A|B) = \frac{P(A \cap B)}{P(B)}$
• Multiply both sides by $P(B)$: $P(A|B) \cdot P(B) = P(A \cap B)$
• Rearrange to obtain the multiplication rule: $P(A \cap B) = P(A|B) \cdot P(B)$

Calculating conditional probabilities

• Identify the events of interest (A and B)
• Calculate the probability of the intersection $P(A \cap B)$ (events occurring together)
• Calculate the probability of the conditioning event $P(B)$
• Divide $P(A \cap B)$ by $P(B)$ to obtain the conditional probability $P(A|B)$

Independence vs dependence

• Independence and dependence describe the relationship between events
• Determines whether the occurrence of one event affects the probability of another event

Definition of independence

• Events A and B are independent if $P(A|B) = P(A)$ and $P(B|A) = P(B)$
• The occurrence of one event does not change the probability of the other event
• If events are not independent, they are considered dependent

Checking for independence

• Calculate the conditional probabilities $P(A|B)$ and $P(B|A)$
• Compare them to the individual probabilities $P(A)$ and $P(B)$
• If the conditional probabilities equal the individual probabilities, the events are independent
• If the conditional probabilities differ from the individual probabilities, the events are dependent

Multiplication rule

• The multiplication rule relates the joint probability of two events to their conditional probabilities
• Allows for calculating the probability of the intersection of events

Deriving the multiplication rule

• Start with the conditional probability formula: $P(A|B) = \frac{P(A \cap B)}{P(B)}$
• Multiply both sides by $P(B)$: $P(A|B) \cdot P(B) = P(A \cap B)$
• The multiplication rule states that $P(A \cap B) = P(A|B) \cdot P(B)$

Applying the multiplication rule

• Identify the events of interest (A and B)
• Calculate the conditional probability $P(A|B)$ or $P(B|A)$
• Calculate the probability of the conditioning event $P(B)$ or $P(A)$
• Multiply the conditional probability by the probability of the conditioning event to obtain the joint probability $P(A \cap B)$

Law of total probability

• The law of total probability calculates the probability of an event by partitioning the sample space
• Breaks down the probability calculation into mutually exclusive and exhaustive cases

Partitioning the sample space

• Identify a set of mutually exclusive and exhaustive events $B_1, B_2, ..., B_n$ that partition the sample space
• Mutually exclusive: the events cannot occur simultaneously ($B_i \cap B_j = \emptyset$ for $i \neq j$)
• Exhaustive: the union of all events covers the entire sample space ($\bigcup_{i=1}^n B_i = S$)

Applying the law of total probability

• Let $A$ be the event of interest and $B_1, B_2, ..., B_n$ be the partitioning events
• The law of total probability states that $P(A) = \sum_{i=1}^n P(A|B_i) \cdot P(B_i)$
• Calculate the conditional probabilities $P(A|B_i)$ for each partitioning event
• Calculate the probabilities $P(B_i)$ for each partitioning event
• Multiply each conditional probability by its corresponding partitioning event probability and sum the results

Bayes' theorem

• Bayes' theorem relates conditional probabilities in reverse order
• Allows for updating probabilities based on new information or evidence

Deriving Bayes' theorem

• Start with the multiplication rule: $P(A \cap B) = P(A|B) \cdot P(B) = P(B|A) \cdot P(A)$
• Divide both sides by $P(B)$: $\frac{P(A \cap B)}{P(B)} = \frac{P(B|A) \cdot P(A)}{P(B)}$
• Substitute the left side with the definition of conditional probability: $P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)}$

Applying Bayes' theorem

• Identify the events of interest (A and B)
• Calculate the conditional probability $P(B|A)$ (likelihood)
• Calculate the prior probabilities $P(A)$ and $P(B)$
• Substitute the values into Bayes' theorem to obtain the posterior probability $P(A|B)$

Updating probabilities with new information

• Bayes' theorem allows for updating probabilities as new information becomes available
• The prior probability represents the initial belief or knowledge about an event
• The likelihood represents the probability of observing the new information given the event
• The posterior probability is the updated probability after incorporating the new information

Conditional probability distributions

• Conditional probability distributions describe the probability distribution of a random variable given the value of another random variable
• Can be discrete or continuous depending on the nature of the random variables involved

Discrete conditional distributions

• Involve discrete random variables (variables that take on a countable number of values)
• The conditional probability mass function (PMF) is defined as $P(X=x|Y=y) = \frac{P(X=x, Y=y)}{P(Y=y)}$
• The conditional PMF gives the probability of X taking on a specific value given the value of Y

Continuous conditional distributions

• Involve continuous random variables (variables that take on an uncountable number of values)
• The conditional probability density function (PDF) is defined as $f_{X|Y}(x|y) = \frac{f_{X,Y}(x,y)}{f_Y(y)}$
• The conditional PDF describes the relative likelihood of X taking on a specific value given the value of Y

Conditional expectation

• Conditional expectation is the expected value of a random variable given the value of another random variable
• Provides a measure of the average value of a random variable under certain conditions

Definition of conditional expectation

• For discrete random variables X and Y, the conditional expectation of X given Y=y is $E[X|Y=y] = \sum_{x} x \cdot P(X=x|Y=y)$
• For continuous random variables X and Y, the conditional expectation of X given Y=y is $E[X|Y=y] = \int_{-\infty}^{\infty} x \cdot f_{X|Y}(x|y) dx$
• Conditional expectation is a function of the conditioning variable Y

Properties of conditional expectation

• Linearity: $E[aX + bY|Z] = aE[X|Z] + bE[Y|Z]$ for constants a and b
• Iterated expectation: $E[X] = E[E[X|Y]]$ (the expectation of the conditional expectation is the unconditional expectation)
• If X and Y are independent, then $E[X|Y] = E[X]$ (conditioning on Y does not change the expectation of X)

Applications of conditional probability

• Conditional probability has numerous applications in various fields
• Allows for making informed decisions and predictions based on available information

Medical testing and diagnosis

• Conditional probability is used to interpret medical test results and update disease probabilities
• Sensitivity (true positive rate) and specificity (true negative rate) are conditional probabilities
• Positive predictive value (probability of having the disease given a positive test result) is calculated using Bayes' theorem

Machine learning and classification

• Conditional probability is fundamental in machine learning algorithms for classification tasks
• Naive Bayes classifier uses conditional probabilities to predict the class of an instance based on its features
• Decision trees and random forests rely on conditional probabilities to split nodes and make predictions

Bayesian inference and parameter estimation

• Bayesian inference uses conditional probability to update beliefs about parameters based on observed data
• Prior probabilities represent initial beliefs about parameter values
• Likelihood function describes the probability of observing the data given the parameter values
• Posterior probabilities are updated using Bayes' theorem to incorporate the observed data