🃏Engineering Probability Unit 12 – Limit Theorems & Convergence in Probability

Key Concepts and Definitions

• Convergence in probability measures how close a sequence of random variables gets to a specific value as the sample size increases
• Almost sure convergence occurs when a sequence of random variables converges to a value with probability 1
• Convergence in distribution happens when the cumulative distribution functions (CDFs) of a sequence of random variables converge to the CDF of a limiting distribution
• Characteristic functions uniquely determine the distribution of a random variable and are useful for proving convergence results
• Stochastic convergence refers to the convergence of sequences of random variables or distributions
• The sample mean is the arithmetic average of a set of observations, given by $\bar{X} = \frac{1}{n} \sum_{i=1}^n X_i$
• The sample variance measures the variability of a set of observations, calculated as $s^2 = \frac{1}{n-1} \sum_{i=1}^n (X_i - \bar{X})^2$

Types of Convergence

• Convergence in probability ($X_n \xrightarrow{p} X$) means that for any $\epsilon > 0$, $\lim_{n \to \infty} P(|X_n - X| > \epsilon) = 0$
  • Intuitively, the probability of the difference between $X_n$ and $X$ being greater than any small value $\epsilon$ approaches 0 as $n$ increases
• Almost sure convergence ($X_n \xrightarrow{a.s.} X$) is stronger than convergence in probability and implies that $P(\lim_{n \to \infty} X_n = X) = 1$
  • This means that the sequence of random variables converges to $X$ with probability 1
• Convergence in distribution ($X_n \xrightarrow{d} X$) occurs when the CDFs of $X_n$ converge to the CDF of $X$ at all continuity points of the limiting CDF
• Convergence in mean square ($X_n \xrightarrow{m.s.} X$) implies that $\lim_{n \to \infty} E[(X_n - X)^2] = 0$
  • This type of convergence is stronger than convergence in probability but weaker than almost sure convergence
• Convergence in $r$-th mean ($X_n \xrightarrow{L^r} X$) generalizes mean square convergence and requires that $\lim_{n \to \infty} E[|X_n - X|^r] = 0$ for some $r > 0$

Law of Large Numbers

• The Law of Large Numbers (LLN) states that the sample mean of a sequence of independent and identically distributed (i.i.d.) random variables converges to the population mean as the sample size increases
• The Weak Law of Large Numbers (WLLN) asserts that the sample mean converges in probability to the population mean
  • Formally, if $X_1, X_2, \ldots$ are i.i.d. with $E[X_i] = \mu$, then $\bar{X}_n \xrightarrow{p} \mu$ as $n \to \infty$
• The Strong Law of Large Numbers (SLLN) is a stronger result, stating that the sample mean converges almost surely to the population mean
  • Under the same conditions as the WLLN, the SLLN states that $\bar{X}_n \xrightarrow{a.s.} \mu$ as $n \to \infty$
• The LLN holds under more general conditions, such as for sequences of independent random variables with finite means and variances (Khinchin's Theorem)
• The LLN is essential for justifying the use of sample means as estimates of population means in various statistical applications

Central Limit Theorem

• The Central Limit Theorem (CLT) describes the asymptotic distribution of the standardized sample mean for a sequence of i.i.d. random variables with finite mean and variance
• If $X_1, X_2, \ldots$ are i.i.d. with $E[X_i] = \mu$ and $Var(X_i) = \sigma^2 < \infty$, then $\frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} \xrightarrow{d} N(0, 1)$ as $n \to \infty$
  • This means that the standardized sample mean converges in distribution to a standard normal random variable
• The CLT holds under more general conditions, such as for sequences of independent random variables with finite means and variances (Lindeberg-Feller Theorem)
• The CLT is the foundation for many statistical inference procedures, such as hypothesis testing and confidence interval construction
• The Berry-Esseen Theorem quantifies the rate of convergence in the CLT, providing bounds on the difference between the CDF of the standardized sample mean and the standard normal CDF

Weak Convergence and Characteristic Functions

• Weak convergence is another term for convergence in distribution, focusing on the convergence of probability measures or distributions rather than random variables
• A sequence of probability measures $\mu_n$ converges weakly to a probability measure $\mu$ if $\int f d\mu_n \to \int f d\mu$ for all bounded, continuous functions $f$
• The Portmanteau Theorem provides several equivalent characterizations of weak convergence, such as the convergence of CDFs at continuity points of the limiting CDF
• Characteristic functions are a powerful tool for proving weak convergence results, as they uniquely determine the distribution of a random variable
• The characteristic function of a random variable $X$ is defined as $\varphi_X(t) = E[e^{itX}]$, where $i$ is the imaginary unit
• Levy's Continuity Theorem states that a sequence of random variables $X_n$ converges in distribution to $X$ if and only if their characteristic functions $\varphi_{X_n}(t)$ converge to $\varphi_X(t)$ for all $t$

Applications in Engineering

• The LLN is used in Monte Carlo simulations to estimate expected values or probabilities by averaging a large number of independent replications
• The CLT is the basis for statistical process control (SPC) techniques, such as control charts for monitoring the mean and variability of a process
• In reliability engineering, the CLT is used to approximate the distribution of the sum of lifetimes of components in a system, enabling the calculation of system reliability
• Convergence results are essential for justifying the use of asymptotic approximations in various engineering applications, such as queueing theory and signal processing
• Weak convergence is used to analyze the performance of estimators and to establish the consistency and asymptotic normality of parameter estimates in engineering models
• Characteristic functions are employed in the analysis of linear time-invariant (LTI) systems, as the output characteristic function is the product of the input characteristic function and the system's transfer function

Common Pitfalls and Misconceptions

• Assuming that convergence in probability implies almost sure convergence or vice versa, as these are distinct concepts with different strengths
• Misinterpreting the CLT as a statement about the distribution of individual random variables, rather than the asymptotic distribution of the standardized sample mean
• Applying the LLN or CLT to dependent sequences of random variables without verifying the appropriate conditions (e.g., mixing conditions or martingale differences)
• Neglecting to check the continuity of the limiting CDF when using the Portmanteau Theorem to prove weak convergence
• Confusing the roles of characteristic functions and moment-generating functions, as they have different domains and uniqueness properties
• Overrelying on asymptotic results without considering the finite-sample performance or the rate of convergence in practical applications

Practice Problems and Examples

• Prove that if $X_n \xrightarrow{p} X$ and $Y_n \xrightarrow{p} c$ (a constant), then $X_n Y_n \xrightarrow{p} cX$
• Show that the sample variance converges in probability to the population variance for a sequence of i.i.d. random variables with finite fourth moments
• Verify that the characteristic function of the sum of independent random variables is the product of their individual characteristic functions
• Use the CLT to approximate the probability that the average of 50 i.i.d. exponential random variables with mean 2 exceeds 2.2
• Prove that the sequence of random variables $X_n \sim \text{Poisson}(n)$ converges in distribution to a standard normal random variable as $n \to \infty$
• Determine the limiting distribution of the maximum of $n$ i.i.d. uniform random variables on $[0, 1]$ as $n \to \infty$ using the CDF convergence criterion
• Establish the consistency of the sample median as an estimator of the population median for a sequence of i.i.d. continuous random variables
• Apply the Lindeberg-Feller CLT to a sequence of independent but non-identically distributed random variables satisfying the Lindeberg condition