13.2 Law of Large Numbers and Central Limit Theorem Revisited

Statistical inference relies on understanding how sample statistics relate to population parameters. The laws of large numbers explain how sample means converge to true population means as sample sizes increase. This convergence forms the basis for consistent estimators, which become more accurate with larger samples.

The Central Limit Theorem is a cornerstone of statistical inference, showing that the distribution of sample means approaches a normal distribution for large samples. This powerful result enables various statistical techniques, including confidence intervals and hypothesis tests, which are widely used in research and decision-making.

Foundations of Statistical Inference

Laws of large numbers

• Weak Law of Large Numbers (WLLN)
  • Convergence in probability measures how sample mean approaches population mean
  • For i.i.d. random variables with finite mean μ, sample mean converges in probability to μ
  • $P(|\bar{X}_n - \mu| > \epsilon) \to 0$ as $n \to \infty$, for any $\epsilon > 0$
  • Proof uses Chebyshev's inequality demonstrating variance of sample mean approaches zero as n increases
  • Applied in estimating population parameters (average height of adults)
• Strong Law of Large Numbers (SLLN)
  • Almost sure convergence provides stronger guarantee than WLLN
  • For i.i.d. random variables with finite mean μ, sample mean converges almost surely to μ
  • $P(\lim_{n \to \infty} \bar{X}_n = \mu) = 1$
  • Proof employs Borel-Cantelli lemma showing sum of probabilities of large deviations is finite
  • Used in proving consistency of machine learning algorithms (gradient descent)
• WLLN vs SLLN
  • SLLN offers stronger convergence guarantee than WLLN
  • WLLN sufficient for many practical applications while SLLN provides theoretical foundation
  • Both crucial in understanding behavior of sample statistics as sample size grows

Consistency of estimators

• Consistency concept
  • Measures how estimator approaches true parameter value as sample size increases
  • Weak consistency based on convergence in probability
  • Strong consistency based on almost sure convergence
• WLLN for weak consistency
  • Demonstrates estimator converges in probability to true parameter
  • Sample mean as consistent estimator of population mean (average test scores)
• SLLN for strong consistency
  • Shows estimator converges almost surely to true parameter
  • Sample variance as consistent estimator of population variance (stock price volatility)
• Consistency importance
  • Ensures estimates become more reliable with larger samples
  • Underpins asymptotic properties of estimators (efficiency, normality)
  • Critical in model selection and evaluation (choosing between competing statistical models)

Central Limit Theorem and Its Applications

Central limit theorem

• CLT statement

  • For i.i.d. random variables with finite mean μ and variance σ², standardized sample mean converges to standard normal distribution
  • $\frac{\sqrt{n}(\bar{X}_n - \mu)}{\sigma} \to N(0,1)$ as $n \to \infty$
  • Holds regardless of underlying distribution shape (uniform, exponential)

• CLT assumptions

  • Random variables must be independent
  • Variables must follow identical distribution
  • Finite variance required

• Proof outline

  1. Utilize characteristic functions
  2. Apply Lévy's continuity theorem
  3. Demonstrate convergence of characteristic function of standardized sample mean to standard normal

• CLT implications

  • Enables approximation of complex sampling distributions
  • Forms basis for numerous statistical inference procedures (hypothesis testing, confidence intervals)
  • Explains prevalence of normal distribution in natural phenomena (height, blood pressure)

Applications of central limit theorem

• Approximating sample mean distribution
  • For large n, $\bar{X}_n \approx N(\mu, \frac{\sigma^2}{n})$
  • Generally, n > 30 considered "large" for many distributions
• Confidence interval construction
  • General form: $\bar{X}n \pm z{\alpha/2} \frac{\sigma}{\sqrt{n}}$
  • 95% confidence level implies 95% of intervals contain true parameter
  • Use t-distribution when population standard deviation unknown (small samples)
• CLT in hypothesis testing
  • Enables Z-test and t-test procedures
  • Facilitates power analysis and sample size determination
  • Applied in A/B testing (comparing conversion rates)
• CLT limitations
  • Approximation accuracy improves with larger sample sizes
  • Remains somewhat robust to minor violations of assumptions
  • Caution needed for highly skewed distributions or presence of outliers