22.3 Variance reduction methods

Monte Carlo simulations estimate quantities by averaging random samples. Variance reduction techniques aim to improve precision with fewer samples, leading to faster convergence and better computational efficiency. These methods are crucial for tackling complex problems in engineering probability.

Stratified sampling divides the sample space into subsets, ensuring coverage and reducing variance. Importance sampling emphasizes critical regions, while control variates use correlated variables to reduce variance. Choosing the right technique depends on the problem's characteristics and available information.

Variance Reduction Techniques

Variance reduction in Monte Carlo

Top images from around the web for Variance reduction in Monte Carlo

• 

  Variance reduction - Simulace.info View original

  Is this image relevant?

• 

  Frontiers | Monte Carlo Simulations for the Analysis of Non-linear Parameter Confidence ... View original

  Is this image relevant?

• 

  Variance reduction - Simulace.info View original

  Is this image relevant?

• 

  Frontiers | Monte Carlo Simulations for the Analysis of Non-linear Parameter Confidence ... View original

  Is this image relevant?

1 of 2

Top images from around the web for Variance reduction in Monte Carlo

• 

  Variance reduction - Simulace.info View original

  Is this image relevant?

• 

  Frontiers | Monte Carlo Simulations for the Analysis of Non-linear Parameter Confidence ... View original

  Is this image relevant?

• 

  Variance reduction - Simulace.info View original

  Is this image relevant?

• 

  Frontiers | Monte Carlo Simulations for the Analysis of Non-linear Parameter Confidence ... View original

  Is this image relevant?

1 of 2

• Monte Carlo simulations estimate quantities by averaging results from multiple random samples
  • Accuracy improves with more samples but computational cost increases (Central Limit Theorem)
• Variance reduction techniques aim to reduce the variance of the estimator
  • Smaller variance leads to more precise estimates with fewer samples (Confidence Intervals)
  • Enables faster convergence and improved computational efficiency

Stratified sampling for efficiency

• Stratified sampling divides the sample space into non-overlapping subsets called strata
  • Each stratum is sampled independently (Latin Hypercube Sampling)
  • Samples are allocated proportionally to the strata based on their importance or size
• Stratification reduces variance by ensuring coverage of the entire sample space
  • Captures the variability within each stratum (Homogeneous Groups)
  • Prevents over-sampling or under-sampling of specific regions
• Optimal allocation of samples to strata minimizes the overall variance
  • Neyman allocation: $n_i \propto \frac{N_i \sigma_i}{\sqrt{c_i}}$, where $n_i$ is the sample size for stratum $i$, $N_i$ is the stratum size, $\sigma_i$ is the stratum standard deviation, and $c_i$ is the sampling cost for stratum $i$

Importance sampling for critical regions

• Importance sampling modifies the sampling distribution to emphasize important regions
  • Draws samples from a proposal distribution that favors the regions of interest (Tails of Distribution)
  • Assigns weights to the samples to compensate for the change in distribution (Likelihood Ratio)
• The optimal proposal distribution is proportional to the product of the original distribution and the quantity of interest
  • $q^*(x) \propto |f(x)p(x)|$, where $q^*(x)$ is the optimal proposal distribution, $f(x)$ is the quantity of interest, and $p(x)$ is the original distribution
• Importance sampling is effective for estimating rare event probabilities or tail expectations
  • Increases the occurrence of rare events in the simulation (Failure Analysis)
  • Reduces the variance of the estimator by focusing on the critical regions

Control variates for variance reduction

• Control variates introduce a correlated variable with known expectation to the estimator
  • The control variate is subtracted from the original estimator and its known expectation is added back
  • $\hat{\theta}_{CV} = \frac{1}{n} \sum_{i=1}^n (Y_i - \beta(X_i - \mathbb{E}[X]))$, where $\hat{\theta}_{CV}$ is the control variate estimator, $Y_i$ is the original estimator, $X_i$ is the control variate, and $\beta$ is the optimal coefficient
• The optimal coefficient $\beta$ minimizes the variance of the control variate estimator
  • $\beta^* = \frac{\text{Cov}(Y, X)}{\text{Var}(X)}$
• Effective control variates have a high correlation with the original estimator
  • The higher the correlation, the greater the variance reduction (Pearson Correlation Coefficient)
• Common control variates include known expectations, such as the mean of a related random variable or the value of a similar system (Antithetic Variates)

Comparison of variance reduction techniques

• Consider the characteristics of the problem and the available information
  • Stratified sampling is suitable when the sample space can be divided into distinct strata (Geographical Regions)
  • Importance sampling is effective for rare event simulation or when certain regions are more critical (Reliability Analysis)
  • Control variates are useful when there are known quantities correlated with the estimator (Derivative Pricing)
• Assess the trade-offs between implementation complexity and potential variance reduction
  • Some techniques may require additional computational overhead or problem-specific knowledge
• Conduct pilot simulations to compare the performance of different variance reduction techniques
  1. Estimate the variance reduction achieved by each technique
  2. Select the technique that provides the best balance between variance reduction and computational efficiency
• Combine multiple variance reduction techniques when appropriate
  • Different techniques can be applied simultaneously to further reduce variance (Stratified Importance Sampling)
  • For example, stratified sampling can be combined with importance sampling or control variates