11.1 Experimental Design for ML

Experimental design is crucial for machine learning engineers to accurately assess model performance and make informed decisions. It involves controlled experiments, A/B testing, and factorial designs to systematically evaluate variables and their impact on ML models.

Real-world constraints like computational resources and data privacy shape experimental design in ML. Addressing biases, determining appropriate sample sizes, and using randomization techniques are key to creating robust experiments that yield reliable insights for improving machine learning systems.

Controlled Experiments for ML Models

Experimental Design Fundamentals

• Controlled experiments in machine learning systematically manipulate variables to assess their impact on model performance while holding other factors constant
• A/B testing compares two versions of a model or system to determine which performs better on a specific metric (click-through rates, conversion rates)
• Factorial designs examine multiple factors and their interactions simultaneously providing a comprehensive understanding of model behavior (feature importance, hyperparameter tuning)
• Cross-validation techniques estimate model performance and generalizability in experimental settings
  • K-fold cross-validation divides data into k subsets, training on k-1 folds and testing on the remaining fold
  • Repeated k-fold cross-validation performs multiple rounds of k-fold cross-validation to obtain more robust estimates
• Clearly defined metrics for success objectively evaluate model performance
  • Accuracy measures overall correctness of predictions
  • F1 score balances precision and recall for imbalanced datasets
  • Business-specific KPIs (customer lifetime value, churn rate)

Real-World Considerations

• Deployment constraints impact experimental design in ML
  • Computational resources limit model complexity and training time
  • Latency requirements influence model architecture and inference speed
  • Data privacy concerns restrict data usage and sharing (federated learning)
• Time series experiments require specialized designs to account for temporal dependencies
  • Backtesting evaluates model performance on historical data
  • Forward-chaining cross-validation simulates real-world forecasting scenarios
  • Rolling window analysis assesses model stability over time

Bias Mitigation in Experiments

Common Biases and Confounding Factors

• Selection bias occurs when the sample is not representative of the population, leading to skewed results and limited generalizability (oversampling high-income individuals)
• Confounding factors correlate with both independent and dependent variables, potentially leading to incorrect conclusions about causal relationships (age influencing both income and credit score)
• Simpson's Paradox reveals trends in subgroups that disappear or reverse when groups are combined, highlighting the importance of considering all relevant variables (college admissions rates by gender and department)
• Survivorship bias in ML experiments occurs when the dataset only includes successful cases, leading to overly optimistic model performance estimates (analyzing only companies that survived an economic downturn)
• Data leakage inadvertently influences the training process with information from the test set, resulting in overly optimistic performance estimates (using future data to predict past events)

Mitigation Strategies

• Careful data collection and preprocessing reduce biases
  • Stratified sampling ensures representation of subgroups
  • Data augmentation techniques balance class distributions
  • Feature scaling and normalization mitigate the impact of outliers
• Causal inference techniques address confounding factors
  • Propensity score matching pairs similar observations across treatment groups
  • Instrumental variables isolate causal effects in the presence of confounders
  • Difference-in-differences analysis compares changes over time between treated and control groups
• Regularization techniques reduce overfitting and mitigate spurious correlations
  • L1 regularization (Lasso) encourages sparsity in feature selection
  • L2 regularization (Ridge) prevents large coefficient values
  • Elastic Net combines L1 and L2 regularization for balanced feature selection

Sample Size and Power for ML

Statistical Power and Effect Size

• Statistical power represents the probability of correctly rejecting the null hypothesis when it is false, influenced by sample size, effect size, and significance level
• Minimum detectable effect (MDE) determines the smallest effect size reliably detected given the experimental setup
  • Smaller MDEs require larger sample sizes to maintain statistical power
  • MDEs vary based on the specific metric and business context (1% improvement in click-through rate)
• Power analysis techniques determine required sample size for desired statistical power
  • A priori power analysis calculates sample size before conducting the experiment
  • Post-hoc power analysis assesses the achieved power after the experiment
  • Sensitivity analysis explores the impact of different effect sizes on required sample size

ML-Specific Considerations

• Curse of dimensionality necessitates larger sample sizes to maintain statistical power in high-dimensional feature spaces
  • Rule of thumb: 10 samples per feature for linear models, more for complex models
  • Dimensionality reduction techniques (PCA, t-SNE) can help mitigate this issue
• Bootstrapping and resampling techniques estimate confidence intervals and assess model stability
  • Bootstrap sampling creates multiple datasets by sampling with replacement
  • Jackknife resampling assesses the impact of individual observations on model performance
• Learning curves determine the relationship between sample size and model performance
  • Plotting training and validation errors against sample size reveals underfitting or overfitting
  • Helps inform decisions about data collection and experimental design
• Bayesian experimental design optimizes sample sizes and experimental parameters
  • Expected information gain quantifies the value of additional data points
  • Thompson sampling balances exploration and exploitation in adaptive experiments

Randomization, Blocking, and Stratification

Randomization and Blocking

• Randomization controls for unknown confounding factors and ensures validity of statistical inferences
  • Simple random sampling assigns treatments completely at random
  • Permuted block randomization ensures balance within blocks of a specified size
• Blocking controls for known sources of variation, grouping experimental units into homogeneous blocks
  • Reduces within-group variability and increases statistical power
  • Example: blocking by geographic region in a multi-site ML experiment
• Latin square designs efficiently allocate treatments across different conditions
  • Useful when controlling for multiple factors (model architecture, dataset, hardware)
  • Reduces the number of required experimental runs while maintaining balance

Advanced Experimental Designs

• Orthogonality in experimental design ensures independent estimation of factor effects
  • Reduces multicollinearity and improves interpretability of results
  • Orthogonal arrays optimize the allocation of factor levels across experimental runs
• Fractional factorial designs efficiently explore multiple factors when full factorial designs are impractical
  • Reduce the number of experimental runs while still capturing main effects and some interactions
  • Resolution III designs estimate main effects, Resolution IV designs estimate main effects and some two-factor interactions
• Adaptive experimental designs dynamically allocate resources to promising treatments
  • Multi-armed bandits balance exploration of new options with exploitation of known good options
  • Thompson sampling uses Bayesian updating to guide treatment allocation based on observed outcomes
  • Useful for online ML experiments with continuous model updates and large parameter spaces