🃏Engineering Probability Unit 18 – Hypothesis Testing & Statistical Inference

Key Concepts and Definitions

• Hypothesis testing assesses the validity of a claim or hypothesis about a population parameter based on sample data
• Null hypothesis ($H_0$) represents the default or status quo position, assuming no significant difference or effect
• Alternative hypothesis ($H_a$ or $H_1$) represents the claim or statement being tested, suggesting a significant difference or effect
• Type I error (false positive) occurs when rejecting a true null hypothesis, denoted by $\alpha$ (significance level)
• Type II error (false negative) occurs when failing to reject a false null hypothesis, denoted by $\beta$
  • The power of a test ($1-\beta$) measures the probability of correctly rejecting a false null hypothesis
• Statistical significance indicates the likelihood of observing the sample results or more extreme outcomes, assuming the null hypothesis is true
• p-value represents the probability of obtaining the observed sample results or more extreme outcomes, given that the null hypothesis is true
  • A small p-value (typically < 0.05) suggests strong evidence against the null hypothesis

Foundations of Probability Theory

• Probability theory provides the mathematical framework for quantifying uncertainty and making inferences about population parameters
• Random variables represent numerical outcomes of a random experiment or process
  • Discrete random variables have countable outcomes (number of defective items)
  • Continuous random variables have uncountable outcomes within an interval (time to failure)
• Probability distributions describe the likelihood of different outcomes for a random variable
  • Common discrete distributions include binomial, Poisson, and geometric
  • Common continuous distributions include normal, exponential, and uniform
• Expected value (mean) and variance characterize the central tendency and dispersion of a random variable, respectively
• Central Limit Theorem states that the sampling distribution of the sample mean approximates a normal distribution as the sample size increases, regardless of the population distribution
• Confidence intervals estimate the range of plausible values for a population parameter based on sample data and a specified confidence level (90%, 95%)

Types of Hypothesis Tests

• One-sample tests compare a sample statistic to a hypothesized population parameter (mean, proportion)
  • One-sample t-test for population mean with unknown variance
  • One-sample z-test for population mean with known variance or large sample size
  • One-sample proportion test for population proportion
• Two-sample tests compare statistics from two independent samples (means, proportions)
  • Two-sample t-test for comparing means of two populations with unknown variances
  • Two-sample z-test for comparing means of two populations with known variances or large sample sizes
  • Two-sample proportion test for comparing proportions of two populations
• Paired tests compare dependent samples or repeated measures (before-after, matched pairs)
  • Paired t-test for comparing means of two related samples
• Analysis of Variance (ANOVA) tests compare means across three or more groups or factors
  • One-way ANOVA for one factor with multiple levels
  • Two-way ANOVA for two factors with multiple levels and their interaction
• Chi-square tests assess the association between categorical variables
  • Goodness-of-fit test compares observed frequencies to expected frequencies
  • Independence test examines the relationship between two categorical variables

Steps in Hypothesis Testing

1. State the null and alternative hypotheses based on the research question or claim
2. Choose the appropriate test statistic and distribution based on the type of data and assumptions
3. Specify the significance level ($\alpha$) and determine the critical value(s) or rejection region
4. Collect sample data and calculate the test statistic value
5. Compare the test statistic value to the critical value(s) or compute the p-value
6. Make a decision to reject or fail to reject the null hypothesis based on the comparison or p-value
7. Interpret the results in the context of the research question and draw conclusions
8. Consider the practical significance and limitations of the findings

Statistical Inference Techniques

• Point estimation provides a single value estimate of a population parameter based on sample data
  • Method of moments estimates parameters by equating sample moments to population moments
  • Maximum likelihood estimation finds parameter values that maximize the likelihood of observing the sample data
• Interval estimation constructs a range of plausible values for a population parameter with a specified confidence level
  • Confidence intervals for means, proportions, and variances
  • Prediction intervals for future individual observations
• Bayesian inference updates prior beliefs about parameters using sample data to obtain posterior distributions
  • Prior distribution represents initial beliefs about the parameter before observing data
  • Likelihood function quantifies the probability of observing the data given different parameter values
  • Posterior distribution combines prior beliefs and sample evidence to update knowledge about the parameter
• Resampling methods generate empirical sampling distributions by repeatedly drawing samples from the original data
  • Bootstrap resamples with replacement to estimate standard errors and construct confidence intervals
  • Permutation tests assess the significance of a test statistic by permuting the original data

Interpreting Test Results

• Reject the null hypothesis if the test statistic falls in the rejection region or the p-value is less than the significance level
  • Concluding significance implies that the sample evidence is strong enough to support the alternative hypothesis
  • Caution against overstating the findings or making causal claims without proper experimental design
• Fail to reject the null hypothesis if the test statistic does not fall in the rejection region or the p-value is greater than the significance level
  • Lack of significance does not prove the null hypothesis, but rather indicates insufficient evidence to support the alternative hypothesis
  • Consider the power of the test and the possibility of Type II error when interpreting non-significant results
• Report the test statistic value, p-value, and confidence interval (if applicable) along with the decision and interpretation
• Assess the practical significance of the findings in addition to statistical significance
  • Effect sizes measure the magnitude of the difference or relationship (Cohen's d, correlation coefficients)
  • Consider the context and implications of the results for the field of study or application

Common Pitfalls and Misconceptions

• Multiple testing issues arise when conducting numerous hypothesis tests simultaneously
  • Increased likelihood of Type I errors (false positives) due to chance alone
  • Apply appropriate corrections or adjustments to maintain the desired overall significance level (Bonferroni, false discovery rate)
• Misinterpretation of p-values as the probability of the null hypothesis being true or the probability of the results occurring by chance alone
  • P-values represent the probability of observing the sample results or more extreme outcomes, assuming the null hypothesis is true
• Overreliance on statistical significance without considering practical significance or effect sizes
  • Statistically significant results may not always be practically meaningful or impactful
• Confusing statistical significance with clinical or practical importance in applied settings
• Failing to check and validate the assumptions of the chosen hypothesis test
  • Violations of assumptions can lead to invalid or misleading results
• Misusing tests for normality (Shapiro-Wilk, Kolmogorov-Smirnov) to determine the appropriate hypothesis test
  • These tests assess the plausibility of normality but do not confirm it definitively
  • With large sample sizes, minor deviations from normality may be flagged as significant

Real-world Applications in Engineering

• Quality control and process monitoring in manufacturing
  • Hypothesis tests to detect shifts in process parameters (mean, variance)
  • Statistical process control charts (Shewhart, CUSUM) to identify out-of-control conditions
• Reliability engineering and failure analysis
  • Hypothesis tests to compare failure rates or mean time to failure between different designs or components
  • Accelerated life testing to make inferences about product reliability under normal use conditions
• Design of experiments and optimization
  • Hypothesis tests to assess the significance of factors and their interactions on a response variable
  • Response surface methodology to model and optimize process parameters
• Biomedical engineering and clinical trials
  • Hypothesis tests to evaluate the efficacy and safety of medical devices or interventions
  • Adaptive designs and interim analyses to make early decisions and allocate resources efficiently
• Environmental engineering and risk assessment
  • Hypothesis tests to compare pollutant levels or environmental indicators across different sites or time periods
  • Dose-response modeling and benchmark dose estimation for setting exposure limits
• Transportation engineering and traffic analysis
  • Hypothesis tests to assess the impact of interventions or policies on traffic flow, safety, or user behavior
  • Time series analysis and forecasting to predict future traffic patterns and demand