🃏Engineering Probability Unit 8 – Expectation, Variance, and Statistical Moments

Key Concepts and Definitions

• Expectation represents the average value of a random variable over a large number of trials
• Variance measures the spread or dispersion of a random variable around its expected value
• Standard deviation is the square root of variance and provides a measure of variability in the same units as the random variable
• Moments are quantitative measures that describe the shape and characteristics of a probability distribution
  • First moment is the mean or expected value
  • Second moment is the variance
  • Third moment is related to skewness (asymmetry) of the distribution
  • Fourth moment is related to kurtosis (peakedness) of the distribution
• Moment generating functions are mathematical tools used to calculate moments of a random variable
• Covariance measures the linear relationship between two random variables
• Correlation coefficient is a standardized measure of the linear relationship between two random variables, ranging from -1 to 1

Probability Distributions Recap

• Probability distributions describe the likelihood of different outcomes for a random variable
• Discrete probability distributions are used for random variables that can only take on a countable number of values (rolling a die)
• Continuous probability distributions are used for random variables that can take on any value within a specified range (height of students in a class)
• Common discrete probability distributions include Bernoulli, Binomial, Poisson, and Geometric distributions
• Common continuous probability distributions include Uniform, Normal (Gaussian), Exponential, and Gamma distributions
• Probability density functions (PDFs) and cumulative distribution functions (CDFs) are used to characterize continuous probability distributions
• Probability mass functions (PMFs) are used to characterize discrete probability distributions

Understanding Expectation (Mean)

• Expectation is a key concept in probability theory and statistics, representing the average value of a random variable
• For a discrete random variable $X$ with probability mass function $P(X = x_i)$, the expectation is calculated as: $E[X] = \sum_{i} x_i P(X = x_i)$
• For a continuous random variable $X$ with probability density function $f(x)$, the expectation is calculated as: $E[X] = \int_{-\infty}^{\infty} x f(x) dx$
• Linearity of expectation states that for random variables $X$ and $Y$ and constants $a$ and $b$: $E[aX + bY] = aE[X] + bE[Y]$
  • This property holds even if $X$ and $Y$ are dependent
• The expected value of a function $g(X)$ of a random variable $X$ is given by: $E[g(X)] = \sum_{i} g(x_i) P(X = x_i)$ for discrete $X$ and $E[g(X)] = \int_{-\infty}^{\infty} g(x) f(x) dx$ for continuous $X$
• The law of the unconscious statistician (LOTUS) is another name for the expected value of a function of a random variable

Variance and Standard Deviation

• Variance measures the average squared deviation of a random variable from its expected value
• For a random variable $X$ with expectation $E[X]$, the variance is calculated as: $Var(X) = E[(X - E[X])^2]$
• Variance can also be calculated using the formula: $Var(X) = E[X^2] - (E[X])^2$
• Standard deviation is the square root of variance: $\sigma_X = \sqrt{Var(X)}$
• Properties of variance include:
  • $Var(aX + b) = a^2 Var(X)$ for constants $a$ and $b$
  • $Var(X + Y) = Var(X) + Var(Y)$ if $X$ and $Y$ are independent
• Chebyshev's inequality provides a bound on the probability that a random variable deviates from its mean by more than a certain number of standard deviations

Higher-Order Moments

• Higher-order moments provide additional information about the shape and characteristics of a probability distribution
• Skewness is a measure of the asymmetry of a distribution, calculated using the third central moment: $Skewness(X) = E[(X - E[X])^3] / \sigma_X^3$
  • Positive skewness indicates a longer right tail, while negative skewness indicates a longer left tail
• Kurtosis is a measure of the peakedness or flatness of a distribution, calculated using the fourth central moment: $Kurtosis(X) = E[(X - E[X])^4] / \sigma_X^4$
  • Higher kurtosis indicates a more peaked distribution, while lower kurtosis indicates a flatter distribution
• Moment generating functions (MGFs) are used to calculate moments of a random variable
  • The MGF of a random variable $X$ is defined as: $M_X(t) = E[e^{tX}]$
  • Moments can be obtained by differentiating the MGF and evaluating at $t = 0$

Properties and Theorems

• Covariance measures the linear relationship between two random variables $X$ and $Y$: $Cov(X, Y) = E[(X - E[X])(Y - E[Y])]$
  • Positive covariance indicates a positive linear relationship, while negative covariance indicates a negative linear relationship
• Correlation coefficient is a standardized measure of the linear relationship between two random variables: $\rho_{X,Y} = Cov(X, Y) / (\sigma_X \sigma_Y)$
  • Correlation ranges from -1 (perfect negative linear relationship) to 1 (perfect positive linear relationship)
• Cauchy-Schwarz inequality states that for random variables $X$ and $Y$: $|Cov(X, Y)| \leq \sigma_X \sigma_Y$
• Markov's inequality provides an upper bound on the probability that a non-negative random variable exceeds a certain value
• Jensen's inequality relates the expectation of a convex function of a random variable to the function of the expectation: $E[g(X)] \geq g(E[X])$ for convex functions $g$

Applications in Engineering

• Expectation and variance are used in signal processing to characterize the properties of random signals (noise)
• In reliability engineering, the expected value and variance of the time to failure are used to assess the reliability of systems and components
• Queueing theory relies on expectation and variance to analyze the performance of queueing systems (customer wait times, server utilization)
• In quality control, the expected value and variance of product characteristics are used to monitor and control manufacturing processes
• Expectation and variance are used in financial engineering to model asset prices, portfolio returns, and risk management (Value at Risk)
• In machine learning, expectation and variance are used to assess the performance of models and to guide the selection of model parameters
• Expectation and variance are fundamental in hypothesis testing and confidence interval estimation in statistical inference

Practice Problems and Examples

• Calculate the expected value and variance of the number of heads obtained when flipping a fair coin 10 times
• For a continuous random variable $X$ with probability density function $f(x) = 2x$ for $0 \leq x \leq 1$, find $E[X]$ and $Var(X)$
• Prove that for independent random variables $X$ and $Y$, $Var(X + Y) = Var(X) + Var(Y)$
• A machine produces bolts with lengths that follow a normal distribution with a mean of 10 cm and a standard deviation of 0.5 cm. What is the probability that a randomly selected bolt has a length between 9.5 cm and 10.5 cm?
• The time between arrivals at a service counter follows an exponential distribution with a mean of 5 minutes. What is the probability that the time between two consecutive arrivals exceeds 10 minutes?
• The weights of apples in a harvest follow a gamma distribution with shape parameter 3 and scale parameter 0.5. Find the expected value and standard deviation of the weight of an apple.
• Determine the moment generating function for a Poisson random variable with parameter $\lambda$, and use it to calculate the mean and variance.