📈Theoretical Statistics Unit 4 – Multivariate distributions

Key Concepts and Definitions

• Multivariate distributions involve multiple random variables and their joint behavior
• Random vectors consist of two or more random variables arranged in a vector format
• Joint probability density functions (PDFs) describe the probability of multiple random variables taking on specific values simultaneously
• Marginal distributions focus on individual random variables within a multivariate setting
  • Obtained by integrating the joint PDF over the other variables
• Conditional distributions describe the probability of one random variable given fixed values of the others
• Covariance measures the linear relationship between two random variables in a multivariate distribution
• Correlation coefficients quantify the strength and direction of the linear relationship between two random variables
  • Range from -1 (perfect negative correlation) to 1 (perfect positive correlation)

Types of Multivariate Distributions

• Multivariate normal distribution is characterized by a mean vector and a covariance matrix
  • Assumes all marginal distributions are normally distributed
• Multivariate t-distribution has heavier tails compared to the multivariate normal distribution
  • Useful for modeling data with outliers or when the sample size is small
• Multivariate Poisson distribution models counts of rare events across multiple dimensions
• Dirichlet distribution is a continuous multivariate probability distribution often used in Bayesian inference
  • Models the probability of a set of non-negative variables summing to a fixed value
• Multinomial distribution is a discrete multivariate distribution that generalizes the binomial distribution
  • Models the outcomes of a fixed number of independent trials, each with a specific probability
• Wishart distribution is a generalization of the chi-squared distribution to multiple dimensions
  • Used for modeling covariance matrices in multivariate statistics

Properties of Multivariate Distributions

• Symmetry: Some multivariate distributions, such as the multivariate normal, exhibit symmetry around their mean vector
• Tail behavior: Different multivariate distributions have varying tail behaviors (light-tailed, heavy-tailed)
  • Affects the likelihood of extreme values occurring simultaneously
• Dependence structure: Multivariate distributions can capture different types of dependence between random variables
  • Linear dependence (covariance, correlation)
  • Nonlinear dependence (copulas)
• Marginal and conditional properties: Marginal and conditional distributions derived from a multivariate distribution may have different properties than the joint distribution
• Transformations: Multivariate distributions can be transformed to obtain new distributions with desired properties
  • Linear transformations (affine transformations)
  • Nonlinear transformations (copula transformations)

Joint Probability Density Functions

• Joint PDFs provide a complete description of the probability distribution for multiple random variables
• For continuous random variables, the joint PDF is a non-negative function that integrates to 1 over the entire domain
• The probability of random variables falling within a specific region is given by the integral of the joint PDF over that region
• Joint PDFs can be used to calculate probabilities, moments, and other quantities of interest
  • Means, variances, covariances
  • Marginal and conditional distributions
• Visualization of joint PDFs can provide insights into the dependence structure and shape of the distribution
  • Contour plots, 3D surface plots

Marginal and Conditional Distributions

• Marginal distributions focus on individual random variables within a multivariate setting
  • Obtained by integrating the joint PDF over the other variables
• Conditional distributions describe the probability distribution of one random variable given fixed values of the others
  • Obtained by dividing the joint PDF by the marginal PDF of the conditioning variables
• Marginal and conditional distributions allow for the analysis of specific subsets of variables
• The relationship between joint, marginal, and conditional distributions is given by the multiplication rule:
  • $P(X,Y) = P(X|Y) \cdot P(Y)$
• Conditional expectations and variances can be calculated using conditional distributions
  • $E[X|Y] = \int x \cdot f_{X|Y}(x|y) dx$
  • $Var[X|Y] = E[X^2|Y] - (E[X|Y])^2$

Covariance and Correlation in Multivariate Settings

• Covariance measures the linear relationship between two random variables in a multivariate distribution
  • Positive covariance indicates variables tend to increase or decrease together
  • Negative covariance indicates variables tend to move in opposite directions
• Correlation coefficients normalize covariance to measure the strength and direction of the linear relationship
  • Pearson correlation coefficient: $\rho_{XY} = \frac{Cov(X,Y)}{\sqrt{Var(X)Var(Y)}}$
• Covariance matrices summarize the pairwise covariances between all variables in a multivariate distribution
  • Diagonal elements are variances, off-diagonal elements are covariances
• Correlation matrices summarize the pairwise correlations between all variables
  • Diagonal elements are always 1, off-diagonal elements are correlation coefficients
• Covariance and correlation help identify linear relationships and dependencies in multivariate data
  • Used in principal component analysis (PCA), factor analysis, and other multivariate techniques

Multivariate Normal Distribution

• The multivariate normal distribution is a generalization of the univariate normal distribution to multiple dimensions
• Characterized by a mean vector $\mu$ and a covariance matrix $\Sigma$
  • PDF: $f_X(x) = \frac{1}{\sqrt{(2\pi)^k |\Sigma|}} \exp\left(-\frac{1}{2}(x-\mu)^T \Sigma^{-1} (x-\mu)\right)$
• All marginal distributions of a multivariate normal are also normally distributed
• Conditional distributions of a multivariate normal, given fixed values of some variables, are also normally distributed
• The multivariate normal distribution has elliptical contours of equal probability density
  • Shape and orientation determined by the covariance matrix
• Many statistical techniques assume or rely on the multivariate normal distribution
  • Multivariate linear regression, discriminant analysis, Hotelling's T-squared test

Applications and Real-World Examples

• Portfolio optimization in finance: Modeling the joint distribution of asset returns to minimize risk and maximize returns
• Image processing and computer vision: Representing pixel intensities or features as multivariate random variables
• Genetics and bioinformatics: Analyzing the joint distribution of gene expression levels or genetic markers
• Environmental sciences: Modeling the joint distribution of pollutant concentrations, weather variables, or ecological indicators
• Marketing and consumer research: Investigating the joint distribution of customer preferences, demographics, and purchasing behavior
• Quality control and manufacturing: Monitoring the joint distribution of product characteristics to detect defects or anomalies
• Medical research: Analyzing the joint distribution of biomarkers, risk factors, or treatment outcomes in clinical studies