⏳Intro to Time Series Unit 13 – Time Series Regression & Intervention Analysis

Key Concepts

• Time series data consists of observations collected sequentially over time, such as daily stock prices or monthly sales figures
• Stationarity is a crucial property for many time series models, requiring constant mean and variance over time
  • Differencing and transformations can help achieve stationarity in non-stationary series
• Autocorrelation measures the correlation between observations at different time lags, providing insights into the temporal dependence structure
• Partial autocorrelation measures the correlation between observations at different lags, after removing the effects of intermediate lags
• White noise is a series of uncorrelated random variables with zero mean and constant variance, serving as a benchmark for model residuals
• Seasonality refers to regular patterns that repeat over fixed time intervals, such as yearly or weekly cycles
• Trend represents the long-term direction of a time series, which can be linear or nonlinear

Time Series Basics

• Time series data is characterized by its temporal ordering, with observations collected at regular intervals (hourly, daily, monthly)
• Stationarity assumes that the statistical properties of a series remain constant over time, enabling reliable forecasting
  • Weak stationarity requires constant mean and variance, while strong stationarity also requires constant covariance structure
• Trend and seasonality are common patterns in time series data, representing long-term changes and periodic fluctuations, respectively
• Differencing is a technique used to remove trend and achieve stationarity by computing the differences between consecutive observations
• Seasonal differencing involves taking differences between observations separated by a fixed seasonal period to remove seasonal patterns
• Transformations, such as logarithmic or power transformations, can stabilize variance and make the series more suitable for modeling
• Decomposition methods, like classical decomposition or STL, separate a time series into trend, seasonal, and residual components

Regression Models for Time Series

• Time series regression models incorporate lagged values of the dependent variable and explanatory variables to capture temporal dependencies
• Autoregressive (AR) models express the current observation as a linear combination of past observations, with order $p$ denoting the number of lags
  • AR(1) model: $y_t = \phi_1 y_{t-1} + \varepsilon_t$, where $\phi_1$ is the autoregressive coefficient and $\varepsilon_t$ is white noise
• Moving Average (MA) models express the current observation as a linear combination of past forecast errors, with order $q$ denoting the number of lags
  • MA(1) model: $y_t = \varepsilon_t + \theta_1 \varepsilon_{t-1}$, where $\theta_1$ is the moving average coefficient
• Autoregressive Moving Average (ARMA) models combine AR and MA terms to capture both autoregressive and moving average dependencies
  • ARMA(1,1) model: $y_t = \phi_1 y_{t-1} + \varepsilon_t + \theta_1 \varepsilon_{t-1}$
• Autoregressive Integrated Moving Average (ARIMA) models extend ARMA to handle non-stationary series by including differencing terms
  • ARIMA(p,d,q) model: $(1-B)^d y_t = \phi_1 (1-B)^d y_{t-1} + \varepsilon_t + \theta_1 \varepsilon_{t-1}$, where $d$ is the differencing order
• Seasonal ARIMA (SARIMA) models incorporate seasonal AR, MA, and differencing terms to capture both non-seasonal and seasonal patterns
• Exogenous variables can be included in regression models to account for external factors influencing the time series

Intervention Analysis

• Intervention analysis assesses the impact of external events or policy changes on a time series, such as the introduction of a new product or a natural disaster
• Step interventions represent permanent level shifts in the series, modeled using a dummy variable that changes from 0 to 1 at the intervention point
• Pulse interventions represent temporary level shifts that decay over time, modeled using a dummy variable that is 1 at the intervention point and 0 elsewhere
• Ramp interventions represent gradual level shifts that occur over a period of time, modeled using a dummy variable that increases linearly during the intervention period
• Transfer function models incorporate the effect of an intervention variable on the time series, allowing for lagged and dynamic responses
  • Transfer function model: $y_t = \omega(B) x_t + \frac{\theta(B)}{\phi(B)} \varepsilon_t$, where $\omega(B)$ is the transfer function and $x_t$ is the intervention variable
• Outlier detection and adjustment are important in intervention analysis to identify and account for unusual observations that may distort the intervention effect
• Significance tests, such as t-tests or likelihood ratio tests, are used to assess the statistical significance of the intervention effect

Statistical Tests and Diagnostics

• Augmented Dickey-Fuller (ADF) test assesses the presence of a unit root in a time series, with the null hypothesis of non-stationarity
  • Rejecting the null hypothesis suggests the series is stationary, while failing to reject indicates the need for differencing
• Kwiatkowski-Phillips-Schmidt-Shin (KPSS) test assesses the stationarity of a time series, with the null hypothesis of stationarity
  • Rejecting the null hypothesis suggests the series is non-stationary, while failing to reject supports stationarity
• Ljung-Box test checks for the presence of autocorrelation in the residuals of a fitted model, with the null hypothesis of no autocorrelation
  • Rejecting the null hypothesis indicates the model may not have captured all the temporal dependencies
• Residual plots, such as standardized residuals versus time or fitted values, help identify patterns or outliers in the model residuals
• Normal probability plots assess the normality assumption of the residuals, with deviations from a straight line indicating non-normality
• Information criteria, such as Akaike Information Criterion (AIC) or Bayesian Information Criterion (BIC), compare the goodness-of-fit of different models while penalizing model complexity
  • Lower values of AIC or BIC indicate better model fit, helping in model selection
• Cross-validation techniques, like rolling-origin or time series cross-validation, assess the out-of-sample performance of time series models

Practical Applications

• Forecasting future values of a time series is a common application, such as predicting sales, demand, or stock prices
  • Point forecasts provide a single estimate of the future value, while interval forecasts give a range of plausible values
• Anomaly detection identifies unusual or unexpected observations in a time series, which can indicate fraud, system failures, or rare events
• Trend analysis helps understand the long-term direction of a time series, informing strategic decision-making and resource allocation
• Seasonal adjustment removes the seasonal component from a time series, revealing the underlying trend and facilitating comparisons across different periods
• Causal analysis investigates the relationship between a time series and external factors, such as the impact of advertising on sales or the effect of weather on energy consumption
• Nowcasting provides real-time estimates of current or very recent values of a time series, often using high-frequency data or proxy variables
• Scenario analysis evaluates the potential impact of different future scenarios on a time series, helping in risk management and contingency planning

Common Pitfalls and Solutions

• Overfitting occurs when a model is too complex and fits the noise in the data, leading to poor out-of-sample performance
  • Regularization techniques, such as L1 (Lasso) or L2 (Ridge) penalties, can help mitigate overfitting by shrinking the model coefficients
• Multicollinearity arises when explanatory variables are highly correlated, making it difficult to interpret individual variable effects
  • Variable selection methods, like stepwise regression or Lasso, can help identify the most relevant variables and reduce multicollinearity
• Autocorrelation in the residuals violates the independence assumption of regression models, leading to biased standard errors and inefficient estimates
  • Including lagged dependent variables or using autoregressive error terms can help capture the autocorrelation structure
• Non-normality of residuals can affect the validity of statistical tests and confidence intervals
  • Transformations, such as Box-Cox or log transformations, can help achieve normality, or robust methods can be used
• Structural breaks or regime shifts can lead to instability in the model parameters over time
  • Piecewise regression, regime-switching models, or time-varying parameter models can accommodate structural breaks
• Outliers can have a disproportionate influence on the model estimates and forecasts
  • Robust estimation methods, such as M-estimation or S-estimation, can reduce the impact of outliers
• Inadequate sample size can lead to imprecise estimates and unreliable inference
  • Aggregating data to a lower frequency or using panel data from multiple related series can increase the effective sample size

Advanced Topics

• Vector Autoregressive (VAR) models extend univariate time series models to multivariate settings, capturing the dynamic relationships among multiple time series
  • VAR models express each variable as a linear function of its own past values and the past values of other variables in the system
• Cointegration occurs when two or more non-stationary time series have a long-run equilibrium relationship, such that a linear combination of them is stationary
  • Error Correction Models (ECMs) incorporate the cointegrating relationship and capture both short-run dynamics and long-run equilibrium
• Generalized Autoregressive Conditional Heteroskedasticity (GARCH) models capture time-varying volatility in financial time series, where the variance of the series depends on its past values and past squared residuals
• State Space Models (SSMs) provide a flexible framework for modeling time series with unobserved components, such as trend, seasonal, and cycle
  • Kalman filtering and smoothing algorithms enable the estimation of the unobserved components and the model parameters
• Bayesian methods, such as Bayesian VAR or Bayesian structural time series models, incorporate prior information and provide a coherent framework for uncertainty quantification
• Machine learning techniques, like neural networks or random forests, can be adapted for time series forecasting, capturing complex nonlinear patterns and interactions
• Functional data analysis treats the entire time series as a single functional object, enabling the analysis of curves, shapes, and patterns in the data