7.2 Logistic regression

Logistic regression is a powerful tool for predicting binary outcomes in supervised learning. It uses the sigmoid function to transform linear combinations of features into probabilities, making it ideal for classification tasks like fraud detection or medical diagnosis.

This method builds on linear regression concepts but adapts them for categorical outcomes. By understanding logistic regression's assumptions, model building techniques, and evaluation metrics, you'll be equipped to tackle a wide range of real-world classification problems effectively.

Logistic Regression Concept

Mathematical Foundation

Top images from around the web for Mathematical Foundation

• 

  Data science: Gentle intro to logistic regression View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Data science: Gentle intro to logistic regression View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

1 of 3

Top images from around the web for Mathematical Foundation

• 

  Data science: Gentle intro to logistic regression View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

• 

  Data science: Gentle intro to logistic regression View original

  Is this image relevant?

• 

  Maximum Likelihood Estimate and Logistic Regression simplified — Pavan Mirla View original

  Is this image relevant?

1 of 3

• Logistic regression predicts binary outcomes based on independent variables
• Sigmoid function transforms real-valued numbers into values between 0 and 1
• Logit function forms the mathematical core (natural logarithm of odds ratio)
• Logistic regression equation expressed as $log(\frac{p}{1-p}) = β₀ + β₁x₁ + β₂x₂ + ... + βₙxₙ$
  • p represents probability of positive class
  • βᵢ denotes coefficients
• Maximum likelihood estimation optimizes model fit to observed data
• Decision boundary separates two classes at 0.5 predicted probability
• Assumes linear relationship between independent variables and log-odds (not probability itself)

Key Assumptions and Characteristics

• Binary outcome prediction (yes/no, true/false)
• Feature selection crucial for model performance
  • Choose relevant predictors (age, income)
  • Transform variables if necessary (log transformation)
• Scaling or normalizing features ensures proportional contribution
  • Standardization (z-score)
  • Min-max scaling
• Adjustable classification threshold balances precision and recall
• Regularization techniques prevent overfitting
  • L1 (Lasso)
  • L2 (Ridge)
• Cross-validation assesses model generalizability
• Imbalanced dataset handling techniques
  • Oversampling (SMOTE)
  • Undersampling
  • Adjusting class weights

Logistic Regression for Classification

Data Preparation and Model Building

• Binary classification problems involve two possible outcomes
• Feature engineering transforms raw data into informative predictors
  • Create interaction terms (age * income)
  • Encode categorical variables (one-hot encoding)
• Scaling features improves model performance
  • Normalize numeric variables to [0,1] range
  • Standardize features to zero mean and unit variance
• Split data into training and testing sets
  • Common split ratios (70/30, 80/20)
• Build logistic regression model using appropriate software
  • Python (

    sklearn.linear_model.LogisticRegression

    )
  • R (

    glm()

    function)

Model Optimization and Validation

• Adjust classification threshold based on problem requirements
  • Lower threshold increases sensitivity (medical diagnosis)
  • Higher threshold increases specificity (fraud detection)
• Apply regularization to prevent overfitting
  • L1 regularization encourages sparse solutions
  • L2 regularization shrinks coefficients towards zero
• Implement cross-validation for robust performance estimation
  • K-fold cross-validation (k=5 or k=10)
  • Stratified cross-validation for imbalanced datasets
• Handle imbalanced datasets to improve model performance
  • Random oversampling of minority class
  • Random undersampling of majority class
  • Synthetic Minority Over-sampling Technique (SMOTE)

Logistic Regression Model Evaluation

Performance Metrics

• Confusion matrix displays classification results
  • True Positives (TP), True Negatives (TN)
  • False Positives (FP), False Negatives (FN)
• Accuracy measures overall correctness of predictions
  • $Accuracy = \frac{TP + TN}{TP + TN + FP + FN}$
  • Can be misleading for imbalanced datasets
• Precision quantifies correct positive predictions
  • $Precision = \frac{TP}{TP + FP}$
  • Important in scenarios with high false positive cost (spam detection)
• Recall measures proportion of actual positives identified
  • $Recall = \frac{TP}{TP + FN}$
  • Crucial in scenarios with high false negative cost (disease screening)
• F1 score balances precision and recall
  • $F1 = 2 * \frac{Precision * Recall}{Precision + Recall}$
  • Useful for imbalanced datasets

Advanced Evaluation Techniques

• ROC curve plots true positive rate against false positive rate
  • Visualizes model performance across thresholds
  • Ideal curve hugs top-left corner
• AUC-ROC provides single value for model discrimination
  • Ranges from 0.5 (random guessing) to 1.0 (perfect classification)
  • Values above 0.8 indicate good model performance
• Log loss measures probabilistic predictions
  • $Log Loss = -\frac{1}{N}\sum_{i=1}^N [y_i log(p_i) + (1-y_i) log(1-p_i)]$
  • Penalizes confident misclassifications heavily
• Calibration plots assess probability estimates accuracy
  • Well-calibrated model produces diagonal line
  • Helps identify overconfident or underconfident predictions

Logistic Regression Coefficient Interpretation

Understanding Coefficients

• Coefficients represent change in log-odds for one-unit predictor increase
• Positive coefficients indicate increased log-odds of outcome
• Negative coefficients indicate decreased log-odds of outcome
• Magnitude of coefficient reflects predictor's impact strength
• Interpret coefficients while holding other variables constant
• Standardized coefficients allow comparison between predictors
  • Useful when predictors have different scales (age vs. income)

Odds Ratios and Practical Interpretation

• Odds ratios derived by exponentiating coefficients (e^β)
• Odds ratio > 1 indicates higher odds of outcome
  • Example: Odds ratio of 1.5 for age means 50% increase in odds for each year
• Odds ratio < 1 indicates lower odds of outcome
  • Example: Odds ratio of 0.8 for exercise means 20% decrease in odds for each unit
• Confidence intervals provide range of plausible odds ratio values
  • Narrow intervals indicate more precise estimates
  • Wide intervals suggest less certainty in estimate
• Practical interpretation considers domain context
  • Medical: Odds ratio of 2.5 for smoking indicates 150% increased odds of lung cancer
  • Marketing: Odds ratio of 1.2 for email opens suggests 20% higher odds of purchase