🕵️Investigative Reporting Unit 11 – Data Analysis for Investigative Reporting

Key Concepts and Definitions

• Data journalism combines traditional journalism with data analysis to uncover stories and insights
• Data literacy is the ability to read, understand, and communicate data effectively
• Data sources can be primary (collected by the journalist) or secondary (obtained from existing sources)
• Data types include numerical (quantitative) and categorical (qualitative) data
  • Numerical data consists of measurements or counts (age, income, etc.)
  • Categorical data represents characteristics or attributes (gender, race, etc.)
• Data cleaning involves identifying and correcting errors, inconsistencies, and missing values in datasets
• Data visualization transforms complex data into easily understandable visual representations (charts, graphs, maps)
• Statistical analysis techniques help journalists identify patterns, trends, and relationships within data
• Correlation measures the relationship between two variables, while causation establishes a cause-and-effect relationship

Data Sources and Collection Methods

• Open data portals provide access to government and public datasets (Data.gov, World Bank Open Data)
• Freedom of Information Act (FOIA) requests allow journalists to obtain data from government agencies
• Web scraping automates the process of extracting data from websites using specialized tools or programming languages
• Surveys and interviews enable journalists to collect primary data directly from sources
  • Online surveys reach a large audience and provide quick results
  • In-person interviews offer more in-depth and personalized responses
• Crowdsourcing involves gathering data from a large group of people, often through online platforms or social media
• Data partnerships with organizations or experts can provide access to specialized datasets and insights
• Sensor data from IoT devices (smartphones, wearables) can be used to track patterns and behaviors

Data Cleaning and Preparation

• Data validation checks for accuracy, completeness, and consistency of data entries
• Removing duplicates ensures that each data point is unique and not counted multiple times
• Handling missing values involves identifying and addressing gaps in the dataset
  • Deletion removes rows or columns with missing values
  • Imputation estimates missing values based on other available data
• Data normalization scales values to a common range to allow for fair comparisons
• Outlier detection identifies and investigates data points that significantly deviate from the norm
• Data aggregation combines data from multiple sources or levels of granularity for analysis
• Feature selection chooses the most relevant variables for analysis while reducing dimensionality
• Data splitting divides the dataset into training and testing subsets for model evaluation

Statistical Analysis Techniques

• Descriptive statistics summarize and describe the main features of a dataset (mean, median, mode, standard deviation)
• Inferential statistics make predictions or draw conclusions about a population based on a sample
• Hypothesis testing evaluates the likelihood of a claim being true by comparing it to a null hypothesis
• Regression analysis models the relationship between a dependent variable and one or more independent variables
  • Linear regression assumes a linear relationship between variables
  • Logistic regression predicts binary outcomes (yes/no, true/false)
• Time series analysis examines data points collected over time to identify trends, seasonality, and forecasts
• Clustering groups data points based on their similarities or differences
• Sentiment analysis determines the emotional tone or opinion expressed in text data
• Geographic analysis explores the spatial relationships and patterns within data

Data Visualization Tools

• Tableau is a powerful and user-friendly platform for creating interactive dashboards and visualizations
• Google Charts provides a free and customizable way to create charts and graphs for web-based projects
• D3.js is a JavaScript library for creating dynamic and interactive visualizations in web browsers
• Python libraries like Matplotlib and Seaborn offer flexibility and customization for data visualization
  • Matplotlib is a comprehensive plotting library for creating static, animated, and interactive visualizations
  • Seaborn is built on top of Matplotlib and provides a high-level interface for creating informative and attractive statistical graphics
• R packages such as ggplot2 and plotly enable the creation of publication-quality graphics
• Infogram and Piktochart allow users to create infographics and visual stories without coding skills
• Mapbox and Carto specialize in creating interactive and customizable maps for data storytelling

Interpreting Results for Reporting

• Statistical significance indicates whether the observed results are likely due to chance or a real effect
• Effect size measures the magnitude or strength of a relationship or difference between variables
• Confidence intervals provide a range of values within which the true population parameter is likely to fall
• Margin of error expresses the amount of random sampling error in survey results
• Correlation does not imply causation; additional evidence is needed to establish a causal relationship
• Contextualizing results involves considering the broader implications and limitations of the findings
• Communicating uncertainty helps readers understand the level of confidence in the reported results
• Data-driven storytelling combines compelling narrative with data insights to engage and inform audiences

Ethical Considerations in Data Journalism

• Protecting privacy and confidentiality is crucial when handling sensitive or personally identifiable information
• Informed consent ensures that individuals understand the purpose and potential risks of their data being used
• Bias and fairness in data collection and analysis can lead to misrepresentation or discrimination of certain groups
• Transparency about data sources, methods, and limitations promotes accountability and trust
  • Providing access to raw data allows others to verify and reproduce the findings
  • Disclosing any potential conflicts of interest maintains journalistic integrity
• Responsible data storage and security measures prevent unauthorized access or breaches
• Ethical data visualization avoids misleading or manipulating the audience through visual choices
• Collaborating with diverse teams and seeking expert input can help identify and mitigate ethical concerns

Case Studies and Real-World Applications

• The Panama Papers investigation revealed a global network of offshore tax havens and financial secrecy
• ProPublica's "Machine Bias" series exposed racial disparities in algorithmic decision-making systems
• The Guardian's "The Counted" project tracked and analyzed data on people killed by police in the United States
• Reuters' "The Child Exchange" investigation uncovered a private online marketplace for adopted children
• The Washington Post's "Fatal Force" database examines police shootings in the United States
• BuzzFeed News' "The Tennis Racket" investigation used data analysis to uncover widespread match-fixing in professional tennis
• The Atlanta Journal-Constitution's "Doctors & Sex Abuse" series revealed a nationwide problem of physician sexual misconduct
• The Seattle Times' "Quantity of Care" investigation exposed unnecessary medical procedures and wasteful spending in hospitals