11.4 Forecasting with Expert Judgment and Delphi Method
5 min read•august 14, 2024
and the are powerful tools for forecasting when data is limited or complex. These approaches tap into the knowledge of experts to make predictions, especially useful for long-term or uncertain situations where historical data might not tell the whole story.
By combining expert opinions with quantitative methods, forecasters can create more robust predictions. The Delphi method, in particular, helps reduce bias and encourages consensus through anonymous rounds of feedback, making it a valuable technique for gathering and refining expert insights.
Expert Judgment in Forecasting
Role of Expert Judgment
Top images from around the web for Role of Expert Judgment
Frontiers | Acupuncture Therapy for Cognitive Impairment: A Delphi Expert Consensus Survey View original
Is this image relevant?
Frontiers | Application and Evaluation of an Expert Judgment Elicitation Procedure for Correlations View original
Is this image relevant?
Frontiers | Defining Spotting in Dance: A Delphi Method Study Evaluating Expert Opinions View original
Is this image relevant?
Frontiers | Acupuncture Therapy for Cognitive Impairment: A Delphi Expert Consensus Survey View original
Is this image relevant?
Frontiers | Application and Evaluation of an Expert Judgment Elicitation Procedure for Correlations View original
Is this image relevant?
1 of 3
Top images from around the web for Role of Expert Judgment
Frontiers | Acupuncture Therapy for Cognitive Impairment: A Delphi Expert Consensus Survey View original
Is this image relevant?
Frontiers | Application and Evaluation of an Expert Judgment Elicitation Procedure for Correlations View original
Is this image relevant?
Frontiers | Defining Spotting in Dance: A Delphi Method Study Evaluating Expert Opinions View original
Is this image relevant?
Frontiers | Acupuncture Therapy for Cognitive Impairment: A Delphi Expert Consensus Survey View original
Is this image relevant?
Frontiers | Application and Evaluation of an Expert Judgment Elicitation Procedure for Correlations View original
Is this image relevant?
1 of 3
- Uses knowledge and experience of subject matter experts to make predictions or provide input for forecasting models
- Particularly useful when historical data is limited, the forecasting problem is complex, or there are important qualitative factors to consider
- Provides valuable insights into market trends, technological advancements, and other external factors that may impact the variable being forecasted
- Can be used as a standalone forecasting method or in combination with quantitative techniques to improve forecast accuracy
Factors Affecting Expert Judgment Quality
- Depends on the expertise and credibility of the individuals involved
- Influenced by the structuring of the judgment elicitation process and the aggregation of multiple expert opinions
- Quality can be improved by carefully selecting diverse experts, using structured elicitation methods (Delphi), and combining multiple opinions
- Regular assessment of expert judgment performance against actual outcomes helps identify areas for improvement
Delphi Method for Expert Opinions
Delphi Method Overview
- Structured approach to eliciting and aggregating expert opinions for forecasting purposes
- Involves multiple rounds of anonymous questionnaires or surveys administered to a panel of experts
- In each round, experts provide their individual judgments or estimates, which are then summarized and shared with the group
- Experts have the opportunity to revise their responses based on the group feedback, promoting convergence towards a consensus opinion
Benefits of the Delphi Method
- Helps to reduce the influence of dominant individuals and encourages independent thinking
- Facilitates the exchange of ideas among experts and promotes a more balanced consideration of different perspectives
- Provides a systematic way to aggregate diverse expert opinions into a single forecast or estimate
- Can be conducted remotely and asynchronously, allowing for the inclusion of geographically dispersed experts
Delphi Method Process
- Selection of a diverse panel of experts with relevant knowledge and experience
- Development of a structured questionnaire or survey to elicit expert judgments
- Administration of the questionnaire to the expert panel, typically over multiple rounds
- Aggregation and summary of expert responses after each round, shared with the group as feedback
- Opportunity for experts to revise their judgments based on the group feedback, leading to convergence
- Final output is a set of aggregated expert judgments that can be used as forecasts or inputs for other models
Expert Judgment vs Quantitative Forecasting
Combining Expert Judgment with Quantitative Methods
- Expert judgment can be combined with quantitative forecasting methods to leverage the strengths of both approaches
- One common approach is to use expert judgment to adjust or refine the outputs of statistical forecasting models based on domain knowledge or insights into external factors (market trends, policy changes)
- Expert judgment can also be used to provide initial estimates or starting points for quantitative models, particularly when historical data is limited
- Bayesian methods can be employed to formally incorporate expert opinions as prior distributions in statistical forecasting models
Integration in Decision Support Systems
- Decision support systems and forecasting software often include features to facilitate the integration of expert judgment with quantitative techniques
- These systems may provide structured interfaces for eliciting and aggregating expert opinions
- They may also offer tools for combining expert judgments with statistical forecasts using methods like Bayesian updating or weighted averaging
- Integration of expert judgment in decision support systems can help to streamline the forecasting process and ensure consistent application of best practices
Benefits of Combining Approaches
- Combining expert judgment with quantitative methods can help to improve forecast accuracy, especially in situations where the underlying assumptions of statistical models may not fully capture the complexity of the forecasting problem
- Leverages the strengths of both approaches - the domain knowledge and adaptability of expert judgment, and the data-driven rigor of quantitative methods
- Allows for the incorporation of qualitative factors and external insights that may be difficult to capture in purely data-driven models
- Provides a more robust and comprehensive approach to forecasting, particularly in uncertain or rapidly changing environments (emerging markets, disruptive technologies)
Strengths and Limitations of Expert Judgment
Strengths of Expert Judgment
- Offers the ability to incorporate domain-specific knowledge and consider qualitative factors that may be difficult to capture in quantitative models
- Can provide valuable insights into the potential impact of external events, policy changes, or strategic decisions on the variable being forecasted
- Particularly useful in situations where data is scarce, the forecasting horizon is long, or the problem is highly complex or uncertain
- Allows for adaptability and flexibility in responding to unique or changing circumstances that may not be well-represented in historical data
Limitations and Biases
- Subject to various cognitive biases and limitations, such as overconfidence (experts may be overly certain of their judgments), anchoring (relying too heavily on initial estimates), and availability bias (overweighting recent or salient information)
- Accuracy can be influenced by factors such as the selection of experts, the framing of questions, and the aggregation method used
- Expert opinions may be prone to subjectivity and inconsistency, particularly when dealing with long-term forecasts or highly uncertain outcomes
- Effectiveness depends on the quality and diversity of the experts involved, as well as the robustness of the elicitation and aggregation processes
Mitigating Limitations
- Use structured elicitation methods (Delphi) to reduce biases and improve consistency
- Combine expert judgment with quantitative methods to balance subjective and objective inputs
- Regularly assess the performance of expert judgment against actual outcomes to identify areas for improvement
- Use multiple experts with diverse backgrounds and perspectives to reduce individual biases
- Frame questions and problems carefully to minimize ambiguity and anchoring effects
- Employ robust aggregation methods (weighted averaging, Bayesian updating) to combine expert opinions in a principled manner
Key Terms to Review (18)
Anonymity of responses: Anonymity of responses refers to the practice of ensuring that individual contributions in a study or survey remain confidential, allowing participants to provide honest and unbiased input without fear of repercussion. This concept is crucial in gathering expert judgments, particularly when using methods like the Delphi Method, as it helps reduce the influence of dominant voices and encourages open expression of opinions.
Consensus building: Consensus building is a collaborative process that seeks to reach an agreement among diverse stakeholders through open communication and negotiation. This approach encourages participation from all involved parties, allowing for the integration of various perspectives and ideas to arrive at a mutually acceptable solution. Consensus building fosters teamwork and trust, making it an essential aspect in the realm of expert judgment and methods like the Delphi technique.
Delphi Method: The Delphi Method is a structured communication technique used for gathering expert opinions and achieving consensus on future predictions or forecasts. By utilizing multiple rounds of questioning and feedback, this method helps to refine ideas and converge on a reliable forecast, making it especially valuable in both qualitative and quantitative forecasting contexts, such as demand and inventory forecasting.
Expert judgment: Expert judgment is a forecasting method that relies on the knowledge and experience of individuals with specialized expertise to make informed predictions about future events or trends. This approach is particularly valuable when quantitative data is limited or when complex, subjective factors must be considered. It often involves gathering insights from experts through structured processes to enhance the reliability and accuracy of forecasts.
Facilitated Discussion: Facilitated discussion is a structured dialogue led by a facilitator, aimed at gathering insights, opinions, and judgments from participants to inform decision-making or forecasting. This method leverages the diverse perspectives of a group to enhance the quality of information gathered, ensuring that various viewpoints are considered and synthesized into actionable knowledge. It is particularly useful in contexts where complex issues require collective expertise for accurate forecasting.
Forecast bias: Forecast bias refers to the systematic tendency of a forecasting method to overestimate or underestimate actual outcomes. It indicates a consistent error in predictions, which can be crucial when evaluating the effectiveness of different forecasting techniques and understanding their implications for decision-making.
Groupthink: Groupthink is a psychological phenomenon that occurs within a group when the desire for harmony and conformity results in irrational or dysfunctional decision-making. This often leads to the suppression of dissenting viewpoints, ultimately impairing the group's ability to critically analyze options and consider alternative solutions. In the context of forecasting with expert judgment and the Delphi method, groupthink can hinder the quality of forecasts by preventing diverse perspectives from being expressed and evaluated.
Iterative feedback: Iterative feedback is a process where information is continuously collected, assessed, and utilized to make incremental improvements in decision-making or forecasting. This approach allows experts to refine their judgments through repeated cycles of evaluation, ensuring that the predictions or recommendations evolve based on newly acquired data and insights.
L. J. van de Ven: L. J. van de Ven is a prominent figure in the field of forecasting, particularly known for his contributions to the Delphi Method, which is a structured communication technique used to gather expert opinions. He emphasized the importance of using expert judgment in forecasting, highlighting how collective intelligence can improve decision-making. His work has significantly influenced methodologies for forecasting in uncertain environments, integrating expert insights with systematic processes.
M. J. McCarthy: M. J. McCarthy is known for his contributions to the field of forecasting, particularly in the area of expert judgment and the Delphi Method. His work emphasizes the importance of gathering insights from experts to improve decision-making and prediction accuracy in uncertain environments. By utilizing structured communication techniques, McCarthy's methods help synthesize expert opinions, making them valuable tools in forecasting practices.
Market research forecasting: Market research forecasting is the process of predicting future market conditions, trends, and consumer behaviors based on data collected from various sources, including surveys, interviews, and market analysis. This method often relies on qualitative insights and expert opinions to inform the forecasts, making it crucial for strategic decision-making in businesses.
Mean Absolute Error: Mean Absolute Error (MAE) is a measure used to assess the accuracy of a forecasting model by calculating the average absolute differences between forecasted values and actual observed values. It provides a straightforward way to quantify how far off predictions are from reality, making it essential in evaluating the performance of various forecasting methods.
Qualitative forecasts: Qualitative forecasts are predictions made based on subjective judgment, intuition, and experience rather than on historical data or statistical methods. These forecasts often rely on insights from experts or stakeholders, making them especially useful in situations where data is scarce or when dealing with new or unprecedented events. Their effectiveness hinges on the knowledge and expertise of the individuals providing input.
Quantitative forecasts: Quantitative forecasts are predictions about future events or trends based on historical data and statistical methods. These forecasts rely on mathematical models to analyze past performance and identify patterns, allowing for data-driven decision making in areas like capacity planning and project management. By using numerical data, quantitative forecasts provide a systematic approach to estimating future outcomes with a focus on accuracy and reliability.
Rounds of Questioning: Rounds of questioning is a structured process used in forecasting, particularly in expert judgment and the Delphi method, where a series of questions are posed to a group of experts to gather insights and opinions on a particular topic or prediction. This method allows for the refinement of ideas through multiple iterations, enabling experts to adjust their responses based on the feedback and responses of others, ultimately leading to a more informed consensus.
Scenario planning: Scenario planning is a strategic planning method used to create flexible long-term plans by considering various possible future scenarios. It involves imagining different future states based on a range of factors, allowing organizations to better understand potential risks and opportunities, and to prepare for uncertainties in their environment. This approach is particularly useful for decision-making processes and can enhance the accuracy of forecasts by integrating qualitative insights.
Subjective probability: Subjective probability is an individual's personal estimate of the likelihood of an event occurring, based on their beliefs, experiences, and information rather than on objective data or formal calculations. This type of probability can vary significantly from person to person, as it is influenced by personal judgment and interpretation of information. In the context of forecasting, subjective probability becomes important when experts use their knowledge and insights to make predictions about uncertain future events.
Technology forecasting: Technology forecasting is the process of predicting future developments in technology, focusing on the potential impacts and opportunities it may create. This method relies on gathering insights from experts in the field and analyzing trends to make informed predictions about technological advancements. It helps organizations prepare for changes and make strategic decisions about investments, research, and development efforts.