Judgmental forecasts
A judgemental forecast assumes no accurate historical precedent (ie coronavirus), so we use judgement.
“Nowcasting” – Judgement must be used because of delays in the data such as GDP data only being only availible quarterly.
Judgement forecasting improves when
- Domain knowledge is present
- Information is timely and up to date
Subjective science which creates limitations
3 situations where it is used
- No availible data – statistical models are not applicable
- Adjusted using judgement (We may have to for covid)
- Combining statistical model with judgement forecast
4.1 Beware of limitations
Bias and limitations are present
memory, even with notes is inconsistent, things may be missed, bias towards more recent info, time of day, nutrition and sleep.
Personal and poltical agendas (wanting to exceed expectations for instance) can bias the judgement models
4.2 Key Principles
Set a forecasting task clearly and concisely
Implement a systematic approach – Like packing a bag, make sure you have everything you need before you start building the model
Document and justify – Keep track of decision rules and assumptions
Systematically evaluate forecasts – go back after the forecast events have transpired and evaluate
Segregate forecasters and users.
Forecasting is about predicting as accurately as possible – do not let personal biases affect the outcome
Forecasts are not targets, goals are targets, forecasts influence goals
Example
- Application Austrailian Public Medical Subsidies
- 4 categories
- Concession Payments
- Concession safety net
- General copayments
- general safety net
- copayments are made until the corresponding safety net is triggered at which point medicine is picked up 100% by the government
- Green = judgement forecast
- Pink = The data
- Blue = Calculations from the data
- forecasts done on a per capita basis
- Several changes reccommended by the book
- Need guidelines for new policies
- need to document forecast methodology
- The new forecasting policies need to be made by at least two different people within the organization
- Annual review of the forecasts
4.3 The Delphi Method
- A panel of experts is assembled
- Forecasting tasks/challenges are set and distributed to the experts
- Experts return initial forecasts and justifications. They are compiled and summarized to provide feedback.
- Experts receive feedback; forecasts reviewed; repeat
- Aggregation of forecasts used to make a final forecast
Experts and anonymity
Group size of 5-20; diverse experiences; experts remain anonymous and free of influence; all experts given equal say
Setting the forecasting task
begin with prelim round of info gathering
initial forecasts then sent in
Feedback
summary of statistics of the forecasts and outlines of qualitative justifications
Point out outliers to group
Iteration
Process is repeated until a rough concensus is reached.
Final Forecasts
Give equal weight to all experts forecasts, extreme outliers can distort final forecast
Limitations and variations
Time consuming; requires focus and interest from 5-20 people
The facilitator
The data scientist – runs the meetings, provides feedback and generates final forecasts.
4.4 Forecasting by analogy
Comparing something with similar things in the area
Designing a high school curriculum
Structured analogy
- A panel of experts assembled
- Tasks/challenges are set and distributed to experts
- Experts ID and describe as many analogies as they can and generate forecasts based on them
- Compare similarities and differences of each analogy to target situation
- Weighted average created by facilitator
4.5 Scenario Forecasting
Creating scenarios that could occur, allows for a range “best, worst, middle”
Good for contingency planning
4.6 New Product forecasting
Sales force composite
Example is a store and it’s sales team
Having a sales team make their own forecast violates segregation of forecasters and users – A lot of reason to either inflate expectations or lower bar for success
Executive opinion
Generating an aggregate forecast in a group meeting; biases exist from a non-anonymous group meeting. Everything needs to be documented.
Customer intentions
Customer questionaires
Customers may not always do what they say they will, key is to get a measurement right before desired behavior
4.7 Judgmental Adjustments
Use when data is available, has some limitations and bias
Use adjustments sparingly
We need to read what the data is telling us, not try to correct for the way things “Should be” sometimes there’s a reason the data seems odd. Adjusting for lack of events during the Covid crisis is smart, but tossing outlier sales may not be because those outlier sales may come up with some predictable regularity.
Apply a structured approach. Documenting and justifying changes will make it harder to override the norm and to make unnessary adjustments
Most important tasks first, members will get fatigued and make worse decisions as the process wears on.
Example: Tourism forecasting
- TFC forecasts were overly optimistic
- Unbiased statistical models were much more accurate
- Bias seems to have been present, were the forecasters also the users?
4.8 Further Reading