2020 has made a mockery of most predictive models but what can you do when a gobal pandemic wreaks havoc on your forecasts? Do you throw them all away and wait for calmer waters or can they still be put to good use?
We spent a good chunk of last year building a demand modelling tool for social care. Due to the huge variety of things that influence social care activity, we decided against trying to build a hugely complex model that attempted to cater for all factors like population changes, hospital visits, weather, and other things that are harder to quantify like commissioning choices. Instead, we decided to see how far we could get with simple time series forecasting (TSF) approach, takes a statistical approach to projection based on patterns that are picked up from past activity.
We optimised the accuracy as much as possible and started to get some great results, predicting the activity of major care categories to less than 2% variance from the actual one year into the future. Here is a typical forecast from a stable care category (Long-term nursing care for people 65+ across all primary support reasons). The black line is the historic activity which shows a slight seasonal pressure that has been picked up by the forecast which was made at the start of 19/20 and half-way through the year is tracking exactly as expected.
At this point, we turned our attention to the obvious weakness of time-series forecasting; what do we do when a category rebels and unexpectedly takes on an entirely new trajectory?
In some cases, the Local Authority is aware of the cause of the change or even responsible for it (e.g. change in policy/strategy). In others, the change may be happening due to external factors that are largely beyond the authority’s control (e.g. a spike in hospital visits).
Whether the cause of an event is known or unknown, once a forecast proves to be unreliable, the next best thing is to be alerted that the change is happeneing and then to be able to monitor and measure the impact of that change.
This is where time-series forecasting can still be a hugely valuable asset. By capturing and storing a forecast from a previously stable category immediately prior to an event, these values can become an important baseline for measuring the change.
Let’s take a look at the same care category that was happily tracking the forecast throughout 2019 (below). From March 2020 the activity sharply drops which triggers an alert that this category needs some attention.
This is clearly a known event and the forecast at the start of 19/20 holds little value because the actual outturn has radically altered from the prediction. As the situation is still unfolding, it is unrealistic to expect a reliable forecast for 20/21 but if a baseline forecast is taken immediately prior to the initial impact from COVID, it can provide an accurate representation of how the activity would have tracked had COVID not happened. This provides an ideal point of reference for measuring the impact of the event.
It is relatively simple to measure how activity has changed between two points in time (e.g. pre/post COVID), but as the crisis has spanned several months already, it becomes increasingly difficult to measure the specific impact of the event, particularly if the category was steadily increasing or decreasing, or following a seasonal pattern.
We know from normal operating conditions that the baseline forecast had a variance of less than 2% over 12 months so after 6 months it is likely to represent the activity had COVID not happened to within 1%. This gives us a highly accurate point of reference, from which we can measure the change in activity as events unfold. In this case, there are 142 less people receiving long-term Nursing care, which represents a 12% decrease.
Once the impact has been measured, the next question is typically “what might the activity levels look like at the end of the year?”. We can use models that heavily weight recent events to attempt to answer that, although these need to be used with caution as they are much less reliable than the stable long-term models that can be used during business-as-usual scenarios.
Your goal right now isn’t predictions. It’s preparation for what comes next. We must shift our mindset from making predictions to being prepared.
As mentioned earlier, it would be foolish to bet too heavily on these short-term predictions butAmy Webb: How futurist cope with uncertainty
In unprecedented times (even for the word unprecedented!) no one should be gambling on predicted outcomes but that doesn’t mean predictive models can’t still play an important role, and by updating the data weekly, measuring and monitoring the behaviour closely, you can get a deep understanding of changes in key areas to inform a plan -> check -> re-plan cycle from a sound evidence base. This should ultimately be pretty high on anyone’s priority list for crisis management.
I gave a brief demo of this and other tools we have for change management with Birmingham City Council at the UK Authority Digital Health and Social Care 2020 Event