AI Modeling in the time of Covid-19
Companies of all sizes are facing unprecedented uncertainty and challenges due to the global impacts of COVID-19. It has created a major systemic shock to the economy: steep market declines, substantial increase in the volatility of all major markets, federal interest rate cuts and government stimulus, and record levels of unemployment claims. Supply chains and distribution networks have been significantly disrupted, as employees and customers are forced to stay at home.
In a time with such radical upheaval, are current forecasting and customer behavior models still accurate? Let’s examine the effectiveness of the models, determine whether they can account for the new normal, and embrace the best practices to improve the accuracy of prediction.
Models are supposed to work at all points throughout the business cycle—not only in good times, but also in a crisis. Unfortunately, this did not happen during the financial crisis of 2007-08.
One model that was widely used to price and manage the risk of Collateralized Debt Obligations (CDOs) and other securitization products was known as the Gaussian copula function. While this model is mathematically “beautiful,” it’s fatally ﬂawed, and the failure to heed warnings about its limitations contributed to a colossal bubble in the securitization market in 2007-08, leading to losses of trillions of dollars.
Regarding subprime loans—which were at the epicenter of the crisis—only a few years of data were available, and there weren’t sufficient data quality controls in place (an unfortunate example of “garbage in, garbage out”). In general, models were not sufficiently tested through-the-cycle. After the crisis, policymakers and regulators pursued a series of regulatory reforms aimed at managing model risk and forecasting losses in stressful times, increasing required levels of equity capital in order to enhance the loss-absorbing capacity of the banks.
Companies need models that look back far enough to include historical catastrophic events (which may influence risk posture, liquidity, supply chain, or customer behavior) and that capture the current and anticipated impacts on their business caused by company-specific, industry-specific, and macro-economic factors. Models that don’t capture the exogenous factors with a sufficient level of granularity will fail to predict the forecast accurately. For companies that don’t have sufficient internal data encompassing several business cycles, historical shocks can be simulated to represent the current situation leveraging relevant external datasets (from third-party vendor/data consortiums/economic data provided by regulators, for example FRED/GeoFRED, ABI, ORX, ORIC International etc.) and models can be retrained. Capturing the dependence relationships with greater precision, taking into account idiosyncrasies of multivariate distributions (especially with respect to the behavior in the extreme tails), helps to improve the model’s performance. The best models incorporate as much history as possible, while also appropriately weighting more recent history to capture current trends.
While extreme events such as COVID-19 are rare and unpredictable, they still play a critical role in forecasting and customer behavior models. Typically, models fail to adequately capture the risk of extreme events; humans tend to focus on what is reasonably possible at the expense of what is remotely probable (i.e., tail distribution). Training and test datasets containing (for example) 99% positive and only 1% adverse events can lead to data imbalance; since tail events are sparse, data samples must be sufficiently representative using data augmentation techniques to minimize this bias.
Thus, accurate characterization of extreme events is critical for understanding the trends and potential impact of such events. AI/ML models may need enhancements to account for the possibility of significant change and avoid any potential algorithm bias. Extreme value distribution can be embedded within the modeling framework for detecting Black Swan events—such as COVID-19—to capture the tail distribution of observations.
How then can we ensure our models accurately account for unprecedented events? Here are some best practices that companies can adopt for models that work—even in times of crisis:
- Maintain a library of forward-looking scenarios based on historical and hypothetical events that are exceptional but plausible to predict the revenue, losses, delinquency, spend behavior of the borrowers, etc.
- Ensure these scenarios reflect macroeconomic/financial conditions that are tailored specifically to stress the organization’s key vulnerabilities and idiosyncratic risks.
- Retrain models with the most recent data available to capture the trends.
- Adjust assumptions to the specific scenarios; model assumptions should be reasonable, supportable, reviewed at regular intervals, and documented in a well-thought-out process.
- Identify the data drift, model decay proactively, and recalibrate models in a timely manner.
- Conduct frequent outcome analysis and back-testing, as/when the ground truth is available.
- Keep the humans in the loop, adjusting the model-based estimates based on careful review and expert judgment to ensure the results make sense.
For ongoing successful usage of the models, organizations must continue to rigorously assess data, maintain conceptual soundness of the methodology, continuously monitor model performance, and ensure a strong understanding of the context. With these best practices, effectiveness and credibility will be enhanced, creating models that work—even in uncertain times like these.
About the Author:
Raj Gangavarapu is Head of Data Science at diwo, your intelligent advisor to turn AI into action. He has two decades of leadership experience helping companies to solve complex business problems by leveraging data and analytics. He is a speaker at various academic and industry conferences on data science, risk, Artificial Intelligence (AI) and Machine Learning (ML).
Why AI Models Eat BI Dashboards for Lunch
Many organizations devote precious time and resources developing predictive AI/ML models to determine what might happen in the future. The ultimate goal of these models... + 2021-03-26
Why More BI Dashboards Don’t Mean Better Decisions
A typical analytics dashboard, whether it’s viewed as a pivot table in Excel or within a business intelligence (BI) platform, is often the primary source... + 2021-03-17
How banks can leverage AI to better manage pandemic relief programs
The recent massive economic shifts due to COVID-19 have driven up risk for the vast majority of financial services institutions. Customer sentiment, spending and behavior... + 2020-07-28
Financial Services: Preparing for the Looming Credit Crisis
The COVID-19 pandemic has significantly affected financial institutions – slowing the growth of loan originations, increasing credit costs, contracting economic activity, and causing record levels... + 2020-05-15
Can retailers find strategies that balance short-term recovery with long-term sustainability to lead them out of the COVID-19 crisis?
Retail and fashion business leaders are currently focusing on business continuity (“keep the lights on”) and crisis management (“sell the right stuff”), but soon they... + 2020-05-05
Transforming Business Decisioning in the Pandemic era is more crucial than ever.
Our human tendency toward incremental thinking limited us from foreseeing how “a few cases of the flu” would balloon into the impact we see today.... + 2020-05-01
Surviving COVID-19: How can Retailers free up working capital tied-up in inventory?
Among the many sectors affected by COVID-19, retail will be one of the hardest hit. The sudden decline is attributed to country-wide store shut-downs and... + 2020-04-23
Don’t let your shiny AI models lose their luster!
Businesses have made significant investments in building AI/ML models in recent times. While models are increasingly driving significant operational efficiencies and differentiation for businesses, they... + 2020-04-22
AI Modeling in the time of Covid-19
Companies of all sizes are facing unprecedented uncertainty and challenges due to the global impacts of COVID-19. It has created a major systemic shock to... + 2020-04-14
How to derive value from your AI investments with Decision Intelligence
Ultimately, a company’s value is measured by the sum of its decisions. In order to succeed, the organization must make and execute decisions—across all levels... + 2020-01-24
Why isn’t my enterprise getting value from AI at scale?
Why do most AI initiatives for business fizzle out? Why do so many teams’ best efforts to develop or deploy new algorithms or predictive models... + 2019-11-21
Here’s why you’re probably losing the AI race
As the AI arms race is becoming more heated, more organizations are looking to beef up their competitive advantage... + 2019-08-01
Data In Wisdom Out
I am often asked by the curious, what do we actually mean by “wisdom out”. Is it just a marketing ploy or is there something more to it?... + 2018-10-31
Is Data a source of value?
We have always been exposed to natural and man-made events and have wondered and been impacted by their outcomes... + 2018-10-16
The Future of Decision Making: Human-AI Symbiosis
When we have to make an important decision, we face numerous challenges: uncertainty, complex data that’s difficult to interpret, competing priorities... + 2018-09-20
Digital Transformation- a $900B failure this year alone
Digital transformation has become a major priority for most organizations in some form or another, but for many, it’s proving to be quite the challenge... + 2018-09-20
So What’s AI’s Dirty Little Secret?
One could read the massively hyped claims of the trillions in productivity gains and then how far AI still has to go for real-world application... + 2018-09-20
How AI can rescue your BI “situation”
Even with a few credible upstarts in the past couple years, Self-Service BI still seems to be dominated by some large players that also require some very large and "very ongoing" commitments... + 2018-03-20
Business First! – how diwo aspires to flatten the knowledge pyramid
We can all agree on the fact that we are sitting on an unprecedented volume of data, and it is continuing to accumulate exponentially... + 2017-09-18
The Growing Market Impact Of AI
The future is always uncharted territory, and in the hype that currently sorrounds AI, with its ambiguous... + 2017-09-18
AI: How It Will Redefine My Job?
In the past few years, there has been an explosion in innovation and interest around Artificial Intelligence... + 2017-09-18
Wait Less, Sense More, Act Fast!
“If I had more legs, I could walk faster. If I had more hands, I could handle more, and if I had extra senses, I could respond wiser” – did you ever wish that?... + 2017-09-11
Unlocking the holy grail of UX design: bias-free user feedback in real time!
The potential of measuring emotional feedback from users is an exciting proposition in many fields, not least because it offers the possibility of unlocking... + 2017-07-25
What’s the Deal With AI Connectors
Artificial Intelligence (AI) is changing our lives, and these days feels more omnipresent than ever before. From Siri to autonomous cars... + 2017-07-25
Business Analytics-Why Search is Not Enough
Due to the popularity of internet searches, many businesses have attempted to adapt the search paradigm to tame their own unruly document clutter... + 2016-09-15
Drowning in data? Still no excuse for inaction!
Wouldn’t it be fantastic if we had a total understanding of the laws governing the reality around us? Imagine for a... + 2016-09-15