
Don’t let your shiny AI models lose their luster!
Posted: 2020-04-22
Businesses have made significant investments in building AI/ML models in recent times. While models are increasingly driving significant operational efficiencies and differentiation for businesses, they are often blamed when they fail to predict accurately in extreme events such as COVID-19. In reality, model decay is inevitable: markets are changing quite fast, and trends are short-lived. Black Swan events can create a ‘new normal’, invalidating the history on which models are trained. Organizations typically have an inventory of several hundred models; is it possible to monitor decay proactively and manage interdependencies in a rapidly changing environment?
Once a model has been formally put into use, model decay can begin to occur due to increased uncertainty over time. Data Science teams will have to ensure that continuous monitoring is in place, performance thresholds are set and triggers are defined to ensure that areas of potential risks are highlighted, mitigated, and escalated as necessary. Organizations typically conduct model monitoring on a quarterly/half-yearly/yearly basis based on the material impact. Since drift can occur anytime (due to data and/or model), continuously monitoring and identifying drift proactively can avoid missed opportunities, mitigate loss and enhance trust in the models used. Models are recalibrated when the drift is beyond set tolerance levels. There is no one model that is right for every situation; organizations often have both champion and challenger models, and continuously track their performance and divert traffic to the model that performs best. Challenger models can be deployed in shadow mode to avoid any negative impact on the end-user experience.
The stability of the population and its underlying characteristics must be reviewed when the model is executed in production to avoid data drift, which can be measured by metrics like Population Stability Index (PSI)/Characteristic Stability Index (CSI) and KS statistic. PSI/CSI help to monitor if there is a shift in the population between development and production data. KS statistic is used to assess the predictive capability and performance of the model.
Since trends and relations that exist in the historical data are used to train models, models become obsolete when trends become obsolete or historical correlations are broken. These changes can be gradual (change over time), recurring (cyclical change), or sudden (abrupt change). Model monitoring must be intelligent enough to identify the root cause of the model drift proactively and provide feedback. Dynamic ensemble selection (DES) determines which combination of the model’s combined result outperforms as compared to using a single model for prediction accuracy. Ensemble models can capture linear and non-linear relationships in the data. In a complex, unseen situation, depending on a single model is like touching and trying to identify an unknown (elephant) in a dark room.
Given that model decay is inevitable, monitoring models in isolation (silo manner) is time-consuming, resource-intensive and inefficient. Organizations must have centralized telemetry that monitors models, captures dependencies before model performance starts deteriorating, and reports in a timely manner. This contributes operational efficiency and scale, especially for organizations that have or expect to have hundreds of models. Data pipelines must support sustainable Continuous Integration (CI)/Continuous Deployment (CD) for speed, agility and repeatability to drive business value.
Tracking data and concept drift is extremely important to ensure that model monitoring is effective and accurately accounts for unprecedented events. Here are some best practices that companies can adopt for effective model monitoring:
- Centralize model monitoring with continuous tracking at a high-frequency basis (typically daily) in a systematic approach across the institution. This must be applied for all models in the inventory—recalibrating models based on annual model review or validation is too late.
- Establish ongoing monitoring through an automated process, such as a global monitoring dashboard with drill-down capability into any individual model, that can detect any unexpected behavior and communicate to the stakeholders. This can provide additional insights on correlations between model behavior.
- Continuously evaluate champion and challenger models and choose the model (or ensemble of models) dynamically that performs best based on the context.
- Identify early warnings in the model’s performance proactively, rather than reacting to issues that have already arisen and may be costly to remediate.
- Develop an aggregate metric that quantifies the overall model risk of the institution at any point in time and ensure that it’s within the model risk appetite set by the board of directors.
- Retrain models with the most recent data available to capture the trends; a first-in first-out (FIFO) data structure can capture the trend changes over time through adaptive learning. Data science pipelines have to be scalable to support CI/CD.
- Keep humans in the loop and employ expert judgment to ensure the results make sense.
For ongoing successful usage of the models, continuous monitoring of model performance and understanding of the context are critical. With these best practices, effectiveness and credibility will be enhanced at all times—even in uncertain times like these.
This post is a sequel to my previous article, “AI Modeling in the time of COVID-19.” Any comments and thoughts are welcome!
About the Author:
Raj Gangavarapu is Head of Data Science at diwo, your intelligent advisor to turn AI into action. He has two decades of leadership experience helping companies to solve complex business problems by leveraging data and analytics. He is a speaker at various academic and industry conferences on data science, risk, Artificial Intelligence (AI) and Machine Learning (ML).
Image source: Himmelfarb et al 2002: 1526 (artist: G. Renee Guzlas)

How banks can leverage AI to better manage pandemic relief programs
The recent massive economic shifts due to COVID-19 have driven up risk for the vast majority of financial services institutions. Customer sentiment, spending and behavior... + 2020-07-28


Financial Services: Preparing for the Looming Credit Crisis
The COVID-19 pandemic has significantly affected financial institutions – slowing the growth of loan originations, increasing credit costs, contracting economic activity, and causing record levels... + 2020-05-15


Can retailers find strategies that balance short-term recovery with long-term sustainability to lead them out of the COVID-19 crisis?
Retail and fashion business leaders are currently focusing on business continuity (“keep the lights on”) and crisis management (“sell the right stuff”), but soon they... + 2020-05-05


Transforming Business Decisioning in the Pandemic era is more crucial than ever.
Our human tendency toward incremental thinking limited us from foreseeing how “a few cases of the flu” would balloon into the impact we see today.... + 2020-05-01


Surviving COVID-19: How can Retailers free up working capital tied-up in inventory?
Among the many sectors affected by COVID-19, retail will be one of the hardest hit. The sudden decline is attributed to country-wide store shut-downs and... + 2020-04-23


Don’t let your shiny AI models lose their luster!
Businesses have made significant investments in building AI/ML models in recent times. While models are increasingly driving significant operational efficiencies and differentiation for businesses, they... + 2020-04-22


AI Modeling in the time of Covid-19
Companies of all sizes are facing unprecedented uncertainty and challenges due to the global impacts of COVID-19. It has created a major systemic shock to... + 2020-04-14


How to derive value from your AI investments with Decision Intelligence
Ultimately, a company’s value is measured by the sum of its decisions. In order to succeed, the organization must make and execute decisions—across all levels... + 2020-01-24


Why isn’t my enterprise getting value from AI at scale?
Why do most AI initiatives for business fizzle out? Why do so many teams’ best efforts to develop or deploy new algorithms or predictive models... + 2019-11-21


Here’s why you’re probably losing the AI race
As the AI arms race is becoming more heated, more organizations are looking to beef up their competitive advantage... + 2019-08-01


Data In Wisdom Out
I am often asked by the curious, what do we actually mean by “wisdom out”. Is it just a marketing ploy or is there something more to it?... + 2018-10-31


Is Data a source of value?
We have always been exposed to natural and man-made events and have wondered and been impacted by their outcomes... + 2018-10-16


The Future of Decision Making: Human-AI Symbiosis
When we have to make an important decision, we face numerous challenges: uncertainty, complex data that’s difficult to interpret, competing priorities... + 2018-09-20


Digital Transformation- a $900B failure this year alone
Digital transformation has become a major priority for most organizations in some form or another, but for many, it’s proving to be quite the challenge... + 2018-09-20


So What’s AI’s Dirty Little Secret?
One could read the massively hyped claims of the trillions in productivity gains and then how far AI still has to go for real-world application... + 2018-09-20


How AI can rescue your BI “situation”
Even with a few credible upstarts in the past couple years, Self-Service BI still seems to be dominated by some large players that also require some very large and "very ongoing" commitments... + 2018-03-20


Business First! – how diwo aspires to flatten the knowledge pyramid
We can all agree on the fact that we are sitting on an unprecedented volume of data, and it is continuing to accumulate exponentially... + 2017-09-18


The Growing Market Impact Of AI
The future is always uncharted territory, and in the hype that currently sorrounds AI, with its ambiguous... + 2017-09-18


AI: How It Will Redefine My Job?
In the past few years, there has been an explosion in innovation and interest around Artificial Intelligence... + 2017-09-18


Wait Less, Sense More, Act Fast!
“If I had more legs, I could walk faster. If I had more hands, I could handle more, and if I had extra senses, I could respond wiser” – did you ever wish that?... + 2017-09-11


Unlocking the holy grail of UX design: bias-free user feedback in real time!
The potential of measuring emotional feedback from users is an exciting proposition in many fields, not least because it offers the possibility of unlocking... + 2017-07-25


What’s the Deal With AI Connectors
Artificial Intelligence (AI) is changing our lives, and these days feels more omnipresent than ever before. From Siri to autonomous cars... + 2017-07-25


Business Analytics-Why Search is Not Enough
Due to the popularity of internet searches, many businesses have attempted to adapt the search paradigm to tame their own unruly document clutter... + 2016-09-15


Drowning in data? Still no excuse for inaction!
Wouldn’t it be fantastic if we had a total understanding of the laws governing the reality around us? Imagine for a... + 2016-09-15
