Drift Detection in Banking: What to Monitor in a Live AI Agent

An AI agent changes when its documents, bank policy, tools or provider model version change. Each of these needs a named owner and a defined re-test, and drift detection on inputs and outputs is the signal that tells you when to run it.

This article is for CIOs, COOs, heads of compliance and heads of digital who already have an agent in production, or are about to. It covers what drift means in banking, how data drift differs from concept drift, which detection methods exist, and how to set up a monitoring and re-test routine that your risk and audit functions can follow.

Key Takeaways

• A live agent has four main change sources: the documents it reads, the bank policy it applies, the tools it calls and the model version behind it. Each needs an owner and a re-test.
• Data drift is a change in the inputs. Concept drift is a change in the relationship between inputs and the correct outcome. They need different checks.
• Labels often arrive late, so prediction drift and input drift act as early signals that you confirm later against outcomes.
• Detection alone is not enough. You also need rollback, lineage and a re-test path that can run on the bank’s actual release timeline.
• FINMA’s December 2024 guidance draws attention to monitoring AI risks. It is guidance, not a rule, and it is a sound anchor for your own monitoring design.

Table of Contents

• What drift detection means for ML in banking
• Data drift vs concept drift
• Why undetected drift is a problem for banking decisions
• Drift detection methods banks use
• The four change sources of a live agent
• Monitoring and re-test table
• What FINMA guidance says about monitoring
• A five-step plan to put the routine in place
• Our perspective
• Building monitored agents with S-PRO
• FAQ
• How S-PRO helps banks and financial institutions
• Sources

What drift detection means for ML in banking

Drift detection is the practice of checking, on a schedule, whether the data and behavior of a production model still match what it was validated on. In banking, ongoing monitoring includes strategies for detecting data drift and concept drift that may affect a model’s effectiveness over time.

A credit scoring model, a transaction monitoring model and an LLM-based agent all age in the same way. The world they were tested on moves: customers behave differently, products change, policies are rewritten, upstream systems are updated. Early detection lets a bank recalibrate or retrain models and avoid costly mispredictions.

For an agent the picture is wider than for a classic scoring model. A scoring model mostly sees numeric features. An agent reads documents, follows instructions, calls tools and relies on a language model that the provider can update or retire. Drift detection for an agent therefore covers more than feature distributions. It covers every part that can change without anyone touching the agent’s own code.

Data drift vs concept drift

Data drift is a change in the distribution of the inputs, written P(X). Concept drift is a change in the relationship between inputs and the correct output, written P(y|X). Both are defined in the drift concepts reference of the Frouros library, which also notes that data drift does not guarantee a performance impact, although an impact is highly probable.

A practical way to tell them apart in a bank:

The order of investigation matters. Concept drift is detected by comparing predictions with actual outcomes, after controlling for input distribution changes. Once input drift is ruled out, a drop in metrics such as accuracy, precision, recall or F1 over rolling time windows is a robust indicator of concept drift.

There is also a timing problem. Performance decay is downstream and often hard to measure in production because labels arrive late or never. That is why prediction drift is frequently used as an early-warning proxy until ground truth is available to confirm it.

Why undetected drift is a problem for banking decisions

Undetected drift means a model keeps producing confident outputs that are no longer grounded in current conditions, and nobody notices until a decision has already been taken. The research on bank model validation describes the benefit of catching it early as avoiding costly mispredictions, which is the other side of the same point.

In an agent workflow the consequences are concrete. An onboarding agent that summarizes client documents against an outdated policy produces files that look complete and fail review later. A reporting agent that reads a changed template may extract the wrong field without raising an error. A relationship-manager copilot may keep citing a product sheet that has been withdrawn. None of these failures is loud, and each is the kind of thing that monitoring and re-testing exist to catch.

The cost of slow reaction is also measurable. An empirical study on a real transaction system at a large European bank found that ignoring the delay between drift detection and deployment of a new model can overestimate a detector’s ability to mitigate performance drift by up to 39.86% in that application. The lesson for planning: measure how long your own validation and release path takes, and design the detection and re-test routine around it.

Drift detection methods banks use

The common approach is to compare current data against a reference period with a statistical test, then route the result to a human owner. The methods differ in what they measure and what they miss.

• Hypothesis tests on features. The Kolmogorov-Smirnov test for continuous features and the chi-square test for categorical ones detect significant changes in input distributions versus training data.
• Distribution distance measures. Drift in credit datasets has been traced with the population stability index (PSI), KS, Jensen-Shannon divergence and chi-square tests in research on credit scoring under concept drift. PSI expresses magnitude of change rather than statistical significance.
• Time-sliced profiling. The open-source package popmon profiles tabular data in time slices, compares stability over time and against a reference, and can alert on trends, shifts, outliers and changing correlations.
• Performance tracking. Held-out data or rolling windows of labeled outcomes confirm whether a signal has real impact.
• Output monitoring. Distribution of scores, categories or escalation rates, useful when labels lag.

Two cautions apply to any of these. First, tests need a baseline that reflects normal operation, because detectors can fail to improve, or even worsen, results when no drift is present. Second, alert levels should be set by your own model risk function on your own data and recalibrated before go-live, not copied from a blog or a library default.

The four change sources of a live agent

A live agent changes through four routes that classic drift tooling does not fully cover: its documents, the bank’s policy, its tools and its provider model version. Treat each as a change source with an owner, a log and a re-test.

Documents

If the agent retrieves from a knowledge base, product sheets, procedures or client files, any edit to those sources changes what the agent says. Log document version identifiers with every answer, so a later review can reconstruct what the agent saw.

Bank policy

Policy changes alter the correct answer without touching the inputs, which makes them a direct cause of concept drift. The policy owner, not the engineering team, is the right person to trigger a review.

Tools

Agents call APIs, core system connectors and internal services. A changed field name, a new response format or a modified permission can alter agent behavior silently. The service owner should announce changes, and the agent team should re-run tool-call tests.

Provider model version

The language model behind the agent is a dependency with its own lifecycle. As of 3 October 2026, Anthropic’s model deprecations page commits to at least 60 days’ notice before retirement of publicly released models. The practical step is to test newer models before the retirement date, rather than on it. Cloud platforms such as Amazon Bedrock and Google set their own retirement schedules, so the notice window depends on how you access the model. Check the provider’s current page before you plan, because these dates move.

Monitoring and re-test table

The table below turns the four change sources, plus input behavior, into a routine you can adopt and adapt. It is S-PRO working practice, not a regulatory template, and it deliberately sets no numeric thresholds: those belong to your model risk function.

Three habits make the table work. Keep one regression set that grows with every incident, so each failure becomes a permanent test. Store the model, document and policy versions with each interaction, so an audit question can be answered from the record. And agree in advance who can pause the agent, because a trigger without a decision right is only a notification.

What FINMA guidance says about monitoring

FINMA’s guidance on AI governance draws attention to the need for appropriate identification, assessment, management and monitoring of AI risks. This comes from the FINMA press release of 18 December 2024, which published Guidance 08/2024 on governance and risk management when using artificial intelligence.

The guidance describes what FINMA observed in supervision, and it is guidance, not a rule. In section 2.4 (page 5), it reports that FINMA assessed whether institutions monitor data drift and user overrides. That makes it a useful anchor for the last row of the table above, and a reasonable checklist item when you prepare for a supervisory conversation. The same press release lists the risks in scope: model risks such as robustness, correctness, explainability and bias, data quality, IT and cyber risk, third-party dependencies, and legal and reputational risk. Third-party dependencies are exactly the provider model version row.

A Swiss industry report on moving AI from proof of concept to production adds a practical point. Explainability, documentation and ongoing monitoring raise the bar for approval, and technical readiness covers reproducible pipelines, monitoring, rollback and lineage, together with integration into complex core landscapes. Drift detection that is not connected to rollback and lineage cannot be acted on.

A five-step plan to put the routine in place

Start from the change sources and work toward the alerts, not the other way round. Statistical tests are quick to install, and an owner and a re-test path take longer to agree.

1. List the change sources. For each live agent, name its documents, policies, tools and model version.
2. Assign an owner to each. One person or role per source, with the authority to request a re-test or pause the agent.
3. Build the regression set. Representative cases with expected outcomes, including edge cases from real escalations.
4. Set the reference period and alerting. Choose a baseline of normal operation, run input and output drift tests, and let your model risk function set alert levels on your own data.
5. Rehearse a change. Run a full cycle, from trigger to re-test to release or rollback, and time it. That duration is the figure your detection design has to respect.

If your agent works well in a pilot and the monitoring and re-test path is still undefined, that is the usual gap between pilot and production. Talk to our engineers about your monitoring and re-test setup.

Our perspective

We see drift in an agent as a change-management problem first and a statistics problem second. The tests on inputs and outputs are useful, but most of the surprises we plan for come from a document edit, a policy revision, a tool release or a provider model update.

That is why we start a production design by listing the change sources and their owners. A bank that can answer who owns each one, what is logged and what gets re-tested can add statistical drift detection on top with confidence. A bank that cannot will receive alerts nobody is positioned to act on.

We also think the re-test should be cheap to run. A regression set that grows with each incident, versioned logs and a rehearsed rollback make a model switch a routine release rather than a project. This is what moving from AI pilot to production requires in practice.

The S-PRO team

Building monitored agents with S-PRO

We build AI agents for banks and run them as a service, with monitoring and re-testing designed in from the start. Our AI agents for banks are developed on a platform built for regulated environments, with logging, human-in-the-loop controls and documentation that your risk and audit teams can follow.

If you want to define change sources, owners and a re-test routine for an existing agent, our technology consulting team can run that work with your compliance and model risk functions. See our technology consulting services for how we work.

FAQ

What is the difference between data drift and concept drift?

Data drift is a change in the distribution of model inputs. Concept drift is a change in the relationship between inputs and the correct outcome. Data drift can occur without hurting performance, while concept drift usually shows up as degrading results against confirmed outcomes. Check input drift first, then performance, to tell which one you are seeing.

How often should a bank re-test a live AI agent?

Re-test when a change source changes, and on a regular review cycle that your model risk function sets. Triggers include an edited document, a revised policy, a tool release or a new model version from the provider. Scheduled reviews catch slow input drift that no single event explains.

Does FINMA require drift monitoring for AI?

FINMA Guidance 08/2024 is guidance, not a rule. It draws attention to the need for monitoring of AI risks, and it reports that FINMA assessed whether institutions monitor data drift and user overrides. Treat it as a reference for designing and documenting your monitoring, and discuss specifics with your compliance function.

What should we do when the model provider retires a version?

Test the newer model against your regression set before the retirement date, compare outputs side by side and release with a rollback option. Anthropic’s deprecations page commits to at least 60 days’ notice for publicly released models, as of 3 October 2026. Cloud platforms set their own schedules, so check the one you use.

How S-PRO helps banks and financial institutions

We design, build and run AI agents for banks, wealth managers, asset managers and fintechs, and we build each solution on our own agentic platform, S-PRO Agents, and run it as a service. For monitoring and re-testing, that means we define change sources, owners, logs and regression sets together with your teams, and connect them to your existing infrastructure.

S-PRO is a Certified partner in the Claude Partner Network, and our team includes Claude-certified engineers and architects. Our solutions are designed with EU AI Act, FINMA and DORA requirements in mind. S-PRO is an international Swiss-American firm with offices in Zurich and Salt Lake City, and global delivery. Founded in 2014, with 250+ engineers, we are headquartered in Zurich and hold ISO 27001 and ISO 27701 certifications.

To discuss your agent and the routine around it, book a discovery call with S-PRO.

Sources

• Model Validation Practice in Banking: A Structured Approach (arXiv, version 2)
• Model Validation Practice in Banking: A Structured Approach (arXiv, version 1)
• Frouros drift concepts reference
• ML Monitoring Metrics: Drift, Data Quality, Model Decay
• What Is AI Data Drift? Causes and How to Detect It
• On the Impact of Industrial Delays when Mitigating Distribution Drifts
• Fair and Explainable Credit-Scoring under Concept Drift
• popmon: Analysis Package for Dataset Shift Detection
• Streaming and Concept Drift: A Comparative Study
• Model deprecation notice windows across providers
• FINMA press release, 18 December 2024: guidance on AI governance
• Bridging the AI PoC-Production Gap, Swiss FinTech Innovations report

Last updated 8 November 2026