[Basics] Part 7. Deployment isn't the finish line: validation, governance, and monitoring

Up through Part 6, we’ve talked about building a good model and then designing policy with it. In finance, though, this is where the real work begins.

In Part 0 I mentioned in passing that “a model starts aging the moment it ships,” and that “there’s a reason things are slow.” This piece is where I unpack that reason. In other fields, deploying a model feels like crossing the finish line; in credit, deployment is closer to the starting line. If building it took a few weeks, validating, overseeing, and watching over it afterward takes months — sometimes years.

Here’s what this piece covers: having someone else validate the model independently before deployment, deploying itself carefully, and — after deployment — keeping watch and retraining it once it ages. And, wrapping all of it, documentation and governance.

The builder shouldn’t be the validator

First, a concept: model risk. It’s the umbrella term for the losses, reputational damage, and regulatory exposure that arise when a model is wrong or misused. A single model flows all the way through to provisioning and capital (Part 0), so when it’s wrong, the whole firm wobbles. That’s why finance treats a model as an asset that carries risk, and the framework for managing it is called model risk management (MRM).

There’s one principle of MRM worth remembering: the person who builds the model and the person who validates it must be different people. This is usually called the three lines of defense. The first line is the team that develops and runs the model; the second is validation and risk management, independent of the first; the third is internal audit. “I validated my own model and it came out great” may fly in other fields, but not in finance. The builder has a hard time seeing the weaknesses of their own model. So validation is always done by a different set of eyes.

What validation looks at

When an independent validator receives a model, they look at roughly five things.

First, conceptual soundness. Do the methodology, the assumptions, and the variables make sense in the first place? Even if performance looks good, a relationship that defies common sense — “higher income means higher default risk” — is a coincidence in the data, not a signal you can trust.

Second, quantitative validation. Confirm discrimination, calibration, and stability in numbers — the three axes from Part 5. Is the AUC good but the PD wrong? Does it stay stable as time passes?

Third, sensitivity and stress testing. Does the score stay put when you nudge the inputs a little, and how does the model behave under a bad scenario like a recession? A model that only fits well in good times collapses exactly when you need it.

Fourth, benchmarking. Does this complicated model actually improve on a simpler one, or on the model already in place? If it’s complex and doesn’t improve much, that complexity is just risk.

Fifth, reproducibility. Run it again on the same code and data — do you get the same result? A model you can’t reproduce can’t be validated or audited.

You don’t switch it on all at once: shadow mode

Passing validation doesn’t earn a model an immediate spot in production, either. The standard safeguard when introducing a new credit model is shadow mode.

Shadow mode means running the new model alongside the live one, computing its scores without letting them drive any real decision. You approve and reject with the existing method, while recording — for months — what the new model would have decided. Only after you’ve confirmed the new model runs stably and doesn’t diverge wildly from the existing results do you actually switch over.

In a similar spirit, you might compare a champion and a challenger in parallel, or do a gradual rollout that turns the model on for a subset of customers first. Either way, there’s one thing in common: a rollback plan laid down in advance so you can undo it the instant something goes wrong. The principle isn’t “switch on the high-performing model fast” — it’s “switch on safely, even if it’s wrong.”

Models age: monitoring and retraining

After deployment it isn’t over — the longest stretch is only just beginning. It’s the aging I mentioned in Part 0. While a model trained on 2024 data scores 2026 customers, the economy, rates, and customer behavior all change. The distribution shifts, and the model slowly ages.

So after deployment you keep watching from several angles. First, how far the inputs have drifted from training time, with PSI and CSI — the stability metrics from Part 5. You track performance too, of course. But there’s a difficulty peculiar to credit here. Since a default label is only settled 12 to 24 months later (Part 0), you can’t see performance decay in real time. So you run backtests — matching predictions against actuals — belatedly, as labels accumulate, and in the meantime you catch warning signs early with leading indicators like PSI. You also watch whether the predicted default rate matches the actual, whether one particular segment is drifting badly, and whether the data pipeline has quietly broken.

And once you judge the model has aged, you retrain it. When to retrain — the trigger — is decided in advance too: when the distribution has moved a lot, when performance has dropped, when the policy changed, when a new product launched. The point is not to retrain on a whim, but to define the cadence and the procedure ahead of time and follow that governance. Because changing a model is itself a risk.

The documentation is the model

Finally, it’s about leaving a written record of everything about the model. This part is surprisingly the crux.

Every model comes with development documentation: what it’s for, what data and what methods and assumptions built it, what its limits are, and what the validation results and monitoring plan look like. To that you add an audit trail recording when what changed, plus version control and change management.

Here’s the crux: documentation isn’t cleanup you do after the model is finished — it’s part of the development process. When the regulator asks “why this model?”, when internal audit asks “what’s the basis for this decision?”, the documentation is the answer. A model you can’t explain on paper is unusable in finance, no matter how well it performs.

So, there’s a reason things are slow

In Part 0 I said the startup reflex of “ship it fast” doesn’t travel well here, that there’s a reason things are slow. This piece has been the explanation of that reason. Someone other than the builder validates; you switch on carefully; you watch it age and retrain; and you leave all of it in writing. It looks like a hassle, but it isn’t a burden regulation forced on you — it’s the infrastructure of trust. Because a single model judges one person’s credit and can shake the firm’s capital.

Predicting well is the entry ticket (Part 5); handling causality is skill (Part 6); and honestly validating the model you built and keeping it alive for the long haul — that is trust. Next time, I’ll continue to the other axis of that trust: interpretability, the demand that you be able to explain why the model decided what it did, along with fairness and regulation.