[Basics] Part 10. Turning a score into a decision: connecting to the business, and communication

This is the last piece.

So far we’ve built a good model (Parts 1 to 4), measured it honestly (Part 5), designed interventions with causation (Part 6), validated it and kept it alive (Parts 7 and 8), and laid the plumbing that makes it actually run (Part 9).

And yet, after all of that, nothing may happen. A model touches reality only when it becomes someone’s decision. The score has to become an approval, a rate, a credit limit. And you have to bring people in other departments along with you.

This piece is about turning a score into a decision, and about conveying the result.

Set the cutoff with expected profit, not accuracy

The first question you hit after building a model is: from what score do we approve?

A common mistake is to settle this with a model metric — picking the point with the highest accuracy, or where F1 peaks. But as we saw in Part 0, costs in credit are asymmetric. Approving one good customer earns a margin; one default costs LGD × EAD. One side is tens of times heavier.

So you set the threshold in money, not in statistics.

expected profit = (1 − PD) × margin − PD × LGD × EAD

You compute this per applicant, and the point where expected profit turns positive is the economic cutoff. Across the portfolio, every move of the cutoff changes who gets approved, and total expected profit traces a hill.

The key here is that the lowest default rate is not the answer. Approve nobody and your default rate is zero, but so is your revenue. This is usually exactly where the risk team and the sales team collide, and talking in the shared unit of expected profit makes that argument far more productive.

It isn’t only approve or decline

A cutoff is the simplest form of decision. In practice there are more levers.

One is varying the terms in proportion to risk. Within the same approval, you price a higher default probability at a higher rate. For that you need the calibrated probability Part 5 insisted on. Ranking alone can’t set a rate; you need the absolute number — “this customer is 3.2 percent” — before a price falls out.

The other is the limit. How much to grant once you approve is a question of revenue, loss, and how much the customer will actually draw. And as Part 6 showed, how defaults change when you raise a limit is a causal question, not a correlational one. Conclude from “customers with higher limits default less” that raising limits lowers defaults, and you have the sign backwards.

Run a swap analysis before changing policy

A new model performing better is not reason enough to switch. Alongside the shadow mode of Part 7, the other common tool is swap analysis.

It’s the work of seeing, in advance, who newly comes in and who drops out when the criteria change. The group newly approved is the swap-in; the group that used to be approved and now isn’t is the swap-out. It’s common for the overall approval rate to barely move while the composition changes completely.

Looking at this first lets you say, not “performance improved by 2 percent,” but “approval rate holds, the default rate drops by several basis points, and approvals fall in one particular segment.” The second version is far more useful for a decision, and it connects directly to the fairness checks of Part 8.

A model flows all the way to the financial statements

Here let me finish something I raised in Part 0 and kept deferring: why one model shakes a whole company.

The PD and LGD a model produces are not used in underwriting and then discarded. They pass into accounting and feed the loan loss provision, and into capital regulation and feed required capital. Touch the model and the quarter’s profit moves, and the capital ratio moves.

This is the answer to the question Part 7 raised about why we validate and document to that extent. A model change is a change to financial figures. Because a data scientist editing code propagates into accounting and regulatory reporting, it has to be slow — and slow is correct.

The language changes with the listener

Even telling the same story about the same model, you have to speak very differently depending on who is listening. This takes up more of the job than you’d expect.

To executives you talk money and risk, not AUC. Not “Gini went up 0.02,” but “we can hold the approval rate and cut annual losses by a few hundred million yen, at the cost of fewer approvals in one segment.” That is the version that lands.

To risk and validation you talk assumptions, limits and stability. Which assumptions you’re standing on, what happens to the conclusion if they wobble, whether it stays stable over time. Those are exactly the questions the independent validation of Part 7 asks.

To compliance you talk explainability and fairness. How adverse-action reasons are produced, how protected attributes and their proxies were checked. The questions from Part 8.

To engineering you talk pipelines and latency. Whether the feature actually exists at serving time, how much lag there is. The subject of Part 9.

One model, four languages. And the person who does this well is the one whose models actually ship.

Overconfidence is itself a model risk

Finally, the most important thing about communication: say the limits honestly.

When you walk in with numbers, you want to speak with confidence. But in this field, overconfidence is a model risk in itself. You estimated on top of an unobserved counterfactual (Parts 0 and 6), the label only settles a year later (Part 0), and the distribution keeps moving (Part 7). Hide that uncertainty and state things flatly, and when it turns out wrong, what you lose isn’t that one model but the credibility of the whole data organization.

“Under these conditions it came out this way, and if this assumption breaks the conclusion changes” is far stronger over time. People come to trust the numbers of someone who narrowed their claims honestly.

After eleven pieces

With that, eleven pieces from Part 0 are done. The opening question was this: why do people who are good at ordinary ML still trip up at least once when they arrive in credit underwriting?

Looking back, the answer was in the structure of the field. We only see the outcomes of those we approved (Part 0), time flows one way (Part 3), costs are asymmetric (Part 0), and the decisions we make are themselves interventions in the outcome (Part 6). So ranking well isn’t enough (Part 5), you have to validate and defend what you built (Part 7), explain why it decided and answer for whether that was fair (Part 8), and keep all of it running consistently on a pipeline (Part 9).

And, as today showed, all of that ends up in someone’s decision and in the company’s financials.

In Part 0 I summarized skill in this field in one line: not a prediction-accuracy contest, but the work of estimating an unobserved counterfactual, in an environment where time flows and costs are asymmetric, in a way that stays explainable and stable. Having written eleven pieces, I still think so. I’d add one thing. Conveying that estimate in a form other people can trust and act on is part of the job too.

The Basics series ends here. The blog doesn’t. From here I’ll build on this foundation with Deep Dive pieces that go hard at a single topic, Paper pieces that put research into the language of practice, and Review pieces where I measure for myself whether a newly released tool is actually useful.

Thank you for staying with a long series.