[Basics] Part 6. Beyond prediction, to intervention: causal inference and experiments
Credit underwriting doesn't just predict outcomes, it intervenes in them. Ask a question like 'does raising a limit increase defaults?' with correlation and you get the sign wrong before you even start. Here is credit's causal inference put in practitioner terms, from experiments and quasi-experiments to regression discontinuity and uplift.
So far this series has mostly been about getting “who is risky” right. Building good features, scoring with trees, checking whether those scores are well calibrated. But in practice the harder question, the one with real money riding on it, is a little different.
If we raise a limit, will defaults go up? If we chase a delinquent customer, will we recover more? If we cut the rate, will fewer people churn?
These aren’t prediction questions, they’re causal ones. In Part 0 I described the core of this field as “estimating the unobserved counterfactual,” and this is exactly where that counterfactual walks in through the front door. And the moment you can answer these questions, a credit data scientist graduates from “the person who guesses who’s risky” into “the person who designs policy.”
Prediction and causation are different questions
Prediction asks: what is this customer’s probability of default? Causation asks: if we raise this customer’s limit, how much does that probability change? These are completely different questions, and the ways you answer them are different too.
The fundamental reason causation is hard is that you can never see two outcomes for the same person at once. What would have happened to this customer, whose limit we raised, if we hadn’t raised it is something we will never observe. We only ever see half of the world. This is the general form of the story from Part 0, where I said we never learn how the customers we rejected would have turned out.
The catch is that in credit, the decisions we make are themselves interventions in the outcome. Whether we approve, how large a limit we grant, what rate we set all directly affect that customer’s default. So the data in our hands isn’t a world we sat back and observed, it’s a world we already shaped by intervening in it.
Here’s where a common trap appears. The data shows that customers with higher limits default less. So if we raise limits, will defaults fall? No. We simply handed the higher limits to customers who already looked creditworthy. The limit didn’t lower defaults; a hidden cause, good creditworthiness, produced both the high limit and the low default at the same time. A hidden cause that moves two outcomes together like this is called a confounder. (I checked this directly with public data in the Deep Dive piece.)
Mistake a correlation for causation when you set policy, and no amount of data will keep you from getting the direction wrong. This is why in credit, causal inference isn’t optional, it’s mandatory.
The gold standard is an experiment
The way to turn a counterfactual from a guess into a fact is actually simple: randomize the intervention. Split the target customers at random into two groups, raise the limit for one and leave the other alone. Because the split is random, the two groups have the same creditworthiness and the same income on average. So whatever difference opens up later in default rate or profit is due to the limit and nothing else. This is a randomized controlled trial (RCT), and it’s the gold standard of causal inference.
Credit has one special form of this experiment: deliberately approving, at random, a small share of applicants you would otherwise have rejected. In Part 0 and the rejectkit piece I said we can never learn how rejected customers actually turn out, and this is the one way to turn that counterfactual into a fact. Reject inference guesses at that blank space with assumptions; an experiment just lets you observe the answer. That’s why I said the most trustworthy solution to reject inference is, in the end, a small randomized approval.
It isn’t free, of course. Approving bad customers on purpose is a loss, and a randomized limit increase carries risk too. So you don’t swing big, you run experiments small, with a controlled design. How large a sample you need to detect a difference statistically, whether to randomize at the customer level or the branch level, where the loss has to cross before you stop, and even the lag from default labels only settling 12 to 24 months later all have to be set before you begin.
When you can’t experiment: quasi-experiments
You can’t always run an experiment. Often it’s too slow, too costly, or ethically or legally out of bounds. When that’s the case, you approximate causation as best you can from observational data you’ve already accumulated. This is called a quasi-experiment.
First, propensity scores and inverse-probability weighting (IPW). This is a way of making the treated group and the untreated group resemble each other on their observed characteristics. It’s the same logic as the bias correction from Part 2, or the reweighting in rejectkit. If you gave larger limits to customers with good creditworthiness, in effect you pair up and compare people of similar creditworthiness.
Second, regression discontinuity (RDD). This one fits credit especially beautifully. Underwriting has a cutoff score. If 630 is the line, an applicant at 631 and one at 629 are essentially the same person, yet only one gets approved. If the outcome jumps sharply across that boundary, that jump is the causal effect of the intervention that is approval. The underwriting cutoff becomes, in itself, a natural experiment.
Beyond these, the toolbox holds more: difference-in-differences (DiD), which compares before and after a policy change, and instrumental variables (IV), which use an outside factor that only affects the intervention as a lever. Quasi-experiments all lean, though, on an assumption of the form “there is no unobserved confounding.” So the order is clear. Experiment when you can; when you can’t, use a quasi-experiment but always state which assumptions you’re standing on. And the honest move is to check, with a sensitivity analysis, whether the conclusion flips when those assumptions wobble a little.
Python has a few tools that help with this process. DoWhy lets you treat causation as a flow of “state your assumptions, estimate, and then refute by testing how much the result shifts if those assumptions are wrong.” It essentially has you write your assumptions down in code and go knock on them yourself, which pairs well with the sensitivity check I just mentioned. EconML collects the latest methods for estimating treatment effects, especially the heterogeneous treatment effect (HTE) that varies from person to person (uplift, just below), such as double machine learning and Causal Forest. CausalML specializes in uplift modeling and targeting, handling everything from meta-learners to the Qini evaluation that comes later, in one place.
Beyond the average: uplift
So far we’ve been asking whether raising a limit increases profit on average. But the real question in practice goes one step further: for whom should we raise it?
Raising a limit for a customer whose profit won’t grow either way just adds risk, which is pure waste. Conversely, raising it for a strong customer who couldn’t spend for lack of headroom generates incremental profit. Estimating, at the individual level, this extra difference an intervention creates for a given person is uplift modeling. It’s not the same as simply predicting who will respond. You’re looking not for the person who would have acted anyway, intervention or not, but for the person the intervention actually changes.
The use in credit is clear. Raise the limit only for customers where the increase creates incremental profit, and intervene only on the delinquents where a chase actually lifts recovery. As for techniques, there are things like the T-learner, the X-learner, Causal Forest, and uplift trees.
One word of caution. You must not evaluate uplift with ordinary AUC. Since what we’re trying to get right isn’t whether someone defaults but the effect of the intervention, you have to look at dedicated metrics like the Qini curve or Uplift@k.
Here’s the difference. AUC measures how well you rank whether this person will default, or respond. But what uplift wants is a ranking of how much the intervention changes this person, so what’s being measured is different from the start. On top of that, since you can’t see both the intervened and the un-intervened outcome for one person at once, there is no per-person ground truth at all, no answer key to grade one person at a time the way an ordinary metric does. So the Qini curve detours around this by working with groups. As you intervene on customers starting from the ones the model scored highest, it plots, cumulatively, how much faster the outcome gap between the treatment and control groups piles up than it would if you’d chosen at random. Uplift@k is the incremental gain piled up by the time you’ve reached the top k%. In a word, where AUC measures “how well do I guess who will default,” Qini measures “when I intervened in the order I chose, how much more did I actually make than intervening on no one at all.”
In Part 5 I said that when the objective differs the metric has to differ too, and this is where that principle applies most dramatically.
So, what expertise in this field really means
I’ll say again that prediction is the price of entry and causation is the expertise. Every decision we make in underwriting is an act of intervening in the outcome, and only when you can estimate that effect honestly can you design policy.
To lay out the order: experiment when you can, even in a small way. When you can’t, approximate with a quasi-experiment but state which assumptions you’re standing on. Credit in particular has a natural experiment in the underwriting cutoff, so don’t leave regression discontinuity going to waste. And beyond the average effect, use uplift to choose whom to intervene on.
This way of thinking is the root solution to reject inference, and the foundation for limit policy, collection strategy, and pricing. In the next part I’ll carry this into governance and monitoring: how you validate, audit, and operate the models and policies you’ve built this way.
The record of debiasing the limit-causation question by hand on public data is in the Deep Dive listed under Related below, so if you’d like to see the concept confirmed hands-on, that’s a good one to read alongside this.