A biased loan approval model doesn’t just cut into lender profits. It skews the incentives that govern access to capital: who gets to start a business, who gets to refinance out of trouble, whose default history becomes the training data for the next model. And the uncomfortable part is that the bias is largely unmeasurable from inside the system that produces it. You cannot ask historical data what would have happened to the applicants you declined, and you cannot ask it whether your model’s counterfactual reasoning — if this borrower had linked a bank feed, what would their default probability be? — is right or wrong. The label you would need to check the claim was never generated.
I think this is where AI tooling can genuinely level the playing field. Not by promising an unbiased model — nobody can deliver that — but by making the failure modes of the modeling pipeline measurable and falsifiable: demonstrating precisely where a lender’s causal claims stop being trustworthy, and ruling out the scenarios the data cannot rule out on its own.
Default modeling looks like one problem and is actually two. The first is familiar: mapping historical applications to repayment outcomes, a supervised learning task any gradient-boosted tree handles competently. The second, more consequential one is causal attribution — answering intervention questions the historical record was never designed to answer. A lender who wants to know whether requiring linked bank feeds would reduce defaults is not asking a prediction question. They are asking what happens under a policy that has never been run.
The two problems are entangled by a structural feature of lending data known in the literature as selective labels: you only observe repayment outcomes for the loans you approved. The declined majority contributes features but no labels, and the approval policy that split the population was itself driven by the very features you now want to reason about. Every model trained on funded loans inherits this selection, and every counterfactual claim it makes propagates it.
Recent history has made it worse. Much of the data available for training was generated in a historically low-rate regime, and the macroeconomic uncertainty since then puts that data’s validity in question — not because the features changed, but because the mechanisms connecting features to default plausibly changed. A model can be perfectly calibrated on the world that produced its training set and confidently wrong about the world it is deployed into.
So the practitioner faces a compounding failure: the quantity they care about — interventional effects on default probability — is unidentifiable from their data, and the distribution their data represents may no longer exist. The standard toolkit answers neither complaint. Cross-validation validates predictions on the funded, historical world. It says nothing about counterfactual accuracy, and nothing about the declined pool.
Which raises the question this project set out to answer: how do you make a synthetic data generation-and-evaluation loop robust — statistically and as engineering? The engineering half has a lower bar than it once did; the libraries are largely available, and everything here runs on scikit-learn, pinned and deterministic. The statistical half is where the real work lives, because you need to add to that library the one thing no library ships: a world where the ground truth of every counterfactual is known, so that estimators can finally be graded rather than trusted.
The approach is a closed-loop harness built around a structural causal model of the borrower population. The SCM is a set of fitted mechanisms — each feature generated as a function of its parents plus noise — validated against real small-business lending data through a marginal-fidelity gate covering base rates, missingness, per-feature quantiles, and categorical frequencies. That gate checks univariate marginals only, and the harness says so: “fidelity passed” means the modeled marginals match the real data, not that the joint or causal structure is faithful. The claim is scoped to what was checked.
Because the harness owns the mechanisms, an intervention is surgery, not conditioning. do(feature = value) deletes the mechanism that normally generates that feature, clamps it, and lets every downstream mechanism respond. Setting a borrower’s bank-feed linkage isn’t overwriting a boolean — it structurally reveals or hides the entire block of cash-flow features that only exist when a feed is linked. The SCM’s post-surgery default probability is the true interventional answer, computable exactly because the world is ours.
Then the harness hides that truth the way reality does. A realistic approval policy funds the top-ranked applicants; labels accrue only for the funded; and a tunable severity knob routes selection through a confounder the downstream estimators are not allowed to observe. The result is a family of worlds that reproduce the selective-labels pathology on demand, at controlled intensity, with the answer key locked in a drawer.
This is also where the earlier claim — that the engineering half has a lower bar — gets cashed out. The whole loop is pip-installable, scikit-learn-only, and byte-deterministic per seed:
pip install closed-loop-default-detection
python
from cldd import SelectiveLabelsLoop
result = SelectiveLabelsLoop(improve_mode="both").run() # "reweight" | "retrain" | "both"print("Operating frontier:", result.frontier_severity)for r in result.rounds: print(r.selection_severity, r.naive.declined_ece, r.passed)
The loop generates a cohort at a given severity, trains on approved rows only, scores against planted truth on the declined subpopulation, applies a correction lever, and escalates — until correction fails. frontier_severity is the deliverable: the last severity at which declined-cohort calibration still holds.
Against this planted truth, the harness grades three approaches. Naive conditioning overwrites the feature column on a model fit to approved rows and re-predicts — ignoring downstream mechanisms, inheriting the selection bias, and closely resembling what a scored what-if analysis in production does today. G-computation fits each child mechanism from observed data using only the causal graph’s topology, with no access to the SCM’s coefficients, clamps the target, and propagates the change through the fitted mechanisms. And for the separate question of calibration on the declined cohort, IPW reweighting weights funded observations by inverse approval propensity.
A fourth path calls the SCM directly and scores near zero error by construction. It is labeled a truth reference and never presented as an estimator — a distinction the harness enforces structurally, for reasons that will become concrete shortly.
Three findings, in increasing order of how much I would stake on them.
First: the deployable causal estimator wins where interventions propagate — and the win is small. At moderate selection severity, g-computation beats naive conditioning on counterfactual mean absolute error, with the advantage concentrated exactly where theory predicts: queries whose effects propagate through downstream mechanisms. Across the full 25-seed sweep at severity 0.4, g-computation cuts strong-propagation counterfactual MAE from 0.099 to 0.086 — a 13.5% reduction, positive on 24 of 25 seeds. One disclosure belongs next to that number: the strong-propagation subset is defined from the answer key — only a graded world can label which queries propagate, so no practitioner could construct this subset from observable data. On the published reference seed, the overall MAE gap is +0.0029, about 3%, against +0.0079 on the strong-propagation subset; at maximal severity the overall advantage collapses to +0.0003 (strong-propagation: +0.0017). That is the honest size of the effect. A deployable method is measurably less wrong where propagation matters; it is not a decisive victory, and this essay would be worthless if it pretended otherwise.
Second: declined-cohort calibration holds to a frontier, then breaks. IPW reweighting keeps expected calibration error on the declined pool under a 0.10 target through severity 0.4–0.0969 on the SCM world, against a naive 0.1123 — and fails at 0.6, where 0.2439 against 0.2728 leaves both far over target. The same shape appears in the harness’s simpler flat-generator world. There is an operating region where the standard correction works, and a wall where it stops.
Third — the finding I care most about: both walls are the same wall. The counterfactual advantage collapses and the calibration correction fails at the same severity, for the same structural reason: selection routed through an unobserved confounder. Backdoor adjustment cannot fix confounding by what you cannot condition on; IPW cannot fix positivity that selection has destroyed. One cause, measured two independent ways, in one world. This is also the precise sense in which conditioning on a rich covariate vector is not causal adjustment: the failure is not that the model lacked features, but that the selection mechanism ran through a variable no feature set contained.
One qualification exists only because the harness was forced to produce it. The frontier is a distribution, not a point. Quoted from a single seed, the wall sits at 0.4. Run across the full 25-seed sweep, the SCM world’s median frontier is 0.2 — the majority of draws fail one step earlier than the headline seed. The honest statement is an operating frontier of 0.2 to 0.4 depending on the draw. A frontier quoted from one seed is a figure published without an error bar.
Everything above is a claim made by code about code, and self-reported success is the cheapest commodity in machine learning. So the harness runs inside a verification discipline — mechanical gates plus adversarial LLM agents whose standing instruction is to refute claims, not confirm them: recompute every number from the committed artifact, re-run every gate, and treat “tests pass” as a claim to reproduce rather than a fact to accept. This is where language models earn their place in this system. Not generating the scenarios — the SCM does that, deterministically — but falsifying the claims made about them.
Three defects this discipline caught in this project, each of which would otherwise have shipped.
An estimator that could not lose. The first version of the counterfactual evaluation contained a tautology: the “estimator” being graded secretly called the ground-truth oracle, so its near-zero error certified nothing. The refutation pass caught it. The fix was the real g-computation estimator described above, fit only on observed data — and the structural rule that the oracle path is labeled a truth reference and never scored as an estimator.
A number sourced from nowhere. A draft results table cited a figure computed by an ad-hoc script that priced the wrong round’s random draw. The committed artifact said otherwise. The rule that numbers come from committed artifacts, recomputed at the point of claiming and never restated from memory, exists precisely for this failure — and it fired.
A verification that verified nothing. A mutation-testing harness reported every planted defect “caught” — because the mutants crashed on an import error before a single test ran. Green output, zero evidential content. The check was itself checked: a gate that has never been seen red on known-bad input is not evidence, and the harness now demonstrates each gate failing on a planted defect before its passes are credited.
That last principle generalizes, and I think it is the transferable lesson. The same logic that motivates planting ground truth in a synthetic world applies to the verification layer itself. You do not trust an estimator that was never graded against a known answer, and you do not trust a test that was never shown to fail.
The claims are scoped, and the scope is the point. The fidelity gate validates marginals, not joint structure — a mechanism the SCM lacks is a scenario the harness cannot probe. The frontier severities are parameters of a synthetic world, not measurements of any real portfolio; what transfers is the shape of the result — a working region, a wall, one cause — not the numbers. And the harness measures where methods fail under selection through unobserved confounding, which is a distinct pathology from the deterministic-cutoff positivity failure you meet when an approval threshold, rather than a confounder, is what severs the data. Conflating the two walls is a category error with real downstream consequences, and this harness only speaks to the first.
What it does deliver is the thing real data structurally withholds: a falsifiable answer to “how wrong is my counterfactual reasoning, and where does it stop working?” For a modeling culture that increasingly makes causal claims off observational lending data, an instrument that can say here, and no further — and show its own answer key — seems like the minimum standard of evidence worth publishing.
Every number in this essay recomputes from CSVs committed in the repository — none of them is quoted from a run you have to take my word for:
git clone https://github.com/hossainpazooki/closed-loop-default-detection.gitcd closed-loop-default-detectionpip install -e ".[dev]"
pytest # 203 tests under pinned sklearn/numpypython scripts/paired_significance.py # the 25-seed counterfactual headlinepython scripts/run_frontier_sweep.py # the frontier as a distribution, not a point
The full independent assessment — methodology, every number, and what did not survive verification — is committed alongside the code as a dated snapshot at docs/assessment.md.
This essay describes version 0.3.0 of the harness (github.com/hossainpazooki/closed-loop-default-detection, MIT-licensed, released 2026–07–22; installable as closed-loop-default-detection from PyPI). Later versions are separate publication events. All figures and code are the author's own work.
When You Can’t Measure What Matters: A Closed-Loop Harness for Probability-of-Default Estimation was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.