I Built a Causal Inference Engine. The Best Thing it Does is Refuse to Answer.

A causal inference engine built by an anonymous developer refuses to answer when it cannot produce a valid estimate, a feature that prevents the most common and expensive mistake in business analytics: mistaking correlation for causation. The engine automatically classifies covariates as confounders, mediators, or colliders and drops dangerous ones before estimation, avoiding the harmful practice of controlling for everything measured. On synthetic data with known true effects, the engine correctly recovered the causal effect in 10 of 10 test cases, including scenarios with no effect and realistic confounding.

A retailer sends a discount to its most valuable customers. Those customers spend more. The report goes out: the discount worked. It almost certainly didn’t — or nowhere near as much as the number says. The discount went to people who were already the biggest spenders. The comparison credits it with behavior they would have shown regardless. This is one of the most expensive mistakes in business analytics, and it is completely invisible. The correlation is real. The data is clean. The dashboard is correct. The conclusion is wrong. An A/B test fixes it by assigning at random. But you frequently cannot run one: the policy already rolled out, the data already exists, randomizing would be unethical, illegal, or impossible. So I built a causal inference engine for exactly those cases. And the feature I ended up caring about most is the one that produces nothing: knowing when to decline. Before any estimate, you have to decide which variables to control for. This is where most competent analyses go wrong, because “control for everything you measured” is actively harmful advice. A variable can play three roles, and they demand opposite treatment: A confounder causes both the treatment and the outcome. Income drives both who gets the discount and how much they spend. Adjust for it — leaving it out is exactly what produces the fake result. A mediator sits on the causal path. The discount increases engagement, which increases spend. Adjust for engagement and you subtract part of the very effect you are trying to measure. Your estimate shrinks toward zero and you conclude the discount did nothing. A collider is caused by both. Adjust for it and you create an association that does not exist in the data at all. So the engine builds a causal graph, classifies every covariate, and drops the dangerous ones automatically — in plain English, before anything is estimated: income — confounder · Adjust for itengagement — mediator · Do not adjust: it sits on the causal pathsatisfaction — collider · Do not adjust: it creates a spurious association Writing that classifier surfaced a bug worth mentioning. My first version labelled an instrument as a confounder. The logic checked “does this variable reach the outcome?” — and every cause of the treatment reaches the outcome through the treatment. The fix was to ask whether a path exists that avoids the treatment. Small distinction, completely different adjustment set. Here is the awkward thing about causal inference: on real data you cannot grade it. Nobody knows the true effect. That is the entire reason you are estimating it. So the engine is graded on synthetic data where I build the true effect in myself. Every estimator can then be asked a question with a right answer: dataset method true naive estimate bias cut expected ok------------------------------------------------------------------------------------------confounded psm 2.000 5.992 2.029 99.3% recover yesconfounded ipw 2.000 5.992 1.715 92.9% recover yesheterogeneous x learner 1.895 1.961 1.900 91.7% recover yesdid panel did 3.000 7.214 2.989 99.7% recover yesinstrumental ipw cannot fix this 1.500 4.122 4.125 -0.1% fail yesinstrumental iv 1.500 4.122 1.665 93.7% recover yesno effect psm 0.000 4.198 0.031 99.3% recover yesno effect ipw 0.000 4.198 0.102 97.6% recover yesrealistic psm 0.150 1.263 1.068 17.5% improve yesrealistic ipw 0.150 1.263 0.825 39.4% improve yes------------------------------------------------------------------------------------------10 of 10 behaved as expected Three rows carry the weight. The no-effect row. The treatment does nothing whatsoever. A raw comparison confidently reports 4.20 , because high-risk units were far likelier to receive it and had worse outcomes regardless. The engine reports 0.03 and marks it non-significant. This is the most important test in the suite. A tool that always finds an effect is worse than no tool, because it launders a guess into a number. Proving it stays quiet when nothing is there is the only thing that earns trust when the answer is unknown. The instrumental row. Here the confounder is deliberately absent from the data. Weighting stays wrong at 4.13 no matter how carefully applied — no amount of adjustment fixes a variable you never measured. Only an instrument recovers 1.67 . That row is graded as a successful demonstration of a limitation , which required me to rethink the grading itself. My first version marked it as a failure. But the row exists precisely to show that adjustment has limits — reporting a correct demonstration as a defect is just wrong. So rows now declare what they are supposed to do: recover the truth, merely improve on the naive answer, or fail . The realistic row. A small effect against heavy noise, one confounder observed only as a noisy proxy, and another missing entirely. The estimators cut the bias but land at 0.83–1.07 against a true 0.15, with confidence intervals that confidently exclude the truth. That is the normal outcome on observational data. I included it deliberately, because publishing only the clean rows would misrepresent what these methods deliver. An average effect can be positive while the treatment actively harms a large minority. On data where the treatment helps one segment strongly, helps a second mildly, and hurts a third, the X-learner recovers all three: A: estimated +5.02 against a true +5.00 B: estimated +1.49 against a true +1.50 C: estimated -1.99 against a true -2.00 harmed correlation with the planted per-unit effect: 0.997 The headline average is a positive +1.9 . Roll the treatment out on that basis and you damage a quarter of your population, and the average will never tell you. Targeting the top 30% by predicted effect captures 64% of the total available benefit. That is the number a marketing team can actually act on, and it does not exist anywhere in an A/B test result. The strongest test is not synthetic at all. The LaLonde job training study is the canonical benchmark in this field. A randomized trial establishes the true effect on earnings at +$1,794 . LaLonde’s contribution was to throw away the experiment’s control group, substitute an unrelated population survey, and ask whether statistical adjustment could recover the experimental answer from that observational comparison. Largely, it could not. Run through my engine, the naive comparison reports −$15,578 . The program appears to have destroyed fifteen thousand dollars of earnings, when the experiment shows it added eighteen hundred. Wrong by more than seventeen thousand dollars, and in the wrong direction. Matching claws most of it back to −$60 . Weighting barely helps at −$12,759 . But the interesting part is what the engine does before producing any of those numbers. The assumption checks report: Verdict: “This data cannot support a credible causal estimate.” The headline refuses to quote an effect size at all. Not a number with a footnote — no number. That is the title of this piece, and it is the whole argument. Every one of those estimates is wrong. Matching lands closest and is still off by $1,854. A tool that hands you the closest wrong answer has done you no favor. Declining is the correct output, and it is the one almost nothing produces. It is easy to write caveats nobody reads. So the honesty is built into the structure instead: The interface withholds results. Click “Results” without opening the assumptions page and you get no effect size — only the reason one is being withheld. A number seen first tends to be remembered regardless of what follows it, so the conditions come before the number. The PDF leads with the verdict. Whether the analysis can be trusted appears above the headline, not appended to it. Every estimate carries its estimand. Matching estimates the effect on the treated . Weighting estimates it across everyone . An instrument estimates it only on units it moved . Reporting one as another is a misstatement, not a rounding error, so the type travels with the number everywhere. Estimates get attacked. A placebo test replaces the treatment with random noise. If a comparable effect survives that, the pipeline is manufacturing signal and the result cannot be causal — full stop, regardless of its confidence interval. Building that test taught me something. My first version compared the placebo estimate against a fraction of the original — which fails every correct null result, since that fraction is near zero by definition. It reported a correctly detected absence of effect as a failure. The fix was to judge the estimate against the spread of placebo estimates themselves. And a result indistinguishable from random assignment now returns a verdict of no effect rather than robust, because calling it robust invites someone to act on an effect that was never found. No statistical method rescues a bad research design: Open source: github.com/Emart29/causal-inference-engine python -m validation.ground truth the table above, on your machinestreamlit run ui/app.py the interface Five methods, four robustness checks, sensitivity analysis, plain-English interpretation, and PDF export. Built on scikit-learn, statsmodels, NetworkX, and Streamlit. This is the fifth project in an ML platform I’m building in public — after a feature store, a lineage tracker, a canary deployment system, and a real-time anomaly detector. It is the first that answers a business question rather than serving a model: not is the model working , but did the thing we did actually work . And the most valuable thing it ever told me was that it didn’t know. Next: a personalization engine with an online feedback loop — recommendations that update from what people actually do, not what they did last month. I Built a Causal Inference Engine. The Best Thing it Does is Refuse to Answer. https://pub.towardsai.net/i-built-a-causal-inference-engine-the-best-thing-it-does-is-refuse-to-answer-1a7fd5493e5e was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.