{"slug": "i-built-a-causal-inference-engine-the-best-thing-it-does-is-refuse-to-answer", "title": "I Built a Causal Inference Engine. The Best Thing it Does is Refuse to Answer.", "summary": "A causal inference engine built by an anonymous developer refuses to answer when it cannot produce a valid estimate, a feature that prevents the most common and expensive mistake in business analytics: mistaking correlation for causation. The engine automatically classifies covariates as confounders, mediators, or colliders and drops dangerous ones before estimation, avoiding the harmful practice of controlling for everything measured. On synthetic data with known true effects, the engine correctly recovered the causal effect in 10 of 10 test cases, including scenarios with no effect and realistic confounding.", "body_md": "A retailer sends a discount to its most valuable customers. Those customers spend more. The report goes out: the discount worked.\n\nIt almost certainly didn’t — or nowhere near as much as the number says. The discount went to people who were already the biggest spenders. The comparison credits it with behavior they would have shown regardless.\n\nThis is one of the most expensive mistakes in business analytics, and it is completely invisible. The correlation is real. The data is clean. The dashboard is correct. The conclusion is wrong.\n\nAn A/B test fixes it by assigning at random. But you frequently cannot run one: the policy already rolled out, the data already exists, randomizing would be unethical, illegal, or impossible.\n\nSo I built a causal inference engine for exactly those cases. And the feature I ended up caring about most is the one that produces nothing: knowing when to decline.\n\nBefore any estimate, you have to decide which variables to control for. This is where most competent analyses go wrong, because “control for everything you measured” is actively harmful advice.\n\nA variable can play three roles, and they demand opposite treatment:\n\n**A confounder** causes both the treatment and the outcome. Income drives both who gets the discount and how much they spend. Adjust for it — leaving it out is exactly what produces the fake result.\n\n**A mediator** sits *on* the causal path. The discount increases engagement, which increases spend. Adjust for engagement and you subtract part of the very effect you are trying to measure. Your estimate shrinks toward zero and you conclude the discount did nothing.\n\n**A collider** is caused by both. Adjust for it and you *create* an association that does not exist in the data at all.\n\nSo the engine builds a causal graph, classifies every covariate, and drops the dangerous ones automatically — in plain English, before anything is estimated:\n\n```\nincome       — confounder · Adjust for itengagement   — mediator   · Do not adjust: it sits on the causal pathsatisfaction — collider   · Do not adjust: it creates a spurious association\n```\n\nWriting that classifier surfaced a bug worth mentioning. My first version labelled an *instrument* as a confounder. The logic checked “does this variable reach the outcome?” — and every cause of the treatment reaches the outcome *through* the treatment. The fix was to ask whether a path exists that avoids the treatment. Small distinction, completely different adjustment set.\n\nHere is the awkward thing about causal inference: on real data you cannot grade it. Nobody knows the true effect. That is the entire reason you are estimating it.\n\nSo the engine is graded on synthetic data where I build the true effect in myself. Every estimator can then be asked a question with a right answer:\n\n```\ndataset        method                  true     naive  estimate  bias cut  expected  ok------------------------------------------------------------------------------------------confounded     psm                    2.000     5.992     2.029     99.3%  recover   yesconfounded     ipw                    2.000     5.992     1.715     92.9%  recover   yesheterogeneous  x_learner              1.895     1.961     1.900     91.7%  recover   yesdid_panel      did                    3.000     7.214     2.989     99.7%  recover   yesinstrumental   ipw (cannot fix this)  1.500     4.122     4.125     -0.1%  fail      yesinstrumental   iv                     1.500     4.122     1.665     93.7%  recover   yesno_effect      psm                    0.000     4.198     0.031     99.3%  recover   yesno_effect      ipw                    0.000     4.198     0.102     97.6%  recover   yesrealistic      psm                    0.150     1.263     1.068     17.5%  improve   yesrealistic      ipw                    0.150     1.263     0.825     39.4%  improve   yes------------------------------------------------------------------------------------------10 of 10 behaved as expected\n```\n\nThree rows carry the weight.\n\n**The no-effect row.** The treatment does nothing whatsoever. A raw comparison confidently reports **4.20**, because high-risk units were far likelier to receive it and had worse outcomes regardless. The engine reports **0.03** and marks it non-significant.\n\nThis is the most important test in the suite. A tool that always finds an effect is worse than no tool, because it launders a guess into a number. Proving it stays quiet when nothing is there is the only thing that earns trust when the answer is unknown.\n\n**The instrumental row.** Here the confounder is deliberately absent from the data. Weighting stays wrong at **4.13** no matter how carefully applied — no amount of adjustment fixes a variable you never measured. Only an instrument recovers **1.67**. That row is graded as a *successful demonstration of a limitation*, which required me to rethink the grading itself.\n\nMy first version marked it as a failure. But the row exists precisely to show that adjustment has limits — reporting a correct demonstration as a defect is just wrong. So rows now declare what they are supposed to do: *recover* the truth, merely *improve* on the naive answer, or *fail*.\n\n**The realistic row.** A small effect against heavy noise, one confounder observed only as a noisy proxy, and another missing entirely. The estimators cut the bias but land at 0.83–1.07 against a true 0.15, with confidence intervals that confidently exclude the truth.\n\nThat is the normal outcome on observational data. I included it deliberately, because publishing only the clean rows would misrepresent what these methods deliver.\n\nAn average effect can be positive while the treatment actively harms a large minority.\n\nOn data where the treatment helps one segment strongly, helps a second mildly, and hurts a third, the X-learner recovers all three:\n\n```\nA: estimated +5.02 against a true +5.00  B: estimated +1.49 against a true +1.50  C: estimated -1.99 against a true -2.00  (harmed)  correlation with the planted per-unit effect: 0.997\n```\n\nThe headline average is a positive **+1.9**. Roll the treatment out on that basis and you damage a quarter of your population, and the average will never tell you.\n\nTargeting the top 30% by predicted effect captures 64% of the total available benefit. That is the number a marketing team can actually act on, and it does not exist anywhere in an A/B test result.\n\nThe strongest test is not synthetic at all.\n\nThe LaLonde job training study is the canonical benchmark in this field. A randomized trial establishes the true effect on earnings at **+$1,794**. LaLonde’s contribution was to throw away the experiment’s control group, substitute an unrelated population survey, and ask whether statistical adjustment could recover the experimental answer from that observational comparison.\n\nLargely, it could not. Run through my engine, the naive comparison reports **−$15,578**. The program appears to have destroyed fifteen thousand dollars of earnings, when the experiment shows it added eighteen hundred. Wrong by more than seventeen thousand dollars, and in the wrong direction.\n\nMatching claws most of it back to **−$60**. Weighting barely helps at **−$12,759**.\n\nBut the interesting part is what the engine does *before* producing any of those numbers. The assumption checks report:\n\nVerdict: **“This data cannot support a credible causal estimate.”**\n\nThe headline refuses to quote an effect size at all. Not a number with a footnote — no number.\n\nThat is the title of this piece, and it is the whole argument. Every one of those estimates is wrong. Matching lands closest and is still off by $1,854. A tool that hands you the closest wrong answer has done you no favor. Declining is the correct output, and it is the one almost nothing produces.\n\nIt is easy to write caveats nobody reads. So the honesty is built into the structure instead:\n\n**The interface withholds results.** Click “Results” without opening the assumptions page and you get no effect size — only the reason one is being withheld. A number seen first tends to be remembered regardless of what follows it, so the conditions come before the number.\n\n**The PDF leads with the verdict.** Whether the analysis can be trusted appears above the headline, not appended to it.\n\n**Every estimate carries its estimand.** Matching estimates the effect *on the treated*. Weighting estimates it *across everyone*. An instrument estimates it *only on units it moved*. Reporting one as another is a misstatement, not a rounding error, so the type travels with the number everywhere.\n\n**Estimates get attacked.** A placebo test replaces the treatment with random noise. If a comparable effect survives that, the pipeline is manufacturing signal and the result cannot be causal — full stop, regardless of its confidence interval.\n\nBuilding that test taught me something. My first version compared the placebo estimate against a fraction of the original — which fails every *correct* null result, since that fraction is near zero by definition. It reported a correctly detected absence of effect as a failure. The fix was to judge the estimate against the spread of placebo estimates themselves. And a result indistinguishable from random assignment now returns a verdict of no_effect rather than robust, because calling it robust invites someone to act on an effect that was never found.\n\nNo statistical method rescues a bad research design:\n\nOpen source: **github.com/Emart29/causal-inference-engine**\n\n```\npython -m validation.ground_truth   # the table above, on your machinestreamlit run ui/app.py             # the interface\n```\n\nFive methods, four robustness checks, sensitivity analysis, plain-English interpretation, and PDF export. Built on scikit-learn, statsmodels, NetworkX, and Streamlit.\n\nThis is the fifth project in an ML platform I’m building in public — after a feature store, a lineage tracker, a canary deployment system, and a real-time anomaly detector. It is the first that answers a business question rather than serving a model: not *is the model working*, but *did the thing we did actually work*.\n\nAnd the most valuable thing it ever told me was that it didn’t know.\n\n*Next: a personalization engine with an online feedback loop — recommendations that update from what people actually do, not what they did last month.*\n\n[I Built a Causal Inference Engine. The Best Thing it Does is Refuse to Answer.](https://pub.towardsai.net/i-built-a-causal-inference-engine-the-best-thing-it-does-is-refuse-to-answer-1a7fd5493e5e) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/i-built-a-causal-inference-engine-the-best-thing-it-does-is-refuse-to-answer", "canonical_source": "https://pub.towardsai.net/i-built-a-causal-inference-engine-the-best-thing-it-does-is-refuse-to-answer-1a7fd5493e5e?source=rss----98111c9905da---4", "published_at": "2026-07-28 06:34:42+00:00", "updated_at": "2026-07-28 07:12:44.732565+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "ai-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/i-built-a-causal-inference-engine-the-best-thing-it-does-is-refuse-to-answer", "markdown": "https://wpnews.pro/news/i-built-a-causal-inference-engine-the-best-thing-it-does-is-refuse-to-answer.md", "text": "https://wpnews.pro/news/i-built-a-causal-inference-engine-the-best-thing-it-does-is-refuse-to-answer.txt", "jsonld": "https://wpnews.pro/news/i-built-a-causal-inference-engine-the-best-thing-it-does-is-refuse-to-answer.jsonld"}}