{"slug": "wilson-95-lower-bounds-on-rule-survival-across-29", "title": "Wilson 95% lower bounds on rule survival across 29", "summary": "A study of two independent extractors parsing Missouri statutes found a 0.43 false-negative rate on numeric-threshold presence, and a survival analysis using 1,000 Monte Carlo trials with a Wilson 95% lower bound of 0.95 showed that 93.2% of held-out chapters fall below the informativeness floor under a globally deployed error model, while per-chapter calibration preserves rule survival. The analysis covered 29,365 Missouri sections and 502 Indian central-Act sections, with a preregistered held-out gate achieving exact matches across 10 statute families in 7 Titles and 16 families across 11 Titles with 5% tolerance.", "body_md": "# Wilson 95% lower bounds on rule survival across 29\n\nThe setup: two independent extractors parsed Missouri statutes, and on numeric-threshold presence they hit a **0.43 false-negative rate**. Almost half the time one misses what the other catches. That number alone should make anyone running [RAG](/en/tags/rag/) over legal corpora pause.\n\nWhat they actually built:\n\n1. Take the Duquenne-Guigues implication basis of the extracted contexts (for anyone who's dealt with FCA, this is the minimal non-redundant rule set).\n\n2. Measure per-attribute disagreement between extractors.\n\n3. Replay that disagreement against the basis in **1,000 Monte Carlo trials**.\n\n4. An implication only \"survives\" if a one-sided Wilson 95% lower bound on its survival rate hits **0.95**.\n\nEach surviving rule ships with its premise spans and a minimal counterexample. That's the audit trail. That's the part I like — you can actually point at a rule and say \"here's the exact statutory text, and here's the smallest thing that would break it.\"\n\nThe empirical results are messy in a useful way:\n\n**29,365 Missouri sections + 502 Indian central-Act sections**- Preregistered held-out gate:\n**10 statute families across 7 Titles exact**,** 16 across 11 with 5% tolerance** - Under one globally deployed error model,\n**93.2% of held-out chapters fall below the informativeness floor** - 2x2 factorial pins that collapse on\n**calibration-rate transfer**, not selection\n\nTranslation: the rules themselves are fine. The extraction error doesn't transfer cleanly across chapters, so a globally-tuned error model quietly kills your rules. If you calibrate per-chapter, or accept some error tolerance, the certificate survives. Otherwise you're shipping dead logic.\n\nThere's also a retracted claim in the released audit trail, which I'm pointing out because more authors should do this.\n\n``` python\n# The Wilson 95% one-sided lower bound (the core gate)\ndef wilson_lower(k, n, z=1.6448536269514722):\n    p = k / n\n    denom = 1 + z*z/n\n    centre = p + z*z/(2*n)\n    adj = z * ((p*(1-p) + z*z/(4*n))/n) ** 0.5\n    return (centre - adj) / denom\n```\n\nTwo takeaways I'm carrying forward into my own LLM-extraction work:\n\n**Don't trust a single extractor's output without a survival check.** Even at 0.43 FNR, rules with premise+counterexample chains can still certify — but only if you measure per-attribute noise, not aggregate accuracy.**Global error models are a trap for legal/scientific text.** Domain drift between chapters is real, and a single threshold turns 93% of your signal into noise.\n\nIf you're doing prompt engineering for legal RAG, structured extraction, or building LLM agents over regulatory corpora, the per-chapter calibration point is the one to internalize. The Duquenne-Guigues machinery is overkill for most use cases, but the discipline — measure disagreement, bound survival, ship counterexamples — isn't.\n\nHas anyone else been hitting this calibration-transfer wall with extraction pipelines? I'm curious whether the 93.2% collapse rate holds up outside US statutes.\n\n[Next NVIDIA is pushing local inference speeds up by 1. →](/en/threads/8721/)", "url": "https://wpnews.pro/news/wilson-95-lower-bounds-on-rule-survival-across-29", "canonical_source": "https://promptcube3.com/en/threads/8753/", "published_at": "2026-09-03 20:50:03+00:00", "updated_at": "2026-09-03 21:22:46.094458+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "natural-language-processing"], "entities": ["Missouri", "Duquenne-Guigues", "Wilson", "Monte Carlo", "RAG", "NVIDIA"], "alternates": {"html": "https://wpnews.pro/news/wilson-95-lower-bounds-on-rule-survival-across-29", "markdown": "https://wpnews.pro/news/wilson-95-lower-bounds-on-rule-survival-across-29.md", "text": "https://wpnews.pro/news/wilson-95-lower-bounds-on-rule-survival-across-29.txt", "jsonld": "https://wpnews.pro/news/wilson-95-lower-bounds-on-rule-survival-across-29.jsonld"}}