Wilson 95% lower bounds on rule survival across 29

A study of two independent extractors parsing Missouri statutes found a 0.43 false-negative rate on numeric-threshold presence, and a survival analysis using 1,000 Monte Carlo trials with a Wilson 95% lower bound of 0.95 showed that 93.2% of held-out chapters fall below the informativeness floor under a globally deployed error model, while per-chapter calibration preserves rule survival. The analysis covered 29,365 Missouri sections and 502 Indian central-Act sections, with a preregistered held-out gate achieving exact matches across 10 statute families in 7 Titles and 16 families across 11 Titles with 5% tolerance.

Wilson 95% lower bounds on rule survival across 29 The setup: two independent extractors parsed Missouri statutes, and on numeric-threshold presence they hit a 0.43 false-negative rate . Almost half the time one misses what the other catches. That number alone should make anyone running RAG /en/tags/rag/ over legal corpora pause. What they actually built: 1. Take the Duquenne-Guigues implication basis of the extracted contexts for anyone who's dealt with FCA, this is the minimal non-redundant rule set . 2. Measure per-attribute disagreement between extractors. 3. Replay that disagreement against the basis in 1,000 Monte Carlo trials . 4. An implication only "survives" if a one-sided Wilson 95% lower bound on its survival rate hits 0.95 . Each surviving rule ships with its premise spans and a minimal counterexample. That's the audit trail. That's the part I like — you can actually point at a rule and say "here's the exact statutory text, and here's the smallest thing that would break it." The empirical results are messy in a useful way: 29,365 Missouri sections + 502 Indian central-Act sections - Preregistered held-out gate: 10 statute families across 7 Titles exact , 16 across 11 with 5% tolerance - Under one globally deployed error model, 93.2% of held-out chapters fall below the informativeness floor - 2x2 factorial pins that collapse on calibration-rate transfer , not selection Translation: the rules themselves are fine. The extraction error doesn't transfer cleanly across chapters, so a globally-tuned error model quietly kills your rules. If you calibrate per-chapter, or accept some error tolerance, the certificate survives. Otherwise you're shipping dead logic. There's also a retracted claim in the released audit trail, which I'm pointing out because more authors should do this. python The Wilson 95% one-sided lower bound the core gate def wilson lower k, n, z=1.6448536269514722 : p = k / n denom = 1 + z z/n centre = p + z z/ 2 n adj = z p 1-p + z z/ 4 n /n 0.5 return centre - adj / denom Two takeaways I'm carrying forward into my own LLM-extraction work: Don't trust a single extractor's output without a survival check. Even at 0.43 FNR, rules with premise+counterexample chains can still certify — but only if you measure per-attribute noise, not aggregate accuracy. Global error models are a trap for legal/scientific text. Domain drift between chapters is real, and a single threshold turns 93% of your signal into noise. If you're doing prompt engineering for legal RAG, structured extraction, or building LLM agents over regulatory corpora, the per-chapter calibration point is the one to internalize. The Duquenne-Guigues machinery is overkill for most use cases, but the discipline — measure disagreement, bound survival, ship counterexamples — isn't. Has anyone else been hitting this calibration-transfer wall with extraction pipelines? I'm curious whether the 93.2% collapse rate holds up outside US statutes. Next NVIDIA is pushing local inference speeds up by 1. → /en/threads/8721/