DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Models

Researchers introduced DeFAb, a benchmark for defeasible abduction in foundation models, converting knowledge bases into 372,648+ logically verifiable instances. Frontier language models achieved at most 65% accuracy, dropping to 23.5% under robust evaluation, while a symbolic solver reached 100% in microseconds. The benchmark aims to measure disciplined theory revision rather than fluent prose, with applications as a reward signal for preference optimization.

arXiv:2606.18557v1 Announce Type: new Abstract: A rule-based logic solver resolves every instance in our benchmark in under 50 microseconds with 100% accuracy; the best frontier language model reaches 65% at best and drops to 23.5% under rendering-robust evaluation worst case over four surface renderings . We introduce DeFAb Defeasible Abduction Benchmark , a dataset and generation pipeline that converts four decades of publicly funded knowledge bases into formally grounded instances for defeasible abduction: constructing hypotheses that explain anomalies by overriding defaults while preserving unrelated expectations. Because every hypothesis must pass polynomial-time checks for valid derivation, conservativity, and minimality, DeFAb makes logical rigor the instrument for measuring creativity and theoretical reasoning, scoring the disciplined construction of theory revisions rather than fluent but theory-destroying prose. The pipeline pairs taxonomic hierarchies OpenCyc, YAGO, Wikidata with behavioral property graphs ConceptNet, UMLS to produce 372,648+ instances across 33.75M materialized rules from 18 sources, in three levels with polynomial-time verifiable gold standards. Four frontier models do not reliably internalize defeasible reasoning: rendering-robust Level 2 accuracy is 7.8-23.5%; chain-of-thought variance ~36 pp exceeds any inter-model gap; and a matched contamination control isolates a +19.4 pp Level 3 gap. We further release DeFAb-Hard a 235-instance Level 3 difficulty variant; best model 53.3% vs 100% symbolic and CONJURE a kernel-verified transformative-creativity variant of 560 Lean 4/Mathlib instances whose gold answers are definitions the proof kernel did not previously contain, judge-free verifier; a pilot finds zero novel concepts . The same verifier doubles as an exact reward for preference optimization DPO, RLVR/GRPO . Released under MIT at https://huggingface.co/datasets/PatrickAllenCooper/DeFAb.