Interventional Grounding Audits: Black-Box Premise-Dependency Tests for LLM Chain-of-Thought via Predicate Substitution

Researchers introduced interventional grounding audits, a black-box method to test whether LLM chain-of-thought reasoning genuinely depends on its stated premises by substituting predicates. Applied to GPT-4o on ProntoQA, the method achieved F1=0.806 for detecting proof-tree dependencies, outperforming a self-consistency baseline, and revealed that 66% of correct solutions had steps insensitive to direct dependencies, indicating "right answer, wrong reasoning.

arXiv:2607.13069v1 Announce Type: new Abstract: Large language models produce chain-of-thought CoT reasoning that appears logically sound yet may not genuinely depend on its stated premises. We introduce interventional grounding audits, a black-box, step-level test of premise dependency: we intervene on a single premise by substituting its target predicate with a fresh symbol, re-run the model, and check whether each reasoning step's normalized conclusion canonical predicate form changes. We evaluate on ProntoQA, a synthetic multi-hop deductive reasoning benchmark with gold proof trees, where step-level premise dependencies are known. Applied to 50 ProntoQA problems with GPT-4o, our method achieves F1 = 0.806 on detecting proof-tree dependencies F1 = 0.885 on predicate-determining dependencies; Recall = 100% , significantly outperforming a self-consistency baseline F1 = 0.343; 95% bootstrap CIs non-overlapping . We further identify that 66% of correctly-solved problems contain at least one aligned step insensitive to a direct proof-tree dependency under consistent substitution -- all involving entity-introduction premises, a documented blind spot of the consistent-substitution evaluator -- a "right answer, wrong reasoning" signal invisible to passive methods. All audit certificates, raw outputs, and reproduction scripts are available in a public GitHub repository, and we discuss scope limits beyond formal, parsable benchmarks.