{"slug": "interventional-grounding-audits-black-box-premise-dependency-tests-for-llm-chain", "title": "Interventional Grounding Audits: Black-Box Premise-Dependency Tests for LLM Chain-of-Thought via Predicate Substitution", "summary": "Researchers introduced interventional grounding audits, a black-box method to test whether LLM chain-of-thought reasoning genuinely depends on its stated premises by substituting predicates. Applied to GPT-4o on ProntoQA, the method achieved F1=0.806 for detecting proof-tree dependencies, outperforming a self-consistency baseline, and revealed that 66% of correct solutions had steps insensitive to direct dependencies, indicating \"right answer, wrong reasoning.", "body_md": "arXiv:2607.13069v1 Announce Type: new\nAbstract: Large language models produce chain-of-thought (CoT) reasoning that appears logically sound yet may not genuinely depend on its stated premises. We introduce interventional grounding audits, a black-box, step-level test of premise dependency: we intervene on a single premise by substituting its target predicate with a fresh symbol, re-run the model, and check whether each reasoning step's normalized conclusion (canonical predicate form) changes. We evaluate on ProntoQA, a synthetic multi-hop deductive reasoning benchmark with gold proof trees, where step-level premise dependencies are known. Applied to 50 ProntoQA problems with GPT-4o, our method achieves F1 = 0.806 on detecting proof-tree dependencies (F1 = 0.885 on predicate-determining dependencies; Recall = 100%), significantly outperforming a self-consistency baseline (F1 = 0.343; 95% bootstrap CIs non-overlapping). We further identify that 66% of correctly-solved problems contain at least one aligned step insensitive to a direct proof-tree dependency under consistent substitution -- all involving entity-introduction premises, a documented blind spot of the consistent-substitution evaluator -- a \"right answer, wrong reasoning\" signal invisible to passive methods. All audit certificates, raw outputs, and reproduction scripts are available in a public GitHub repository, and we discuss scope limits beyond formal, parsable benchmarks.", "url": "https://wpnews.pro/news/interventional-grounding-audits-black-box-premise-dependency-tests-for-llm-chain", "canonical_source": "https://arxiv.org/abs/2607.13069", "published_at": "2026-07-16 04:00:00+00:00", "updated_at": "2026-07-16 04:30:27.767060+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-research"], "entities": ["GPT-4o", "ProntoQA", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/interventional-grounding-audits-black-box-premise-dependency-tests-for-llm-chain", "markdown": "https://wpnews.pro/news/interventional-grounding-audits-black-box-premise-dependency-tests-for-llm-chain.md", "text": "https://wpnews.pro/news/interventional-grounding-audits-black-box-premise-dependency-tests-for-llm-chain.txt", "jsonld": "https://wpnews.pro/news/interventional-grounding-audits-black-box-premise-dependency-tests-for-llm-chain.jsonld"}}