{"slug": "credit-without-ground-truth-auditing-step-level-credit-assignment-in-llm-agents", "title": "Credit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replay", "summary": "A new arXiv study (2608.19760v1) auditing step-level credit assignment in LLM agents against causal ground truth from executed replay in ALFWorld found that none of the credit signals used to train LLM agents—LLM-judge scores, outcome-conditioned logprob ratios, or the policy's own confidence—identifies causally important steps better than chance. The ground truth is sparse (30.5% of decision points carry measurable effect), and measurability varies by policy (13.1% vs. 26.8%). A confidence-only router cuts judge cost by 13.1% per turn but recovers pivotal steps at chance level, and a seven-arm pre-registered training experiment showed no arm reliably outperforms the untrained policy, with apparent differences explained by training dose, not credit content.", "body_md": "arXiv:2608.19760v1 Announce Type: new\nAbstract: Audited against causal ground truth from executed replay in a single-agent tool environment (ALFWorld), none of the step-level credit signals used to train LLM agents -- LLM-judge scores, outcome-conditioned logprob ratios, or the policy's own confidence -- identifies which steps causally matter better than chance. Existing evaluations grade these signals against annotated step *correctness*; we audit them against step *contribution* -- what re-sampling the policy's own alternatives at each decision point and rolling forward actually changes about the outcome -- and the two come apart. The ground truth itself is structured: causal contribution is sparse (30.5% of decision points where ground truth is defined carry measurable effect), and measurability is model-dependent -- the fraction of points with no policy-supported counterfactual differs by a factor of two (13.1% vs. 26.8%) between two similar-scale policies. The failure mode is identifiable: implicit credit echoes the policy's fluency (median rank correlation +0.75, replicating at +0.70 in a second family under a corrected instrument), while conditioning on the outcome adds no causal information (partial correlation -0.004, Qwen). A confidence-only router recovers pivotal steps at chance level, but cuts judge cost by 13.1% per turn (14.0% per trajectory). In a seven-arm pre-registered training experiment, no arm reliably outperforms the untrained policy, and the checkpoints' apparent instrument signature is fully explained by training dose -- sparser credit retains fewer examples, an order-of-magnitude spread in optimizer steps -- not credit content. Comparisons of credit rules must therefore match effective sample size, or they measure dose, not credit.", "url": "https://wpnews.pro/news/credit-without-ground-truth-auditing-step-level-credit-assignment-in-llm-agents", "canonical_source": "https://www.machinebrief.com/news/credit-without-ground-truth-auditing-step-level-credit-assig-cm6t", "published_at": "2026-08-21 04:00:00+00:00", "updated_at": "2026-08-21 04:14:35.388354+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "ai-agents"], "entities": ["arXiv", "ALFWorld", "Qwen"], "alternates": {"html": "https://wpnews.pro/news/credit-without-ground-truth-auditing-step-level-credit-assignment-in-llm-agents", "markdown": "https://wpnews.pro/news/credit-without-ground-truth-auditing-step-level-credit-assignment-in-llm-agents.md", "text": "https://wpnews.pro/news/credit-without-ground-truth-auditing-step-level-credit-assignment-in-llm-agents.txt", "jsonld": "https://wpnews.pro/news/credit-without-ground-truth-auditing-step-level-credit-assignment-in-llm-agents.jsonld"}}