{"slug": "where-does-hindsight-enter-clinical-agent-benchmarks-an-audit-of-simulators-and", "title": "Where Does Hindsight Enter Clinical Agent Benchmarks? An Audit of Simulators and Case Construction", "summary": "An audit of LLM diagnostic-agent benchmarks, HindAudit, found that benchmark-style case summaries built from retrospective records inflated zero-shot accuracy for four LLMs from 25.4% to 39.7%, and discharge histories pushed it to 45.3%, compared with a presentation-time reconstruction from triage records. Across 27,489 emergency admissions from MIMIC-IV, the study's five registered tests found no leakage into simulated laboratory results — giving the diagnosis to five simulators had a negligible effect — while prompted gatekeepers diluted the diagnostic signal even as exploratory results suggested they can raise agent scores. The authors report the case-text gap was reproduced in a registered replication on held-out patients and recommend construction practices, noting that because MIMIC lacks a contemporaneous bedside history the gap is an upper bound on hindsight.", "body_md": "arXiv:2611.22241v1 Announce Type: new \nAbstract: Benchmarks for LLM diagnostic agents are built from retrospective records: the case an agent reads is rewritten from discharge summaries, and unrecorded test results are generated by a simulator that often sees the final diagnosis. HindAudit separates three hindsight channels (the diagnosis given to the simulator, the retrospective context it conditions on, and the agent-facing case text) and measures each with judges trained only on real data. On 27,489 emergency admissions from MIMIC-IV, five registered tests found no leakage into simulated laboratory results: providing the diagnosis to five simulators had a negligible effect, and prompted gatekeepers diluted the diagnostic signal, although exploratory results suggest they can raise agent scores. The largest effect lay in the case text. Against a presentation-time reconstruction from triage records, benchmark-style summaries raised the zero-shot accuracy of four LLMs from 25.4% to 39.7% and discharge histories to 45.3%, and raised six interactive agents' scores while barely changing their ranking. The exploratory gap was reproduced in a registered replication on held-out patients. Masking diagnosis names left it unchanged. Because MIMIC lacks a contemporaneous bedside history, the gap is an upper bound on hindsight. We report all registered outcomes and recommend construction practices.", "url": "https://wpnews.pro/news/where-does-hindsight-enter-clinical-agent-benchmarks-an-audit-of-simulators-and", "canonical_source": "https://www.machinebrief.com/news/where-does-hindsight-enter-clinical-agent-benchmarks-an-audi-m0nj", "published_at": "2026-11-18 05:00:00+00:00", "updated_at": "2026-11-18 05:16:24.006606+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-agents", "ai-safety"], "entities": ["HindAudit", "MIMIC-IV", "arXiv"], "also_reported_by": [], "takeaway": "Benchmark-style case summaries raise four LLMs' zero-shot diagnostic accuracy from 25.4% to 39.7% and discharge histories to 45.3% versus a triage-based presentation-time reconstruction, so diagnostic-agent benchmarks should be built from presentation-time records rather than retrospective summaries", "recommendation": {"advice": "Build diagnostic-agent eval cases from presentation-time triage records instead of retrospective summaries or discharge histories, which inflated zero-shot accuracy from 25.4% to 39.7% and 45.3% in this audit.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/where-does-hindsight-enter-clinical-agent-benchmarks-an-audit-of-simulators-and", "markdown": "https://wpnews.pro/news/where-does-hindsight-enter-clinical-agent-benchmarks-an-audit-of-simulators-and.md", "text": "https://wpnews.pro/news/where-does-hindsight-enter-clinical-agent-benchmarks-an-audit-of-simulators-and.txt", "jsonld": "https://wpnews.pro/news/where-does-hindsight-enter-clinical-agent-benchmarks-an-audit-of-simulators-and.jsonld"}}