arXiv:2611.22241v1 Announce Type: new Abstract: Benchmarks for LLM diagnostic agents are built from retrospective records: the case an agent reads is rewritten from discharge summaries, and unrecorded test results are generated by a simulator that often sees the final diagnosis. HindAudit separates three hindsight channels (the diagnosis given to the simulator, the retrospective context it conditions on, and the agent-facing case text) and measures each with judges trained only on real data. On 27,489 emergency admissions from MIMIC-IV, five registered tests found no leakage into simulated laboratory results: providing the diagnosis to five simulators had a negligible effect, and prompted gatekeepers diluted the diagnostic signal, although exploratory results suggest they can raise agent scores. The largest effect lay in the case text. Against a presentation-time reconstruction from triage records, benchmark-style summaries raised the zero-shot accuracy of four LLMs from 25.4% to 39.7% and discharge histories to 45.3%, and raised six interactive agents' scores while barely changing their ranking. The exploratory gap was reproduced in a registered replication on held-out patients. Masking diagnosis names left it unchanged. Because MIMIC lacks a contemporaneous bedside history, the gap is an upper bound on hindsight. We report all registered outcomes and recommend construction practices.
Where Does Hindsight Enter Clinical Agent Benchmarks? An Audit of Simulators and Case Construction
An audit of LLM diagnostic-agent benchmarks, HindAudit, found that benchmark-style case summaries built from retrospective records inflated zero-shot accuracy for four LLMs from 25.4% to 39.7%, and discharge histories pushed it to 45.3%, compared with a presentation-time reconstruction from triage records. Across 27,489 emergency admissions from MIMIC-IV, the study's five registered tests found no leakage into simulated laboratory results — giving the diagnosis to five simulators had a negligible effect — while prompted gatekeepers diluted the diagnostic signal even as exploratory results suggested they can raise agent scores. The authors report the case-text gap was reproduced in a registered replication on held-out patients and recommend construction practices, noting that because MIMIC lacks a contemporaneous bedside history the gap is an upper bound on hindsight.
Build diagnostic-agent eval cases from presentation-time triage records instead of retrospective summaries or discharge histories, which inflated zero-shot accuracy from 25.4% to 39.7% and 45.3% in this audit.
more in playbook: Evals and testing for LLM appsRun your AI side-project on zahid.host
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.