cd /news/robotics/phantom-success-silent-failure-audit… · home › topics › robotics › article
[ARTICLE · art-173031] src=arxiv.org ↗ pub= topic=robotics verified=true sentiment=↓ negative

Phantom Success, Silent Failure: Auditing the Success Checkers of Simulated Manipulation Benchmarks

An LLM-based auditing pipeline called SimAudit found 61 distinct defects across 49 tasks in three simulated manipulation benchmarks — RoboTwin 2.0, RoboCasa365 and RoboDojo — including 40 phantom successes (checkers accepting incomplete executions) and 21 silent failures (checkers rejecting executions that satisfy the instruction), according to the arXiv paper 2611.25179v1. The pipeline analyzes task instructions and hand-written success-checker code to predict disagreement scenarios, recreates them in simulation, and uses multi-view renderings to verify whether observed behavior satisfies the instruction; 183 simulated test cases produced the predicted checker verdict. Most defects in RoboCasa365 accepted incomplete executions, most in RoboDojo rejected valid executions, and both error types were common in RoboTwin 2.0, leading the authors to call for testing incomplete executions and valid alternatives to expert demonstrations in future benchmark evaluation protocols.

recommendation

Audit benchmark success checkers by simulating partial and alternative task completions and verifying outcomes with multi-view renderings; expert-demo acceptance alone missed 61 defects across 49 tasks.

more in playbook: Evals and testing for LLM apps
by read1 min views2 publishedNov 20, 2026

arXiv:2611.25179v1 Announce Type: new Abstract: Simulated manipulation benchmarks measure policy performance using hand-written success checkers that label each episode as a success or failure. Passing an expert demonstration shows that a checker accepts one way of completing the task. Learned policies, however, may stop after partial progress or complete the task in a different way. A checker may then accept an incomplete execution (phantom success) or reject an execution that satisfies the instruction (silent failure). In this paper, we introduce SimAudit, an LLM-based pipeline that audits success checkers against task instructions. SimAudit analyzes the instruction and checker code to identify scenarios where the two may disagree, recreates those scenarios in simulation, and runs the original checker. Multi-view renderings are then used to verify whether the observed behavior satisfies the instruction. Across RoboTwin 2.0, RoboCasa365, and RoboDojo, 183 simulated test cases produced the predicted checker verdict. Visual verification and deduplication yielded 61 distinct defects across 49 tasks, including 40 phantom successes and 21 silent failures. Most defects in RoboCasa365 accepted incomplete executions, most in RoboDojo rejected valid executions, and both error types were common in RoboTwin 2.0. These findings suggest that validating success checkers requires testing both incomplete executions and valid alternatives to expert demonstrations, and motivate more rigorous evaluation protocols for future benchmarks.

── more in #robotics 4 stories · sorted by recency
── more on @simaudit 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/phantom-success-sile…] indexed:0 read:1min 2026-11-20 · —