{"slug": "opendiscoverytrace-proves-that-output-only-benchmarks-for-ai-scientists-are-to", "title": "OpenDiscoveryTrace proves that output-only benchmarks for AI scientists are lying to us", "summary": "OpenDiscoveryTrace released a dataset of 558 full AI-agent trajectories across 124 scientific tasks in drug discovery, genomics, and materials science, hosted on arXiv as 2609.09203v1, to expose how output-only benchmarks hide divergent agent behavior. The researchers found GPT-5.4 and Claude Opus 4.6 had similar success rates of 84% to 89%, but Claude Opus 4.6 averaged 2.5 errors per trajectory versus GPT-5.4's 0.08 — 30 times more — with 66.7% of Claude's errors from tool misuse and 83.6% of GPT-5.4's from reasoning gaps. The trace schema, released under CC BY 4.0, records internal thoughts, tool calls, observations, error logs, revision triggers, and self-reported confidence per step.", "body_md": "# OpenDiscoveryTrace proves that output-only benchmarks for AI scientists are lying to us\n\nEvaluating an AI scientist by just looking at the final paper or code is like grading a math test by looking at the answer key without seeing the work. If the model gets it right, you don't know if it actually reasoned through the problem or just hallucinated a lucky guess. OpenDiscoveryTrace changes this by releasing 558 full trajectories of AI agents actually doing science—drug discovery, genomics, and materials science—recording every thought, tool call, and error.\n\n## Why process traces matter more than final answers\n\nI've noticed that when I use agents for research, they often arrive at the right conclusion through a chaotic mess of failed API calls and circular reasoning. This dataset confirms that on a massive scale. The researchers found that while frontier models like GPT-5.4 and [Claude](/en/tags/claude/) Opus 4.6 have similar success rates (between 84% and 89%), their internal \"thought processes\" are wildly different.\n\nThe most shocking part is the error rate. Claude Opus 4.6 triggered 30 times more errors than GPT-5.4, averaging 2.5 errors per trajectory compared to GPT-5.4's 0.08. Even crazier, the *type* of failure differed: Claude mostly struggled with tool misuse (66.7% of its errors), whereas GPT-5.4 failed primarily due to reasoning gaps (83.6% of its errors). If you only looked at the final output, you'd think these models are identical.\n\n## What is actually inside the dataset\n\nThe dataset covers 124 scientific tasks. They didn't just dump text; they used a structured 9-field-per-step schema. If you're building an agent harness, this is the part to look at. Every single step in a trajectory includes:\n\n- Internal thoughts\n- Specific tool calls\n- Observations from the environment\n- Error logs\n- Revision triggers (when the model realizes it messed up)\n- Self-reported confidence levels\n\nThe data isn't just from the big players. It includes 124 trajectories each for GPT-5.4, Claude Opus 4.6, and\n\n[Gemini](/en/tags/gemini/)3.1 Pro, plus smaller samples (30 each) for open-weight models like Qwen2.5-7B and Phi-3.5-mini.\n\n## How to use this for your own agent auditing\n\nIf you are trying to optimize a scientific agent, stop looking at the success rate and start looking at the \"revision trigger\" and \"error profile\" fields. This dataset provides five benchmark tasks with baselines using LSTMs, Random Forests, and Transformers to show where LLMs actually add value versus where a simple regression would have worked.\n\nFor those wanting to implement a similar auditing system, the trace schema is available under CC BY 4.0. You can basically clone their harness to track your own agent's \"confidence\" vs \"actual success\" to see if your model is overconfident in its hallucinations.\n\nThe full technical details and the dataset are hosted on arXiv:2609.09203v1.\n\n[Next GPT-Live-1 allows for real-time voice interruptions at $0.05 per minute →](/en/threads/9181/)\n\n[these real-world AI monetization case studies](https://tanyan888.com/), with plenty of directly applicable cases.\n\n## All Replies （3）\n\nCurious if this happens with GPT-4o too. I once had a model hallucinate a whole library just to pass a test case.\n\nI want to try this tonight. I've spent hours debugging \"correct\" results that actually relied on a hidden 2022 leak. Maybe with LangSmith?\n\nFrustrated by this because I've seen \"perfect\" scores hide massive data leaks. Does this apply to the 401-v2 dataset?", "url": "https://wpnews.pro/news/opendiscoverytrace-proves-that-output-only-benchmarks-for-ai-scientists-are-to", "canonical_source": "https://promptcube3.com/en/threads/9224/", "published_at": "2026-09-11 17:45:26+00:00", "updated_at": "2026-09-11 18:13:41.147435+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "ai-safety", "large-language-models", "ai-tools"], "entities": ["OpenDiscoveryTrace", "GPT-5.4", "Claude Opus 4.6", "Gemini 3.1 Pro", "Qwen2.5-7B", "Phi-3.5-mini", "arXiv", "LangSmith"], "alternates": {"html": "https://wpnews.pro/news/opendiscoverytrace-proves-that-output-only-benchmarks-for-ai-scientists-are-to", "markdown": "https://wpnews.pro/news/opendiscoverytrace-proves-that-output-only-benchmarks-for-ai-scientists-are-to.md", "text": "https://wpnews.pro/news/opendiscoverytrace-proves-that-output-only-benchmarks-for-ai-scientists-are-to.txt", "jsonld": "https://wpnews.pro/news/opendiscoverytrace-proves-that-output-only-benchmarks-for-ai-scientists-are-to.jsonld"}}