# OpenDiscoveryTrace proves that output-only benchmarks for AI scientists are lying to us > Source: > Published: 2026-09-11 17:45:26+00:00 # OpenDiscoveryTrace proves that output-only benchmarks for AI scientists are lying to us Evaluating an AI scientist by just looking at the final paper or code is like grading a math test by looking at the answer key without seeing the work. If the model gets it right, you don't know if it actually reasoned through the problem or just hallucinated a lucky guess. OpenDiscoveryTrace changes this by releasing 558 full trajectories of AI agents actually doing science—drug discovery, genomics, and materials science—recording every thought, tool call, and error. ## Why process traces matter more than final answers I've noticed that when I use agents for research, they often arrive at the right conclusion through a chaotic mess of failed API calls and circular reasoning. This dataset confirms that on a massive scale. The researchers found that while frontier models like GPT-5.4 and [Claude](/en/tags/claude/) Opus 4.6 have similar success rates (between 84% and 89%), their internal "thought processes" are wildly different. The most shocking part is the error rate. Claude Opus 4.6 triggered 30 times more errors than GPT-5.4, averaging 2.5 errors per trajectory compared to GPT-5.4's 0.08. Even crazier, the *type* of failure differed: Claude mostly struggled with tool misuse (66.7% of its errors), whereas GPT-5.4 failed primarily due to reasoning gaps (83.6% of its errors). If you only looked at the final output, you'd think these models are identical. ## What is actually inside the dataset The dataset covers 124 scientific tasks. They didn't just dump text; they used a structured 9-field-per-step schema. If you're building an agent harness, this is the part to look at. Every single step in a trajectory includes: - Internal thoughts - Specific tool calls - Observations from the environment - Error logs - Revision triggers (when the model realizes it messed up) - Self-reported confidence levels The data isn't just from the big players. It includes 124 trajectories each for GPT-5.4, Claude Opus 4.6, and [Gemini](/en/tags/gemini/)3.1 Pro, plus smaller samples (30 each) for open-weight models like Qwen2.5-7B and Phi-3.5-mini. ## How to use this for your own agent auditing If you are trying to optimize a scientific agent, stop looking at the success rate and start looking at the "revision trigger" and "error profile" fields. This dataset provides five benchmark tasks with baselines using LSTMs, Random Forests, and Transformers to show where LLMs actually add value versus where a simple regression would have worked. For those wanting to implement a similar auditing system, the trace schema is available under CC BY 4.0. You can basically clone their harness to track your own agent's "confidence" vs "actual success" to see if your model is overconfident in its hallucinations. The full technical details and the dataset are hosted on arXiv:2609.09203v1. [Next GPT-Live-1 allows for real-time voice interruptions at $0.05 per minute →](/en/threads/9181/) [these real-world AI monetization case studies](https://tanyan888.com/), with plenty of directly applicable cases. ## All Replies （3） Curious if this happens with GPT-4o too. I once had a model hallucinate a whole library just to pass a test case. I want to try this tonight. I've spent hours debugging "correct" results that actually relied on a hidden 2022 leak. Maybe with LangSmith? Frustrated by this because I've seen "perfect" scores hide massive data leaks. Does this apply to the 401-v2 dataset?