Which AI models stay consistent across repeated runs?

Deepsense.ai released an October update to its EDA Benchmark, which runs language models and agentic systems five times each across 10 synthetic exploratory data analysis tasks to measure both average quality and run-to-run consistency. claude-opus-5-5 topped the leaderboard with a mean score of 0.65 and a reliability-adjusted score of 0.62, ahead of gpt-6-astra at 0.63 and 0.55, a gap that widens from 0.02 to 0.07 once variation across runs is factored in. The tasks, configuration and evaluation artifacts are published in the project's GitHub repository.

By Rafał Łabędzki, PhD https://deepsense.ai/blog/author/rafal-labedzki/ , Senior AI Engineering Manager at deepsense.ai and Assistant Professor at SGH Warsaw School of Economics https://www.sgh.waw.pl/knop/en/about-institute-human-capital I have worked with AI for more than a decade, starting in the video game industry. In recent years, my work has focused on building evaluation and training environments for machine-learning models, alongside studying human-AI collaboration. What the EDA Benchmark measures The EDA Benchmark evaluates how language models and agentic systems handle exploratory data analysis tasks. These tasks can require a model to inspect files, clean inconsistent records, combine data sources, detect anomalies and return structured results. The benchmark includes 10 synthetic tasks across different domains. We run each model on every task five times, which gives us 50 trajectories per model. Repeated runs matter because one strong answer does not show whether the system can produce a similar result again. We report two scores. The mean score shows average analytical performance. The reliability-adjusted score also accounts for variation across runs, so a model receives a lower score when its results are less consistent. The benchmark uses synthetic data and evaluation methods https://deepsense.ai/blog/synthetic-data-and-evaluation-methods-for-the-next-generation-of-ai-models?utm source=Substsack&utm medium=Post 05 11 2026&utm campaign=Eval designed to test both answer quality and repeatability. A new group at the top In the October round, claude-opus-5-5 leads with a mean score of 0.65 and a reliability-adjusted score of 0.62. gpt-6-astra follows with 0.63 and 0.55, while gpt-6.1-sol scores 0.59 and 0.46. The mean-score gap between claude-opus-5-5 0.65 and gpt-6-astra 0.63 is 0.02. After adjustment for run-to-run variation, the gap grows to 0.07 0.62 vs 0.55 , indicating greater relative variation in gpt-6-astra’s results. This difference matters when teams compare models for analytical work, because the mean shows average quality while the adjusted score also reflects how consistently a model performs. Why I look beyond the mean The October leaderboard also includes models from the previous measurement, but those models were not tested again for this update. Their scores remain in the chart as reference values, so they provide historical context rather than a direct, same-round comparison with every new model. For example, the previous score for gpt-5.6-sol was a mean of 0.48 and a reliability-adjusted score of 0.46. The new gpt-6-sol scores 0.53 and 0.43. These figures illustrate why mean and repeatability can tell different stories, although they come from different measurement rounds and should not be treated as a controlled head-to-head result. This is one reason I would avoid choosing a model from a single leaderboard number. The mean helps describe average quality, while repeated runs show how much that quality varies. The reliability-adjusted score brings those two signals together, but it still describes performance on this benchmark, not every task a model might face. Reading benchmark results in context A benchmark can help teams compare models on a defined set of tasks, and repeated runs can make that comparison more informative. The results still need to be read with the evaluation setup in mind: what tasks were used, how often each model ran, what the score measures and whether every model was tested in the same round. The complete EDA Benchmark update https://deepsense.ai/blog/eda-benchmark-leaderboard-october-28th-2026/?utm source=Substsack&utm medium=Post 05 11 2026 2&utm campaign=Eval includes the full leaderboard and scoring details. The tasks, configuration and evaluation artifacts are available in the EDA Benchmark GitHub repository https://github.com/deepsense-ai/eda-benchmark . The public benchmark is one example of evaluation work at deepsense.ai. We also build custom datasets and evaluation environments https://deepsense.ai/tech-expertise/llms-rag/custom-synthetic-datasets-for-llm-vlm-evaluation-and-training/?utm source=Substsack&utm medium=Post 05 11 2026 3&utm campaign=Eval for teams assessing LLMs, VLMs and agentic systems against specific capabilities and goals.