{"slug": "which-ai-models-stay-consistent-across-repeated-runs", "title": "Which AI models stay consistent across repeated runs?", "summary": "Deepsense.ai released an October update to its EDA Benchmark, which runs language models and agentic systems five times each across 10 synthetic exploratory data analysis tasks to measure both average quality and run-to-run consistency. claude-opus-5-5 topped the leaderboard with a mean score of 0.65 and a reliability-adjusted score of 0.62, ahead of gpt-6-astra at 0.63 and 0.55, a gap that widens from 0.02 to 0.07 once variation across runs is factored in. The tasks, configuration and evaluation artifacts are published in the project's GitHub repository.", "body_md": "By [Rafał Łabędzki, PhD](https://deepsense.ai/blog/author/rafal-labedzki/), Senior AI Engineering Manager at deepsense.ai and Assistant Professor at [SGH Warsaw School of Economics](https://www.sgh.waw.pl/knop/en/about-institute-human-capital)\n\nI have worked with AI for more than a decade, starting in the video game industry. In recent years, my work has focused on building evaluation and training environments for machine-learning models, alongside studying human-AI collaboration.\n\n## What the EDA Benchmark measures\n\nThe EDA Benchmark evaluates how language models and agentic systems handle exploratory data analysis tasks. These tasks can require a model to inspect files, clean inconsistent records, combine data sources, detect anomalies and return structured results.\n\nThe benchmark includes 10 synthetic tasks across different domains. We run each model on every task five times, which gives us 50 trajectories per model. Repeated runs matter because one strong answer does not show whether the system can produce a similar result again.\n\nWe report two scores. The **mean score** shows average analytical performance. The **reliability-adjusted score** also accounts for variation across runs, so a model receives a lower score when its results are less consistent.\n\nThe benchmark uses [synthetic data and evaluation methods](https://deepsense.ai/blog/synthetic-data-and-evaluation-methods-for-the-next-generation-of-ai-models?utm_source=Substsack&utm_medium=Post_05_11_2026&utm_campaign=Eval) designed to test both answer quality and repeatability.\n\n## A new group at the top\n\nIn the October round, claude-opus-5-5 leads with a mean score of 0.65 and a reliability-adjusted score of 0.62. gpt-6-astra follows with 0.63 and 0.55, while gpt-6.1-sol scores 0.59 and 0.46.\n\nThe mean-score gap between claude-opus-5-5 (0.65) and gpt-6-astra (0.63) is 0.02. After adjustment for run-to-run variation, the gap grows to 0.07 (0.62 vs 0.55), indicating greater relative variation in gpt-6-astra’s results. This difference matters when teams compare models for analytical work, because the mean shows average quality while the adjusted score also reflects how consistently a model performs.\n\n## Why I look beyond the mean\n\nThe October leaderboard also includes models from the previous measurement, but those models were not tested again for this update. Their scores remain in the chart as reference values, so they provide historical context rather than a direct, same-round comparison with every new model.\n\nFor example, the previous score for gpt-5.6-sol was a mean of 0.48 and a reliability-adjusted score of 0.46. The new gpt-6-sol scores 0.53 and 0.43. These figures illustrate why mean and repeatability can tell different stories, although they come from different measurement rounds and should not be treated as a controlled head-to-head result.\n\nThis is one reason I would avoid choosing a model from a single leaderboard number. The mean helps describe average quality, while repeated runs show how much that quality varies. The reliability-adjusted score brings those two signals together, but it still describes performance on this benchmark, not every task a model might face.\n\n## Reading benchmark results in context\n\nA benchmark can help teams compare models on a defined set of tasks, and repeated runs can make that comparison more informative. The results still need to be read with the evaluation setup in mind: what tasks were used, how often each model ran, what the score measures and whether every model was tested in the same round.\n\nThe complete [EDA Benchmark update](https://deepsense.ai/blog/eda-benchmark-leaderboard-october-28th-2026/?utm_source=Substsack&utm_medium=Post_05_11_2026_2&utm_campaign=Eval) includes the full leaderboard and scoring details. The tasks, configuration and evaluation artifacts are available in the [EDA Benchmark GitHub repository](https://github.com/deepsense-ai/eda-benchmark).\n\nThe public benchmark is one example of evaluation work at deepsense.ai. We also build [custom datasets and evaluation environments](https://deepsense.ai/tech-expertise/llms-rag/custom-synthetic-datasets-for-llm-vlm-evaluation-and-training/?utm_source=Substsack&utm_medium=Post_05_11_2026_3&utm_campaign=Eval) for teams assessing LLMs, VLMs and agentic systems against specific capabilities and goals.", "url": "https://wpnews.pro/news/which-ai-models-stay-consistent-across-repeated-runs", "canonical_source": "https://deepsenseai.substack.com/p/which-ai-models-stay-consistent-across", "published_at": "2026-11-05 16:12:39+00:00", "updated_at": "2026-11-05 16:20:19.074295+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-research", "mlops"], "entities": ["deepsense.ai", "claude-opus-5-5", "gpt-6-astra", "gpt-6.1-sol", "gpt-6-sol", "gpt-5.6-sol", "Rafał Łabędzki", "SGH Warsaw School of Economics"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/which-ai-models-stay-consistent-across-repeated-runs", "markdown": "https://wpnews.pro/news/which-ai-models-stay-consistent-across-repeated-runs.md", "text": "https://wpnews.pro/news/which-ai-models-stay-consistent-across-repeated-runs.txt", "jsonld": "https://wpnews.pro/news/which-ai-models-stay-consistent-across-repeated-runs.jsonld"}}