{"slug": "the-silent-failures-in-agentic-data-science", "title": "The Silent Failures in Agentic Data Science", "summary": "A large-scale arXiv study of agentic data science (arXiv:2610.14115v1) found that the majority of agentic solutions contain at least one failure mode across 220 data science tasks, regardless of the complexity of the agentic harness or model capacity. The authors report a positive correlation between data leakage failures and predictive performance, and their qualitative comparison against Kaggle competition-winning data scientists shows DS agents generally lack problem-solving creativity and domain knowledge. The paper proposes a reusable, extensible test suite covering six specific failure modes and argues that principled failure-mode analysis should be an integral part of agentic DS evaluations.", "body_md": "arXiv:2610.14115v1 Announce Type: new \nAbstract: Large language models have significantly lowered the barrier to writing code. Naturally, this has also led to the development of data science agents that promise automating the design and execution of data science from task formulation to model development. Existing evaluations of agentic data science (agentic DS) assess downstream predictive performance and code correctness, providing preliminary insight in agentic DS capabilities. We argue that these evaluations overlook the fundamental principles of data science: methodological data science- and domain knowledge. As such, only assessing downstream performance masks methodological failures such as data leakage and inappropriate validation, limiting the development of robust agentic DS systems. In this paper, we evaluate agentic DS beyond accuracy from three different angles: 1) a large-scale analysis of failure modes in common agentic DS frameworks and models across 220 data science tasks, 2) a reusable and extensible test suite with curated data science tasks that test for six specific failures in a controlled manner, and 3) a qualitative assessment of agentic DS pipelines against expert solutions on Kaggle competitions. We find that the majority of agentic solutions contain at least one failure mode despite the complexity of agentic harness or model capacity. Moreover, we identify a positive correlation between the occurrence of data leakage failures and predictive performance, confirming the necessity to evaluate agents beyond accuracy. Our qualitative analysis shows that DS agents generally lack problem solving creativity and domain knowledge, compared to competition-winning data scientists. We believe that principled failure mode analysis, as presented here, should be integral part of agentic DS evaluations, and that the insights we surface will be key in developing trustworthy agentic DS pipelines.", "url": "https://wpnews.pro/news/the-silent-failures-in-agentic-data-science", "canonical_source": "https://arxiv.org/abs/2610.14115", "published_at": "2026-10-15 04:00:00+00:00", "updated_at": "2026-10-15 05:43:40.320344+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "ai-safety", "machine-learning"], "entities": ["arXiv", "Kaggle"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-silent-failures-in-agentic-data-science", "markdown": "https://wpnews.pro/news/the-silent-failures-in-agentic-data-science.md", "text": "https://wpnews.pro/news/the-silent-failures-in-agentic-data-science.txt", "jsonld": "https://wpnews.pro/news/the-silent-failures-in-agentic-data-science.jsonld"}}