{"slug": "can-llms-be-trusted-as-eda-analysts-read-our-paper", "title": "Can LLMs be trusted as EDA analysts? Read our paper", "summary": "A new study evaluating 15 LLM configurations across 8 model families for exploratory data analysis in business settings found that most are not reliable enough for autonomous use. GPT-5.4 with extra-high reasoning effort achieved the best performance, with an average score of 0.8748 and a business utility metric of 0.6952, but variability in outputs undermines trust. The researchers propose a risk-adjusted metric to assess operational trustworthiness beyond average performance.", "body_md": "# Computer Science > Computers and Society\n\n[Submitted on 8 May 2026]\n\n# Title:Business Utility of Large Language Models as Exploratory Data Analysis Agents\n\n[View PDF](/pdf/2606.00051)\n\n[HTML (experimental)](https://arxiv.org/html/2606.00051v1)\n\nAbstract:Large Language Models (LLMs) are increasingly used in analytical workflows, but their suitability as exploratory data analysis (EDA) agents in business settings remains uncertain. In practice, a deployable EDA agent must provide not only useful average performance but also sufficient repeatability to support trust in its outputs. We evaluate this requirement in a controlled, business-relevant benchmark built on an agent-based supply chain simulation. The task is to identify supplier-product combinations responsible for low quality and downstream sales loss by reasoning from indirect operational traces rather than from explicit labels. Fifteen model-variant configurations from eight model families were evaluated under four experimental conditions that varied data representation, prompt clarity, and signal strength, with five trajectories per condition. Outputs were scored against deterministic ground truth using the Jaccard index and assessed through a framework that combines mean score (ms), coefficient of variation (CV), exploratory cross-condition significance tests, and Business utility, a risk-adjusted metric that we propose to summarise quality and repeatability in a single operational measure. The results show that most configurations are not reliable enough for autonomous EDA use, even when their average scores appear acceptable. GPT-5.4 with extra-high reasoning effort achieved the strongest overall profile, with an experiment-averaged ms of 0.8748 and an experiment-averaged Business utility of 0.6952, while the next-best configurations lost substantially more utility after variability discounting. Our findings suggest that evaluation of EDA agents should treat average quality, repeatability, and condition sensitivity as complementary dimensions of operational trustworthiness.\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer\n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers\n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps\n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations\n\n*(*[What are Smart Citations?](https://www.scite.ai/))# Code, Data and Media Associated with this Article\n\nalphaXiv\n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers\n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub\n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub\n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face\n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast\n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower\n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender\n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [ Learn more about arXivLabs](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/can-llms-be-trusted-as-eda-analysts-read-our-paper", "canonical_source": "https://arxiv.org/abs/2606.00051", "published_at": "2026-06-17 14:08:10+00:00", "updated_at": "2026-06-17 14:25:26.783009+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-safety"], "entities": ["GPT-5.4", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/can-llms-be-trusted-as-eda-analysts-read-our-paper", "markdown": "https://wpnews.pro/news/can-llms-be-trusted-as-eda-analysts-read-our-paper.md", "text": "https://wpnews.pro/news/can-llms-be-trusted-as-eda-analysts-read-our-paper.txt", "jsonld": "https://wpnews.pro/news/can-llms-be-trusted-as-eda-analysts-read-our-paper.jsonld"}}