{"slug": "xyeval-agents-say-yes-to-bad-advice", "title": "XYEval: Agents say yes to bad advice", "summary": "A paper submitted to arXiv on 20 Sep 2026 introduces XYEval, a meta-evaluation framework that converts existing benchmarks into tests of whether AI agents resist plausible but misleading user suggestions, the so-called XY problem. Evaluating five models across six benchmark suites, the authors found relative performance drops of up to 46.7% under XY mutation, and on τ²-bench agents performed worse when a pedantic user demanded detailed explanations before approving a better solution. A simple system instruction encouraging awareness of XY problems offered only partial mitigation, and the authors conclude current agents lack the ability to reason and communicate effectively when facing misleading suggestions.", "body_md": "# Computer Science > Computation and Language\n\n  [Submitted on 20 Sep 2026]\n\n# Title:XYEval: Agents say yes to bad advice\n\n[View PDF](https://arxiv.org/pdf/2609.23939)\n\n[HTML (experimental)](https://arxiv.org/html/2609.23939v1)\n\nAbstract:Effective communication between users and AI agents is essential for human-AI collaboration. The XY problem is a well-known communication pitfall where a person asks about their attempted solution rather than their actual problem. We extend prior sycophancy evaluation to the XY problem in agentic settings, evaluating whether agents can resist plausible but misleading suggestions from users and communicate their reasoning. We introduce XYEval, a meta-evaluation framework that can transform an existing benchmark into an XY problem evaluation. We evaluate five models across six diverse benchmark suites. Agents suffer large XY drops under XY mutation across benchmarks, with relative drops reaching up to 46.7%. With $\\tau^2$-bench, we further show that agent performance drops more when encountering a pedantic user who requires detailed explanations before approving a better solution. Our findings suggest that current agents lack the ability to effectively reason and communicate when facing misleading suggestions. A simple system instruction baseline that encourages awareness of XY problems only offers partial mitigation. Extensive trace analyses provide behavioral insights into how and why these XY drops occur across execution trajectories. Our results show that mitigating the XY problem remains challenging, requiring agents to both recognize user misdirection and clearly communicate the underlying problem.\n    \n\n### Additional Features\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/xyeval-agents-say-yes-to-bad-advice", "canonical_source": "https://arxiv.org/abs/2609.23939", "published_at": "2026-10-22 00:00:39+00:00", "updated_at": "2026-10-22 00:12:23.676344+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "ai-safety", "large-language-models"], "entities": ["XYEval", "arXiv", "τ²-bench"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/xyeval-agents-say-yes-to-bad-advice", "markdown": "https://wpnews.pro/news/xyeval-agents-say-yes-to-bad-advice.md", "text": "https://wpnews.pro/news/xyeval-agents-say-yes-to-bad-advice.txt", "jsonld": "https://wpnews.pro/news/xyeval-agents-say-yes-to-bad-advice.jsonld"}}