{"slug": "xybench-helping-with-the-problem-behind-the-question", "title": "XYBench: Helping with the problem behind the question", "summary": "A benchmark of 8,115 real-world questions, XYBench, found that Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro often answer users' stated requests while staying focused on the mistaken approach behind them, unlike human reference answers. The benchmark, built with researchers at Carnegie Mellon, NVIDIA and Johns Hopkins from 1,557 Stack Overflow questions, 184 from Stack Exchange, 102 manually curated examples and 6,272 generated from WikiHow guides, won the Best Resource Paper Award at EMNLP 2026. XYBench targets the XY problem, where a user asks for help with an assumed approach (Y) instead of the underlying goal (X), and measures whether models identify the misconception and prioritize solutions to the actual goal.", "body_md": "Truly useful AI assistants need to be able to help users who ask misguided questions—not just users who ask the right question with the right amount of detail. We’re releasing XYBench, a benchmark of 8K+ questions that tests how AI assistants respond to requests based on misconceptions, such as asking how to thicken frosting that keeps sliding off cupcakes when the real problem is that the cupcakes are still too warm to frost. XYBench won the Best Resource Paper Award at EMNLP 2026.\n\nA mismatch between the approach someone asks about and the goal they want to achieve is known as the XY problem: a person wants to accomplish X, assumes a particular approach Y will work, and asks for help with Y. In the cupcake example, the person is so committed to making the frosting stick that they fail to consider the warmth of the cupcake as the true problem to be addressed.\n\nDeveloped with researchers at Carnegie Mellon, NVIDIA, and Johns Hopkins, XYBench exposes a critical gap in LLMs' ability to troubleshoot problems: recognizing when a user's request is based on an implicit misconception and appropriately redirecting them toward a solution that addresses their actual goal. Drawing on 8,115 real-world questions spanning programming and everyday activities such as cooking and home repair, we introduce a benchmark that measures whether models identify these misconceptions and are able to prioritize effective solutions in their responses.\n\nThe models we evaluated on XYBench – Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro – often suggested a solution to the user’s problem but kept their answers focused on the original, mistaken approach. Unlike human reference answers, all five models were less likely to steer users away from that approach and toward a solution suited to their goal.\n\nPeople use AI systems with different levels of knowledge and may not always know how best to approach a problem. We hope XYBench helps researchers measure whether model improvements make assistants better at recognizing mistaken assumptions, explaining why a proposed approach might not work, and suggesting more effective alternatives—even when users don't know the right questions to ask.\n\nBuilding XYBench\n\nWe built XYBench from four sources: 1,557 questions from Stack Overflow, 184 from Stack Exchange, 102 manually curated examples, and 6,272 questions generated from WikiHow guides. For each question, we identified the stated request (Y), what the person explicitly asks for, and the intended request (X)—the underlying goal that can be inferred from context.\n\nFor the public forum questions, we used the three highest-scored human answers and the accepted answer, where available, to establish the intended request and possible solutions. We filtered out cases where answering the stated request was judged sufficient. For WikiHow, we used Sonnet 4.5 to generate questions around plausible misconceptions in procedural guides, then refined them to include enough context for an expert to infer the underlying goal without stating it outright. We removed examples where answering the literal request would largely resolve the problem.\n\nEvaluating how an answer helps\n\nA model might spend several paragraphs recommending a more powerful vacuum cleaner, then briefly suggest checking for a clogged hose. If a clog is causing the weak suction, clearing it could solve the problem.\n\nA more helpful answer would recognize that the user wants to restore suction and focus on diagnosing the cause. This draws on pragmatics, the branch of linguistics concerned with how context and communicative intent shape meaning. To test whether models use this understanding in their answers, XYBench measures three aspects of each response:\n\nPragmatic solution—records whether a model’s response includes solutions to the stated request (Y) and the intended request (X), checking each separately.\n\nPragmatic emphasis—measures whether a model’s answer focuses on the stated request (Y), focuses on the intended request (X), or balances the two.\n\nMisconception identification—measures whether a model’s response steers the user toward the underlying goal (X) while discouraging or avoiding the mistaken approach (Y).\n\nIn XYBench, Claude Sonnet 4.5 serves as the automated judge. It classifies individual suggestions within each model’s response and assesses the answer’s pragmatic emphasis.\n\nTo validate Sonnet 4.5’s judgments, two human reviewers independently assessed 500 model responses to XYBench questions. Agreement with the judge, averaged across the reviewers, ranged from 97.5–100% on whether a solution appeared, 84.6–93.3% on misconception identification, and 74.2–79.8% on pragmatic emphasis.\n\nWhat the experiments show\n\nWe evaluated Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro on XYBench in a single-turn setting, with one response per question. We evaluated human reference answers separately using the same metrics.\n\nAcross all four sources, every model was more likely to include a solution to the stated request (Y) than to the underlying goal (X). On Stack Overflow, stated-request solutions appeared in 75–86% of responses, compared with 59–71% for underlying-goal solutions.\n\nA passing mention of a solution does not make it the focus of an answer. In the vacuum example, a model could briefly suggest unclogging the hose but focus most of its response on buying a more powerful vacuum. We would count that response as containing a solution to the underlying goal, even though the model emphasized the stated request. We measured overall emphasis separately: on Stack Overflow, models focused primarily on the underlying goal in only 35–41% of their responses, and on the stated request in 47–57%. In the remaining responses, models gave similar emphasis to the stated request and the underlying goal.\n\nFive models and top human answers on XYBench's Stack Overflow questions\n\nMentions a fix for the underlying goal\n\n(e.g., unclog the vacuum hose)\n\nGPT-5\n\n71%\n\nTop human answer\n\n67%\n\nDeepSeek-V4 Pro\n\n66%\n\nQwen3.5-9B\n\n64%\n\nGemini 3.1 Pro\n\n62%\n\nClaude Sonnet 4.6\n\n59%\n\nFocuses its answer on the underlying goal\n\n(e.g., most of its advice is about unclogging the vacuum hose)\n\nTop human answer\n\n55%\n\nGPT-5\n\n41%\n\nQwen3.5-9B\n\n41%\n\nGemini 3.1 Pro\n\n40%\n\nClaude Sonnet 4.6\n\n36%\n\nDeepSeek-V4 Pro\n\n35%\n\nSteers users to the goal, not the mistaken approach\n\n(e.g., unclog the vacuum hose rather than buy a new one)\n\nTop human answer\n\n79%\n\nGPT-5\n\n60%\n\nQwen3.5-9B\n\n60%\n\nDeepSeek-V4 Pro\n\n55%\n\nGemini 3.1 Pro\n\n54%\n\nClaude Sonnet 4.6\n\n47%\n\nWe also tested whether making the user’s goal explicit would help. In an oracle experiment, we supplied both the stated request (Y) and the intended request (X) alongside each question. On Stack Overflow, Sonnet 4.6’s rate of including a solution to the underlying goal rose from 59% to 67%, but its misconception-identification rate fell from 47% to 34%. Across the sources with human reference answers, every model in this experiment still trailed humans on misconception identification. Supplying intent helped, but did not consistently close the gap with human responses. This suggests that detecting intent is only part of the challenge: models also need to act on it when it diverges from the stated request.\n\nA separate experiment tested whether models could identify useful advice when judging responses instead of writing them. We gave the same five models, plus GPT-4.1, each question with two possible answers: one focused on the literal request and one explaining the misconception and addressing the underlying goal. On Stack Overflow, all six chose the goal-focused answer in 71–84% of cases. By comparison, only 35–41% of the answers written by the five models in our main evaluation primarily focused on the underlying goal. The results expose a gap between recognizing useful advice and producing it, which a multiple-choice test alone would miss.\n\nPicking and writing goal-focused answers on XYBench's Stack Overflow questions\n\nPicks the goal-focused answer when shown two\n\n(e.g., unclogging the vacuum hose over buying a new vacuum)\n\nDeepSeek-V4 Pro\n\n84%\n\nQwen3.5-9B\n\n83%\n\nGemini 3.1 Pro\n\n81%\n\nGPT-5\n\n72%\n\nClaude Sonnet 4.6\n\n71%\n\nRandom pick (50%)\n\nFocuses its answer on the underlying goal\n\n(e.g., most of its advice is about unclogging the vacuum hose)\n\nGPT-5\n\n41%\n\nQwen3.5-9B\n\n41%\n\nGemini 3.1 Pro\n\n40%\n\nClaude Sonnet 4.6\n\n36%\n\nDeepSeek-V4 Pro\n\n35%\n\nWhat this means for evaluation\n\nResearchers can use XYBench to test whether changes to prompting or training help models identify misconceptions, offer solutions to users’ underlying problems, and make those solutions the focus of their answers. Measuring each behavior individually lets researchers see whether progress in one also improves the others.\n\nFuture work could test which training methods help models act on inferred intent—AKA what users need and not just what they literally asked for. The XYBench currently covers English-language queries and mainly procedural problems; extending it to other languages, cultures, and conversational settings remains an open direction.\n\nWe hope XYBench enables researchers to build models that offer useful advice even when a user’s first question rests on a mistaken assumption.\n\nJoin us\n\nAt Ai2 we’re building the future of transparent, open-source AI — built in the open to empower scientific progress and fundamental understanding of this world changing technology. We’re not here to make profits, we’re here to make sure benefits of AI are shared widely and for the benefit of humanity. If this appeals to you, please take a look at our open roles.", "url": "https://wpnews.pro/news/xybench-helping-with-the-problem-behind-the-question", "canonical_source": "https://allenai.org/blog/xybench", "published_at": "2026-11-13 08:00:00+00:00", "updated_at": "2026-11-13 17:23:35.716394+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-safety", "natural-language-processing"], "entities": ["XYBench", "Carnegie Mellon", "NVIDIA", "Johns Hopkins", "Claude Sonnet 4.6", "GPT-5", "Gemini 3.1 Pro", "DeepSeek-V4 Pro"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/xybench-helping-with-the-problem-behind-the-question", "markdown": "https://wpnews.pro/news/xybench-helping-with-the-problem-behind-the-question.md", "text": "https://wpnews.pro/news/xybench-helping-with-the-problem-behind-the-question.txt", "jsonld": "https://wpnews.pro/news/xybench-helping-with-the-problem-behind-the-question.jsonld"}}