{"slug": "the-ai-was-right-the-answer-was-still-wrong", "title": "The AI Was Right. The Answer Was Still Wrong.", "summary": "A developer built a benchmark that separately measures task correctness and instruction compliance for coding prompts given to multiple AI models, scoring compliance as the fraction of individual constraints followed (e.g., 3 of 4 requirements = 75%). The benchmark is designed to surface patterns such as whether models struggle more with multiple constraints and which instruction types are most often ignored, testing whether a model can be technically correct yet practically wrong.", "body_md": "Have you ever asked AI to do something very specific and it did almost everything right?\n\nExcept the one thing you specifically told it **not** to do. \n\nThat made me curious.\n\nInstead of asking **“Which AI model is smartest?”** I wanted to test something much more specific:\n\n**Can AI actually follow small instructions while solving a task?**\n\nI created a benchmark focused on **instruction following in coding tasks**.\n\nFor example:\n\nFix this JavaScript function.\n\nDon't use `map()`.\n\nReturn only the corrected code.\n\nThe model isn't only expected to fix the bug.\n\nIt also needs to follow every instruction.\n\nSo I check:\n\nBecause sometimes **the answer is correct but the task isn't.**\n\nThe testing flow is simple:\n\n**Same prompt → Multiple models → Same evaluation → Compare**\n\nFor every task, I:\n\nThis keeps the test focused on the model's behavior rather than changing the task for each model.\n\nI didn't want a simple **right/wrong** score.\n\nA response can solve the coding problem while still breaking one of the instructions.\n\nSo I measure two things separately:\n\n**1. Task Correctness**\n\nDid the model actually solve the underlying coding problem?\n\n**2. Instruction Compliance**\n\nDid it follow the individual constraints in the prompt?\n\nFor example if a task contains four requirements and the model follows three:\n\n**3 / 4 = 75% instruction compliance**\n\nAcross the benchmark these scores can show whether a model is not only capable of solving the problem but also capable of **doing exactly what was asked**.\n\nAI is becoming a bigger part of everyday coding.\n\nWe often focus on whether a model can generate working code but real developer tasks usually come with constraints:\n\n“Don't change this.”\n\n“Use this approach.”\n\n“Return only the code.”\n\n“Don't use this method.”\n\nThose tiny instructions can completely change whether the response is actually useful.\n\nThat's the behavior I wanted to investigate.\n\nThe interesting part isn't simply finding a model with the highest number.\n\nI'm looking for **patterns**.\n\nDo models struggle more when there are several constraints?\n\nDoes solving the code correctly make instruction following more reliable?\n\nWhich types of instructions are most commonly ignored?\n\nAnd can a model be **technically right while still being practically wrong?**\n\nThose are the questions this benchmark is designed to explore.\n\nIf I expand the benchmark I'd add:\n\nThat could reveal whether instruction following remains reliable as tasks become more realistic.\n\nNow I'm curious about your experience.\n\n**What's the smallest instruction you've given an AI that it completely ignored?** 😭\n\nDrop it in the comments.\n\nMaybe your most frustrating AI moment becomes the next benchmark task.", "url": "https://wpnews.pro/news/the-ai-was-right-the-answer-was-still-wrong", "canonical_source": "https://dev.to/akanksha_sharma/the-ai-was-right-the-answer-was-still-wrong-2pl4", "published_at": "2026-09-25 14:37:12+00:00", "updated_at": "2026-09-25 15:00:57.053889+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-tools"], "entities": [], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-ai-was-right-the-answer-was-still-wrong", "markdown": "https://wpnews.pro/news/the-ai-was-right-the-answer-was-still-wrong.md", "text": "https://wpnews.pro/news/the-ai-was-right-the-answer-was-still-wrong.txt", "jsonld": "https://wpnews.pro/news/the-ai-was-right-the-answer-was-still-wrong.jsonld"}}