The AI Was Right. The Answer Was Still Wrong.

A developer built a benchmark that separately measures task correctness and instruction compliance for coding prompts given to multiple AI models, scoring compliance as the fraction of individual constraints followed (e.g., 3 of 4 requirements = 75%). The benchmark is designed to surface patterns such as whether models struggle more with multiple constraints and which instruction types are most often ignored, testing whether a model can be technically correct yet practically wrong.

Have you ever asked AI to do something very specific and it did almost everything right? Except the one thing you specifically told it not to do. That made me curious. Instead of asking “Which AI model is smartest?” I wanted to test something much more specific: Can AI actually follow small instructions while solving a task? I created a benchmark focused on instruction following in coding tasks . For example: Fix this JavaScript function. Don't use map . Return only the corrected code. The model isn't only expected to fix the bug. It also needs to follow every instruction. So I check: Because sometimes the answer is correct but the task isn't. The testing flow is simple: Same prompt → Multiple models → Same evaluation → Compare For every task, I: This keeps the test focused on the model's behavior rather than changing the task for each model. I didn't want a simple right/wrong score. A response can solve the coding problem while still breaking one of the instructions. So I measure two things separately: 1. Task Correctness Did the model actually solve the underlying coding problem? 2. Instruction Compliance Did it follow the individual constraints in the prompt? For example if a task contains four requirements and the model follows three: 3 / 4 = 75% instruction compliance Across the benchmark these scores can show whether a model is not only capable of solving the problem but also capable of doing exactly what was asked . AI is becoming a bigger part of everyday coding. We often focus on whether a model can generate working code but real developer tasks usually come with constraints: “Don't change this.” “Use this approach.” “Return only the code.” “Don't use this method.” Those tiny instructions can completely change whether the response is actually useful. That's the behavior I wanted to investigate. The interesting part isn't simply finding a model with the highest number. I'm looking for patterns . Do models struggle more when there are several constraints? Does solving the code correctly make instruction following more reliable? Which types of instructions are most commonly ignored? And can a model be technically right while still being practically wrong? Those are the questions this benchmark is designed to explore. If I expand the benchmark I'd add: That could reveal whether instruction following remains reliable as tasks become more realistic. Now I'm curious about your experience. What's the smallest instruction you've given an AI that it completely ignored? 😭 Drop it in the comments. Maybe your most frustrating AI moment becomes the next benchmark task.