{"slug": "claude-code-s-eval-grader-can-t-see-a-no-break-space", "title": "Claude Code's eval grader can't see a no-break space", "summary": "Driftproof found that LLM-as-judge graders in LangChain's OpenEvals, Promptfoo, NVIDIA's SkillEvaluator and Claude Code's built-in `claude plugin eval` command misgrade code answers that differ only by invisible Unicode characters such as no-break spaces, scoring as few as 4 of 36 correct calls. Adding a short note listing each unusual character and its exact position, without altering the answer, raised accuracy to 34 of 36 and 6 of 6 in the Claude Code case, while a generic Unicode warning did not. The company has offered the fix upstream as an opt-in option, with a pull request to NVIDIA's SkillEvaluator approved by the maintainer but not yet merged.", "body_md": "Here are two answers to the same coding question. One is right. Which one?\n\n**The task:** Write a Python function `accepts(s: str) -> bool`. It must return True exactly when `s` consists of zero or more ordinary ASCII spaces (U+0020), then the ASCII text \"42\", then zero or more ordinary ASCII spaces (U+0020). Every other string must return False. Other whitespace characters are not permitted.\n\n**Answer A**\n\n``` php\ndef accepts(s: str) -> bool:\n    return s.strip(\" \") == \"42\"\n\nassert accepts(\"42\") is True\nassert accepts(\" 42 \") is True\nassert accepts(\" 42 \") is False\nassert accepts(\"43\") is False\n```\n\n**Answer B**\n\n``` php\ndef accepts(s: str) -> bool:\n    return s.strip(\"  \") == \"42\"\n\nassert accepts(\"42\") is True\nassert accepts(\" 42 \") is True\nassert accepts(\" 42 \") is True\nassert accepts(\"43\") is False\n```\n\nMost people pick B. A's tests seem to say that `\" 42 \"` should pass and also that `\" 42 \"` should fail, so A looks broken.\n\nA is correct. The third test in each answer doesn't use ordinary spaces. It uses no-break spaces (U+00A0), the character typesetters use to keep \"28 October\" on one line. On screen it looks exactly like a space. The task says to refuse it. A refuses it, so its test says False. B quietly accepts it.\n\nThere is an [interactive version on our site](https://driftproofhq.com/invisible-space/) with an X-ray switch that lights up every hidden character on the page, including one in the headline.\n\nMany evaluation tools use a second AI as the grader: it reads an answer and decides pass or fail. We gave these two answers to the graders inside three popular open-source tools, LangChain's OpenEvals, Promptfoo and NVIDIA's SkillEvaluator, with Claude Haiku 5.5 and Claude Sonnet 5.5 doing the grading.\n\n|  | Right calls | \n|---|---|\n| Reading the answers as written | 4 of 36 | \n| With a list of the hidden characters added | 34 of 36 | \n| Anthropic's skill-creator grader, which runs the code | 12 of 12 | \n\nIn the explanations we read, a common pattern was this: the grader saw two tests in A that seemed to check the same thing and expect opposite results, and decided A contradicted itself. One rejected A because its tests used \"identical visible inputs\" and said running them would raise an error. Running them works fine.\n\n**Claude Code's own eval command has the same blind spot.** Its built-in `claude plugin eval` grader, judging the same two answers, got 0 of 6 right with Haiku 5.5 and 1 of 6 with Sonnet 5.5. We then wrote the identical code with the no-break space spelled out as `\\u00a0`, which Python reads as exactly the same program. Both graders got 6 of 6. The only thing that changed was whether the grader could see the character.\n\n[Driftproof](https://driftproofhq.com/) measures whether the instructions people give AI coding assistants actually make them better. Before any paid run we do a small practice round to check our own grader. In the practice round for our next report, one coding question turned on exactly this. Our graders got it backwards every time: 0 of 6 correct calls each. We rebuilt the question from scratch to see whether the tools other people use behave the same way. They did.\n\nThe fix is deliberately small. Alongside the answer, a short note lists any unusual spaces and invisible characters, with their exact position:\n\n```\nCharacter facts (not a verdict):\nline 6, col 17: U+00A0 NO-BREAK SPACE\nline 6, col 20: U+00A0 NO-BREAK SPACE\n```\n\nThe answer is never changed, and the note doesn't say the character is wrong, because sometimes it is exactly right. Tested through OpenEvals' own code, a general \"watch out for Unicode\" warning left Haiku at 0 of 6 and got Sonnet to 3 of 6. The exact list got both to 6 of 6, and 72 control checks, including legitimate Unicode that must be kept, stayed correct.\n\nWe've offered it upstream as an opt-in option:\n\n| Project | Status | \n|---|---|\n| [NVIDIA SkillEvaluator #242](https://github.com/NVIDIA/SkillEvaluator/pull/242) | Approved by the maintainer, not yet merged | \n| [OpenEvals #293](https://github.com/langchain-ai/openevals/pull/293) | Awaiting review | \n| [Promptfoo #12005](https://github.com/promptfoo/promptfoo/pull/12005) | Awaiting review | \n| [Claude Code #105497](https://github.com/anthropics/claude-code/issues/105497) | Reported, no fix proposed | \n\nFor our own next report, we now grade coding answers mainly by running hidden tests that include the tricky characters, with an AI grader only for what tests can't check.\n\n[Full write-up, with the interactive quiz, methods and limits](https://driftproofhq.com/invisible-space/)\n\n*I maintain Driftproof, the open-source tool where this turned up. Drafted with AI assistance; every number is from our run reports and was checked against them.*", "url": "https://wpnews.pro/news/claude-code-s-eval-grader-can-t-see-a-no-break-space", "canonical_source": "https://dev.to/driftproofhq/claude-codes-eval-grader-cant-see-a-no-break-space-5d8c", "published_at": "2026-10-27 20:07:16+00:00", "updated_at": "2026-10-27 20:12:16.708757+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "large-language-models", "ai-agents"], "entities": ["Driftproof", "Anthropic", "Claude Code", "Claude Haiku 5.5", "Claude Sonnet 5.5", "LangChain", "OpenEvals", "Promptfoo"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/claude-code-s-eval-grader-can-t-see-a-no-break-space", "markdown": "https://wpnews.pro/news/claude-code-s-eval-grader-can-t-see-a-no-break-space.md", "text": "https://wpnews.pro/news/claude-code-s-eval-grader-can-t-see-a-no-break-space.txt", "jsonld": "https://wpnews.pro/news/claude-code-s-eval-grader-can-t-see-a-no-break-space.jsonld"}}