Claude Code's eval grader can't see a no-break space

Driftproof found that LLM-as-judge graders in LangChain's OpenEvals, Promptfoo, NVIDIA's SkillEvaluator and Claude Code's built-in `claude plugin eval` command misgrade code answers that differ only by invisible Unicode characters such as no-break spaces, scoring as few as 4 of 36 correct calls. Adding a short note listing each unusual character and its exact position, without altering the answer, raised accuracy to 34 of 36 and 6 of 6 in the Claude Code case, while a generic Unicode warning did not. The company has offered the fix upstream as an opt-in option, with a pull request to NVIDIA's SkillEvaluator approved by the maintainer but not yet merged.

Here are two answers to the same coding question. One is right. Which one? The task: Write a Python function accepts s: str - bool . It must return True exactly when s consists of zero or more ordinary ASCII spaces U+0020 , then the ASCII text "42", then zero or more ordinary ASCII spaces U+0020 . Every other string must return False. Other whitespace characters are not permitted. Answer A php def accepts s: str - bool: return s.strip " " == "42" assert accepts "42" is True assert accepts " 42 " is True assert accepts " 42 " is False assert accepts "43" is False Answer B php def accepts s: str - bool: return s.strip " " == "42" assert accepts "42" is True assert accepts " 42 " is True assert accepts " 42 " is True assert accepts "43" is False Most people pick B. A's tests seem to say that " 42 " should pass and also that " 42 " should fail, so A looks broken. A is correct. The third test in each answer doesn't use ordinary spaces. It uses no-break spaces U+00A0 , the character typesetters use to keep "28 October" on one line. On screen it looks exactly like a space. The task says to refuse it. A refuses it, so its test says False. B quietly accepts it. There is an interactive version on our site https://driftproofhq.com/invisible-space/ with an X-ray switch that lights up every hidden character on the page, including one in the headline. Many evaluation tools use a second AI as the grader: it reads an answer and decides pass or fail. We gave these two answers to the graders inside three popular open-source tools, LangChain's OpenEvals, Promptfoo and NVIDIA's SkillEvaluator, with Claude Haiku 5.5 and Claude Sonnet 5.5 doing the grading. | | Right calls | |---|---| | Reading the answers as written | 4 of 36 | | With a list of the hidden characters added | 34 of 36 | | Anthropic's skill-creator grader, which runs the code | 12 of 12 | In the explanations we read, a common pattern was this: the grader saw two tests in A that seemed to check the same thing and expect opposite results, and decided A contradicted itself. One rejected A because its tests used "identical visible inputs" and said running them would raise an error. Running them works fine. Claude Code's own eval command has the same blind spot. Its built-in claude plugin eval grader, judging the same two answers, got 0 of 6 right with Haiku 5.5 and 1 of 6 with Sonnet 5.5. We then wrote the identical code with the no-break space spelled out as \u00a0 , which Python reads as exactly the same program. Both graders got 6 of 6. The only thing that changed was whether the grader could see the character. Driftproof https://driftproofhq.com/ measures whether the instructions people give AI coding assistants actually make them better. Before any paid run we do a small practice round to check our own grader. In the practice round for our next report, one coding question turned on exactly this. Our graders got it backwards every time: 0 of 6 correct calls each. We rebuilt the question from scratch to see whether the tools other people use behave the same way. They did. The fix is deliberately small. Alongside the answer, a short note lists any unusual spaces and invisible characters, with their exact position: Character facts not a verdict : line 6, col 17: U+00A0 NO-BREAK SPACE line 6, col 20: U+00A0 NO-BREAK SPACE The answer is never changed, and the note doesn't say the character is wrong, because sometimes it is exactly right. Tested through OpenEvals' own code, a general "watch out for Unicode" warning left Haiku at 0 of 6 and got Sonnet to 3 of 6. The exact list got both to 6 of 6, and 72 control checks, including legitimate Unicode that must be kept, stayed correct. We've offered it upstream as an opt-in option: | Project | Status | |---|---| | NVIDIA SkillEvaluator 242 https://github.com/NVIDIA/SkillEvaluator/pull/242 | Approved by the maintainer, not yet merged | | OpenEvals 293 https://github.com/langchain-ai/openevals/pull/293 | Awaiting review | | Promptfoo 12005 https://github.com/promptfoo/promptfoo/pull/12005 | Awaiting review | | Claude Code 105497 https://github.com/anthropics/claude-code/issues/105497 | Reported, no fix proposed | For our own next report, we now grade coding answers mainly by running hidden tests that include the tricky characters, with an AI grader only for what tests can't check. Full write-up, with the interactive quiz, methods and limits https://driftproofhq.com/invisible-space/ I maintain Driftproof, the open-source tool where this turned up. Drafted with AI assistance; every number is from our run reports and was checked against them.