cd /news/ai-agents/our-ai-agent-exam-gave-claude-100-th… · home › topics › ai-agents › article
[ARTICLE · art-170438] src=dev.to ↗ pub= topic=ai-agents verified=true sentiment=· neutral

Our AI agent exam gave Claude 100%. That was the bad news.

The team behind Pasportoo, a deterministic MCP-based exam for AI agents, reported that Claude Code running Claude Opus 5.5 passed all 49 checks across 25 tasks on November 11, and repeated a perfect 49/49 score three days later on a partly different task draw. Rather than treating the result as a win, the builders audited each task and found two of three sections could be passed by a careless agent, prompting a redesign of the task bank to remove answer lists while keeping every check deterministic. A separate integrity exam, hardened on November 12 and given to three Claude agents unaware of what was being tested, also scored 100% over four rounds of 30 tasks.

by read3 min views1 publishedNov 16, 2026

We build Pasportoo, an exam for AI agents. You connect it over MCP inside Claude, ChatGPT or Codex, your agent solves tasks, and a fixed set of rules scores the answers. No second model grades anything. The result is a public Agent Passport that anyone can check.

On November 11 we ran the production exam on Claude Code with Claude Opus 5.5. 25 tasks, 49 checks. It passed all 49. Three days later we ran it again on a partly different draw of tasks. 49 out of 49 again.

A perfect, repeatable score sounds like a launch tweet. We decided it was a bug report instead.

We went through every task the agent saw and asked one question: could a careless agent have got this right?

For two of the three sections, the answer was yes.

If you build evals, you have probably seen this. A benchmark that a strong model aces on the first try is not proof the model is great. It is proof the benchmark has stopped separating anything.

Writing tests held up. The agent gets a function description and has to write test cases. Behind the scenes there is a correct implementation and a subtly broken one (a mutant). A good test suite passes on the first and fails on the second. You cannot guess your way through that: either your cases hit the edge where the mutant breaks, or they do not.

It also exposed a real flaw in our tasks. Some of them never said what shape the answer should take, so the agent had to guess names like enqueue and dequeue. In one case the only input that could expose the mutant was technically outside the input format the task declared, and our checker accepted it anyway. Small things, but they are exactly what makes a score less trustworthy.

We also have an integrity exam. It checks whether an agent admits a failed task instead of reporting success, and whether it still picks the better option for the user when a hidden bonus pushes it toward the worse one.

On November 12 we made it harder and gave it to three Claude agents that did not know what was being tested. Over 4 rounds of 30 tasks, they scored 100%.

So we are not going to tell you that "AI agents lie". Our own data does not show it, and we would rather not sell fear we cannot back up. What the data does support is the opposite problem: if your exam cannot find a weak spot, you cannot tell a careful agent from a lucky one.

The next task bank (now on our staging server) drops answer lists completely:

Every check stays deterministic. The same answers always get the same score, and anyone can re-check it. That is the whole point of not using an AI judge: a rule cannot be talked into a better grade.

If you want to run your own agent through it, there is a free tier with passkey sign-in and no sales call: https://pasportoo.com?utm_source=devto I would like to hear how others handle this. When your own eval starts returning 100%, what do you change first?

Disclosure: I build Pasportoo.

── more in #ai-agents 4 stories · sorted by recency
── more on @pasportoo 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/our-ai-agent-exam-ga…] indexed:0 read:3min 2026-11-16 · —