We build Pasportoo, an exam for AI agents. You connect it over MCP inside Claude, ChatGPT or Codex, your agent solves tasks, and a fixed set of rules scores the answers. No second model grades anything. The result is a public Agent Passport that anyone can check.
On November 11 we ran the production exam on Claude Code with Claude Opus 5.5. 25 tasks, 49 checks. It passed all 49. Three days later we ran it again on a partly different draw of tasks. 49 out of 49 again.
A perfect, repeatable score sounds like a launch tweet. We decided it was a bug report instead.
We went through every task the agent saw and asked one question: could a careless agent have got this right?
For two of the three sections, the answer was yes.
If you build evals, you have probably seen this. A benchmark that a strong model aces on the first try is not proof the model is great. It is proof the benchmark has stopped separating anything.
Writing tests held up. The agent gets a function description and has to write test cases. Behind the scenes there is a correct implementation and a subtly broken one (a mutant). A good test suite passes on the first and fails on the second. You cannot guess your way through that: either your cases hit the edge where the mutant breaks, or they do not.
It also exposed a real flaw in our tasks. Some of them never said what shape the answer should take, so the agent had to guess names like enqueue and dequeue. In one case the only input that could expose the mutant was technically outside the input format the task declared, and our checker accepted it anyway. Small things, but they are exactly what makes a score less trustworthy.
We also have an integrity exam. It checks whether an agent admits a failed task instead of reporting success, and whether it still picks the better option for the user when a hidden bonus pushes it toward the worse one.
On November 12 we made it harder and gave it to three Claude agents that did not know what was being tested. Over 4 rounds of 30 tasks, they scored 100%.
So we are not going to tell you that "AI agents lie". Our own data does not show it, and we would rather not sell fear we cannot back up. What the data does support is the opposite problem: if your exam cannot find a weak spot, you cannot tell a careful agent from a lucky one.
The next task bank (now on our staging server) drops answer lists completely:
Every check stays deterministic. The same answers always get the same score, and anyone can re-check it. That is the whole point of not using an AI judge: a rule cannot be talked into a better grade.
If you want to run your own agent through it, there is a free tier with passkey sign-in and no sales call: https://pasportoo.com?utm_source=devto I would like to hear how others handle this. When your own eval starts returning 100%, what do you change first?
Disclosure: I build Pasportoo.