By Igor Eduardo · Austin, TX · https://igoreduardo.com A new workshop paper, Do Verifier Judgments Survive Abstraction? Measuring Verification Transport for LLM Agent Evaluation (arXiv 2611.18471, posted 2026-11-13), tests an assumption buried in almost every agent eval pipeline: that a verifier which judges an agent run correctly will keep judging it correctly when it sees the same run in a different form.
That assumption matters because of how agent systems are built now. A verifier (an LLM that reads an agent's trajectory and decides whether it did the right thing) is usually validated on a benchmark's clean, step-by-step trace. In production, the same execution reaches the verifier through layers:
None of these change what the agent actually did. They change what the verifier can see. The paper calls the property it measures Verification Transport: does a correct verdict survive when only the view of the execution changes?
The authors froze real trajectories from four public agent-evaluation benchmarks (process correctness, safety, pairwise preference and interactive agent behavior; 250, 758, 441 and 754 eligible instances), and built three matched views of each one: native, abstracted, and restored. Task, trajectory, outcome, ground truth, criterion and verifier are held fixed. Only the evidence exposed to the verifier changes. They ran seven verifier models across several families under four verification protocols: direct judgment, evidence-grounded judgment, an agentic verifier that can request provenance, and a deliberative second-pass audit.
Counting only judgments that were correct on the native view:
Scope, in their words: three controlled abstraction boundaries, frozen trajectories rather than abstractions that arise during live execution, restoration that assumes the original provenance is available, and a metric that measures preservation of initially correct verdicts, not overall verifier competence. Results may differ in other domains and verifier designs. Those are their numbers, not mine. I have not re-run the experiments, and I am deliberately not naming the verifier models or ranking them.
Teams are about to treat "our LLM judge agrees with labels on the benchmark" as "our LLM judge can grade our agent".
Those are different claims. The first was measured on a trace format your production stack does not produce. The second depends on what your harness actually forwards: the wrapper or the calls inside it, the sub-agent's return value or its work, the summary or the steps it replaced. If a third of correct verdicts can flip or vanish when only the view changes, a native-view accuracy number tells you surprisingly little about the verifier you deployed.
And it compounds. Verifier verdicts don't just sit in dashboards. They pick prompts, filter training data, gate releases and trigger safety stops. A verifier that silently loses evidence at a delegation boundary will reward agents that delegate the risky part.
An agent verifier's accuracy is a property of the verifier and the view it was given, so any reported verifier accuracy should name the trajectory representation it was measured on, and a verifier should be re-validated on every view your production harness actually produces.
That is a preference with a limiter, not a benchmark result. If your agent is a short, flat, single-agent loop with no tool composition, no sub-agents and no summarization, native-view validation may be enough. The moment you add any of those three layers, you have changed the input distribution of your grader.
The detail I would steal first is the split between wrong verdict and no verdict.
A verifier that answers "insufficient evidence" more often under abstraction looks safer than one that confidently guesses. Sometimes it is. But in a pipeline, a missing verdict has to go somewhere: it gets defaulted to pass, defaulted to fail, dropped from the denominator, or sent to a human queue nobody has staffed. Each of those choices changes your reported success rate without anything about the agent changing.
So "my evidence-grounded verifier has fewer wrong verdicts" is only half a claim. The other half is "and here is what happens to the runs it refused to grade". The paper's numbers make this concrete: the protocols with the fewest wrong verdicts under abstraction were also the ones that stopped answering most often.
| Column | Question it answers | Failure if skipped |
|---|---|---|
| Native-view accuracy | Does the verifier agree with labels on the clean trace? | No baseline at all |
| Per-view accuracy | Does it still agree on composed, delegated and compacted views of the same runs? | Benchmark accuracy credited to a production grader |
| Wrong vs. no-verdict split | When it fails, does it guess wrong or refuse? | Abstention silently counted as a pass or dropped |
| Abstention routing | Where do refused runs go, and who reviews them? | A hidden default decides your success rate |
| Provenance recovery | Does forwarding the inner steps fix the flips? | You can't tell missing evidence from a weak verifier |
| Harness diff | Which layer (wrapper, sub-agent, summarizer) hides what? | Regressions blamed on the agent, caused by the view |
None of that needs a new model or a private harness. It needs matched views of runs you already have, and the discipline to report the verifier's accuracy per view instead of once.
My public work is mostly retrieve-first and evaluation-as-contract systems, and the habit here is the one I keep coming back to: an evidence path is part of the system under test. In retrieval, an answer can only be audited if the passages that produced it travel with it. For agents, this paper shows the same thing one layer up. A verdict can only be trusted if the execution evidence that justifies it reached the verifier. If your harness drops it, the verifier is grading a story about the run, not the run.
Pick one agent whose runs are graded by an LLM verifier and add four things before your next release:
Then ask whether your verifier's headline accuracy still describes the verifier you are running.
A verifier doesn't grade a run. It grades the view of the run you handed it. Validate it on every view your harness produces, count refusals separately from wrong answers, and forward the evidence the verdict depends on.
Igor Eduardo builds retrieve-first and evaluation-as-contract systems. More at igoreduardo.com.