Self-Verification with DeepSeek V4 Flash Beats Claude Fable 5 on Terminal-Bench

LLM-as-a-Verifier, a framework for fine-grained agent feedback, reports that self-verification with DeepSeek V4 Flash outperforms Claude Fable 5 on Terminal-Bench 2.1, achieving 86.5% ± 1.1% Pass@1 for Best-of-3 and 88.0% ± 0.6% for Best-of-5, compared to Pass@1 baselines of 79.4% and 78.7%, respectively, with oracle scores of 92.1% and 96.6%. The framework, available via pip install llm-verifier, also introduces version 0.2.0 with prefix-cache optimization reducing uncached input tokens by ~3.4× on trajectory-heavy benchmarks.

| Documentation https://llm-as-a-verifier.com/docs/ | | https://llm-as-a-verifier.com Website | https://arxiv.org/abs/2607.05391 Paper | https://github.com/llm-as-a-verifier/TurboAgent Claude Code Plugin | https://x.com/jackyk02/status/2042347578139033628 Twitter/X | https://join.slack.com/t/llm-as-a-verifier/shared invite/zt-3utx6oe8m-86ACBqtPGfsOnpOoMJQwng Slack 🔥 LLM-as-a-Verifier achieves SOTA performance across agentic benchmarks, including Terminal-Bench, SWE-Bench Verified, MedAgentBench, RoboRewardBench and more. We invite the community to contribute more use cases pip install llm-verifier To install the latest from a clone: pip install -e . What's new in 0.2.0 full notes in CHANGELOG.md /llm-as-a-verifier/llm-as-a-verifier/blob/main/CHANGELOG.md : - Prefix-cache optimization: ~3.4× fewer uncached input tokens on trajectory-heavy benchmarks - Terminal-Bench 2.1 self-verification benchmark self-verification-terminal-bench-21 deepseek-v4-flash verifier backend- Token accounting llm verifier.token usage LLM-as-a-Verifier is a general-purpose framework that provides fine-grained feedback for any agent. The key idea is simple: 1 use fine-grained scoring granularity, 2 take the expectation over the full logprob distribution of LLM score tokens, and 3 scale repeated evaluation and criteria decomposition. The resulting fine-grained feedback can be used for test-time scaling, progress tracking, and reinforcement learning. Run a first end-to-end selection requires DEEPSEEK API KEY or VERTEX API KEY in .env , or an OpenAI-compatible server that returns logprobs — e.g. vllm serve Qwen/Qwen3.5-9B with OPENAI BASE URL=http://localhost:8000/v1 : python import llm verifier problem = "Write a function that reverses a string." candidates = "def rev s : return s ::-1 ", "def rev s : return s", "def rev s : return ''.join sorted s ", result = llm verifier.select problem=problem, candidates=candidates, criteria={"Correctness": "Does the code actually reverse the string?"}, print result.index index of the best candidate: 0 print result.scores candidate scores: 0.73104, 0.38446, 0.38449 select is built on a pairwise reward model. For the raw fine-grained rewards of a single comparison, call compare : reward a, reward b = llm verifier.compare problem, candidates 0 , candidates 1 , criteria={"Overall": "Does the code solve the problem?"}, print reward a, reward b fine-grained rewards in 0, 1 : 0.99994 0 The same fine-grained reward can also score an agent's progress after each step with track : python steps = 'Read the problem statement', 'Wrote def rev s : return s ', 'Tested: rev "abc" returned "abc"', 'Changed to def rev s : return s ::-1 ', 'Tested: rev "abc" returned "cba"', result = llm verifier.track problem=problem, steps=steps, checkpoint steps= 1, 2, 3, 4, 5 , n evaluations=4 print result.scores progress after each step: 0.00106, 0.02417, 0.03143, 0.62004, 0.99978 Can a model verify its own rollouts? On Terminal-Bench 2.1 we generate 5 mini-swe-agent trajectories per task with deepseek-v4-flash and use the same model as the verifier. Selection lands well above Pass@1 even though the verifier is judging its own model's work: | Config | Pass@1 | LLM-as-a-Verifier | Oracle | |---|---|---|---| | Best-of-3 | 79.4% | 86.5% ± 1.1% | 92.1% | | Best-of-5 | 78.7% | 88.0% ± 0.6% | 96.6% | The trajectories ship in data/terminal bench 2.1 trajs/ ; scoring only needs DEEPSEEK API KEY in .env . Each configuration has its own reproduction script: python scripts/run bo3.py best-of-3 python scripts/run bo5.py best-of-5 Each benchmark ships with its agent trajectories data/ . We use Gemini 2.5 Flash gemini-2.5-flash as the verifier for all benchmarks below. Expected results: | Benchmark | Base Model | Harness | Pass@1 | LLM-as-a-Verifier | Oracle | |---|---|---|---|---|---| | Terminal-Bench V2 | GPT-5.5 Best-of-5 | Capy | 83.1% | 86.5% | 92.1% | | SWE-Bench Verified | Opus 4.5 / Opus 4.6 / Gemini 3 Flash Best-of-3 | mini-swe-agent | 76.1% | 78.2% | 84.4% | | MedAgentBench | Claude Opus 4.8 Best-of-5 | AgentBench | 70.2% | 73.3% | 75.0% | Run a benchmark by name python scripts/run.py with no argument lists them : python scripts/run.py terminal bench python scripts/run.py swe bench python scripts/run.py medagentbench The tournament defaults can be overridden on the command line: python scripts/run.py swe bench --pivots 2 --n-evaluations 8 --seed 0 --max-workers 50 Benchmarks are defined in llm verifier/benchmarks.py — add or tweak one there. Given a task and a pool of agent trajectories, pick the best one in a few lines of code. python import llm verifier problem = "Fix the failing test in utils.py." candidates = traj 1, traj 2, traj 3, traj 4, traj 5 result = llm verifier.select problem=problem, candidates=candidates, criteria={"Root cause": "Did the agent fix the real cause?", "Verification": "Did the agent confirm the fix?"}, model="gemini-2.5-flash", verifier model n evaluations=4, repeated evaluations per criterion pivots=2, pivots < N; reduced verification cost print "Best candidate:", result.index print "Ranking:", result.ranking Under the hood, select runs the Probabilistic Pivot Tournament probabilistic-pivot-tournament to rank all N trajectories using O Nk pairwise verifications instead of a full O N² round-robin. pivots trades cost for accuracy: more pivots = more comparisons = higher accuracy. Use the verifier for your own task in three steps — Claude Code does the rest generates the criteria, writes a runner, and selects the best-of-N for you : Add your data. Copy your agent trajectories into data/task name trajs/ . Update naming. Replace every task name inwith the name of your task. add new benchmark.md Spin up Claude Code in this repo or Codex, or whatever you like — with permissions disabled and paste the contents of add new benchmark.md to let it run. The same fine-grained reward can score a trajectory at every step see track in the Quickstart fine-grained-progress-tracking . Below, we track two Terminus-2 runs of the Terminal-Bench task pytorch-model-cli . The successful trajectory exhibits consistently increasing verifier scores, whereas the failed trajectory is characterized by erroneous behaviors, resulting in lower scores throughout the execution. Reproduce it with: python scripts/terminal bench progress.py scores both runs then plots track scores a finished trajectory. To monitor an agent while it runs , use ProgressTracker : feed it each step as it happens and get a live progress score back — e.g. to stop a hopeless rollout early or decide when to resample. Since the verifier only ever sees the steps so far, it cannot peek at the future. tracker = llm verifier.ProgressTracker problem, n evaluations=4 score = tracker.update 'Read the problem statement' 0.00002 score = tracker.update 'Wrote def rev s : return s' 0.00013 score = tracker.update 'Changed to def rev s : return s ::-1 ' 0.73938 score = tracker.update 'Tested: rev "abc" returned "cba"' 0.98604 if score < 0.05: after any step: abandon a hopeless rollout early ... Replay the two Terminal-Bench trajectories step-by-step through ProgressTracker — printing a live score bar after every step, as an agent harness would see it: python scripts/terminal bench progress.py --online With a multimodal verifier model e.g. Gemini 2.5 Flash or vllm serve Qwen/Qwen3.5-9B , every entry point accepts images — a single image images="frame.png" or a list of images, each a local file path, an http s URL, or raw bytes: result = llm verifier.select problem, candidates, criteria=criteria, images= "before.png", "after.png" tracker = llm verifier.ProgressTracker problem score = tracker.update step, images="camera frame.png" per-step frame Per-step frames stay part of the trajectory for all later updates, so the verifier always sees the full visual history — e.g. camera frames while tracking a robot rollout. See the multimodal documentation https://llm-as-a-verifier.com/docs/multimodal/image inputs.html for accepted input forms, backend notes, and verified examples. TurboAgent https://github.com/llm-as-a-verifier/TurboAgent brings LLM-as-a-Verifier to Claude Code https://claude.com/claude-code as a drop-in LLM API proxy. It sits between your client and the model provider, generating multiple candidate responses in parallel and selecting the best one with a Probabilistic Pivot Tournament probabilistic-pivot-tournament . pip install git+https://github.com/llm-as-a-verifier/TurboAgent Point Claude Code at the proxy and run as usual: turbo-agent starts on port 8888 ANTHROPIC BASE URL=http://localhost:8888 claude It ships a built-in visualizer at http://localhost:8888/visualizer that shows the pipeline DAG, progress scores, candidate responses, and the final selection. See the TurboAgent repository https://github.com/llm-as-a-verifier/TurboAgent for configuration and setup details. . ├── scripts/ command-line entry points │ ├── run.py registry-driven benchmark launcher │ ├── run bo3.py reproduce the best-of-3 self-verification run │ ├── run bo5.py reproduce the best-of-5 self-verification run │ └── terminal bench progress.py re-score + plot the progress-tracking example ├── criteria/ verifier criteria + ground-truth notes │ ├── TEMPLATE.md copy this to write your own │ ├── terminal bench.md │ ├── swe bench.md │ └── medagentbench.md ├── llm verifier/ the reusable framework import llm verifier │ ├── init .py llm verifier.select ... / .compare ... │ ├── main .py python -m llm verifier <file.md : preview criteria │ ├── benchmarks.py BENCHMARKS registry one Benchmark / launch │ ├── fine grained reward.py R x,τ : logprob scoring + score cache │ ├── progress.py llm verifier.track ... : per-step progress curve │ ├── pivot tournament.py PPT: O Nk selection Bradley-Terry │ ├── prompts.py load criteria/ .md + normalize criteria args │ └── loaders.py per-benchmark trajectory loaders └── data/ agent trajectories per benchmark Runs write their verifier score caches to cache/ and result tables to results/ ; both are created on demand and git-ignored. Rather than reducing each distribution into a single discrete score as in LLM-as-a-Judge , LLM-as-a-Verifier approximates the reward of a trajectory - $C$ = number of evaluation criteria - $K$ = number of repeated verifications - $G$ = number of score tokens granularity level - $p {\theta} v g \mid x, c, \tau $ = probability assigned by model$\theta$ to score token$v g$ - $\phi v g $ = maps each scoring token to a scalar value - $V {\text{score}} = {v 1, \ldots, v G}$ = ordered set of discrete score tokens This lives in llm verifier/fine grained reward.py . To pick the best of N candidate trajectories, a round-robin tournament scores all O N² . Probabilistic Pivot Tournament PPT is a cost efficient ranking algorithm in which every candidate is compared only against a small set of pivots, reducing the budget from - Candidates: the pool${\tau 1,\dots,\tau N}$ to be ranked. - Ring pass: a random Hamiltonian cycle scores the$N$ adjacent pairs so every candidate appears once in the "A" slot and once in "B", canceling the model's positional bias. - Pivot selection: candidates are ranked by their ring-pass scores$w { i }$ , and the top-$k$ candidates form the pivot set$\mathcal{P}$ . - Pivot tournament: every non-pivot–vs–pivot and pivot–vs–pivot pair is scored via the pairwise preference$p a \succ b = \sigma R a - R b $ , concentrating the budget on uncertain top candidates and cutting cost from$\mathcal{O} N^2 $ to$\mathcal{O} Nk $ . Repeated evaluations of a pair alternate the A/B prompt slots, so positional bias cancels here as well. - Selection: comparisons are aggregated into win mass$w i$ and count$c i$ , and the candidate with the highest normalized$w i/c i$ is returned. This lives in llm verifier/pivot tournament.py . You are an expert domain reviewer. You will see a task description and two trajectories. Evaluation Criteria: domain specific criteria Task: {task prompt} Trajectory A: {A} Trajectory B: {B} Carefully analyze each trajectory, then provide your final scores: <score A INTEGER 1 TO 20 </score A <score B INTEGER 1 TO 20 </score B Rating Rules: Rate correctness on a 1-20 scale based on evaluation criteria 1 = incorrect, 10 = borderline, 20 = correct You are an evaluator of domain agent attempts. Trust observed output — NOT the agent's narration. Task: {task prompt} Agent trajectory {N} steps : {trajectory} You will score the trajectory at {N} checkpoints. Given everything the agent has done up to and including this step, would the agent's CURRENT state already complete the task? Score each checkpoint INDEPENDENTLY, then output exactly N lines: <c1 INTEGER 1 TO 20 </c1 ... <cN INTEGER 1 TO 20 </cN Rating Rules: Rate completion on a 1-20 scale 1 = certainly not complete, 10 = uncertain, 20 = verified complete Note: we use a letter-based scale A-T instead of digits in the actual implementation to enable logprob extraction for granularity scaling. Each verification prompt carries two full trajectories ~80k tokens on Terminal-Bench 2.1 and is re-scored per criterion and repeat, so on a backend that caches prompt prefixes almost all of that input can be reused. Two things make it happen: the prompt keeps the criterion at the tail , so everything before it task, both trajectories, rating scale is a shared prefix, and scoring warms one request per distinct prefix to completion before fanning out the rest. Together these take the cache hit rate from 5.2% to 78.4% on terminal bench 2.1 , cutting uncached input tokens by ~3.4×. Every verifier call records what it was billed for, so the cache hit rate above is measured rather than assumed. scripts/run.py prints the totals under the result table and writes them to results/<benchmark .txt : Verifier tokens 4,320 verifier calls input 272,551,552 cached input 214,712,320 78.8% hit rate uncached input 57,839,232 output 32,441,600 reasoning 26,102,144 Only calls this run actually made are counted — comparisons served from the score cache add nothing. Reasoning tokens are a subset of output tokens, and cached input is a subset of input. The counter is process-wide and thread-safe, so library users get the same numbers out of select / compare / track : python import llm verifier llm verifier.USAGE.reset result = llm verifier.select problem, trajectories, criteria="terminal bench" print llm verifier.token usage {'calls': 24, 'input tokens': 1512480, 'cached input tokens': 1190208, 'uncached input tokens': 322272, 'output tokens': 180224, 'reasoning tokens': 145408, 'cache hit rate': 0.787} llm verifier.USAGE is a TokenUsage : .snapshot for the dict above, .reset to zero it, and format usage ... for the report block. Counts come from the backend's own usage block; a backend that reports no usage simply contributes zeros. If you find this work useful, please cite: @misc{kwok2026llmasaverifiergeneralpurposeverificationframework, title={LLM-as-a-Verifier: A General-Purpose Verification Framework}, author={Jacky Kwok and Shulu Li and Pranav Atreya and Yuejiang Liu and Yixing Jiang and Chelsea Finn and Marco Pavone and Ion Stoica and Azalia Mirhoseini}, year={2026}, eprint={2607.05391}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2607.05391}, }