{"slug": "self-verification-with-deepseek-v4-flash-beats-claude-fable-5-on-terminal-bench", "title": "Self-Verification with DeepSeek V4 Flash Beats Claude Fable 5 on Terminal-Bench", "summary": "LLM-as-a-Verifier, a framework for fine-grained agent feedback, reports that self-verification with DeepSeek V4 Flash outperforms Claude Fable 5 on Terminal-Bench 2.1, achieving 86.5% ± 1.1% Pass@1 for Best-of-3 and 88.0% ± 0.6% for Best-of-5, compared to Pass@1 baselines of 79.4% and 78.7%, respectively, with oracle scores of 92.1% and 96.6%. The framework, available via pip install llm-verifier, also introduces version 0.2.0 with prefix-cache optimization reducing uncached input tokens by ~3.4× on trajectory-heavy benchmarks.", "body_md": "| [ Documentation](https://llm-as-a-verifier.com/docs/) |\n\n[|](https://llm-as-a-verifier.com)\n\n**Website**[|](https://arxiv.org/abs/2607.05391)\n\n**Paper**[|](https://github.com/llm-as-a-verifier/TurboAgent)\n\n**Claude Code Plugin**[|](https://x.com/jackyk02/status/2042347578139033628)\n\n**Twitter/X**[|](https://join.slack.com/t/llm-as-a-verifier/shared_invite/zt-3utx6oe8m-86ACBqtPGfsOnpOoMJQwng)\n\n**Slack**🔥 LLM-as-a-Verifier achieves SOTA performance across agentic benchmarks, including Terminal-Bench, SWE-Bench Verified, MedAgentBench, RoboRewardBench and more. We invite the community to contribute more use cases!\n\n```\npip install llm-verifier\n```\n\nTo install the latest from a clone:\n\n```\npip install -e .\n```\n\n**What's new in 0.2.0** (full notes in [CHANGELOG.md](/llm-as-a-verifier/llm-as-a-verifier/blob/main/CHANGELOG.md)):\n\n- Prefix-cache optimization: ~3.4× fewer uncached input tokens on trajectory-heavy benchmarks\n- Terminal-Bench 2.1\n[self-verification benchmark](#self-verification-terminal-bench-21) `deepseek-v4-flash`\n\nverifier backend- Token accounting (\n`llm_verifier.token_usage()`\n\n)\n\nLLM-as-a-Verifier is a general-purpose framework that provides **fine-grained\nfeedback** for any agent. The key idea is simple: 1) use fine-grained scoring\ngranularity, 2) take the expectation over the full logprob distribution of LLM\nscore tokens, and 3) scale repeated evaluation and criteria decomposition. The\nresulting fine-grained feedback can be used for test-time scaling, progress\ntracking, and reinforcement learning.\n\nRun a first end-to-end selection (requires `DEEPSEEK_API_KEY`\n\nor `VERTEX_API_KEY`\n\nin `.env`\n\n, or an OpenAI-compatible server that returns\nlogprobs — e.g. `vllm serve Qwen/Qwen3.5-9B`\n\nwith\n`OPENAI_BASE_URL=http://localhost:8000/v1`\n\n):\n\n``` python\nimport llm_verifier\n\nproblem = \"Write a function that reverses a string.\"\ncandidates = [\n    \"def rev(s): return s[::-1]\", \"def rev(s): return s\", \"def rev(s): return ''.join(sorted(s))\",\n]\n\nresult = llm_verifier.select(\n    problem=problem,\n    candidates=candidates,\n    criteria={\"Correctness\": \"Does the code actually reverse the string?\"},\n)\nprint(result.index)   # index of the best candidate: 0\nprint(result.scores)  # candidate scores: [0.73104, 0.38446, 0.38449]\n```\n\n`select`\n\nis built on a pairwise reward model. For the raw fine-grained rewards\nof a single comparison, call `compare`\n\n:\n\n```\nreward_a, reward_b = llm_verifier.compare(\n    problem, candidates[0], candidates[1],\n    criteria={\"Overall\": \"Does the code solve the problem?\"},\n)\nprint(reward_a, reward_b)   # fine-grained rewards in [0, 1]: 0.99994 0\n```\n\nThe same fine-grained reward can also score an agent's progress after each\nstep with `track`\n\n:\n\n``` python\nsteps = [\n    'Read the problem statement',\n    'Wrote def rev(s): return s ',\n    'Tested: rev(\"abc\") returned \"abc\"',\n    'Changed to def rev(s): return s[::-1]',\n    'Tested: rev(\"abc\") returned \"cba\"',\n]\n\nresult = llm_verifier.track(problem=problem, steps=steps,\n                            checkpoint_steps=[1, 2, 3, 4, 5], n_evaluations=4)\nprint(result.scores)  # progress after each step: [0.00106, 0.02417, 0.03143, 0.62004, 0.99978]\n```\n\nCan a model verify its own rollouts? On Terminal-Bench 2.1 we generate 5\nmini-swe-agent trajectories per task with `deepseek-v4-flash`\n\nand use the\n**same model** as the verifier. Selection lands well above Pass@1 even though\nthe verifier is judging its own model's work:\n\n| Config | Pass@1 | LLM-as-a-Verifier | Oracle |\n|---|---|---|---|\n| Best-of-3 | 79.4% | 86.5% ± 1.1% |\n92.1% |\n| Best-of-5 | 78.7% | 88.0% ± 0.6% |\n96.6% |\n\nThe trajectories ship in `data/terminal_bench_2.1_trajs/`\n\n; scoring only needs\n`DEEPSEEK_API_KEY`\n\nin `.env`\n\n. Each configuration has its own reproduction\nscript:\n\n```\npython scripts/run_bo3.py                    # best-of-3\npython scripts/run_bo5.py                    # best-of-5\n```\n\nEach benchmark ships with its agent trajectories (`data/`\n\n). We use Gemini 2.5\nFlash (`gemini-2.5-flash`\n\n) as the verifier for all benchmarks below. Expected\nresults:\n\n| Benchmark | Base Model | Harness | Pass@1 | LLM-as-a-Verifier | Oracle |\n|---|---|---|---|---|---|\n| Terminal-Bench V2 | GPT-5.5 (Best-of-5) | Capy | 83.1% | 86.5% |\n92.1% |\n| SWE-Bench Verified | Opus 4.5 / Opus 4.6 / Gemini 3 Flash (Best-of-3) | mini-swe-agent | 76.1% | 78.2% |\n84.4% |\n| MedAgentBench | Claude Opus 4.8 (Best-of-5) | AgentBench | 70.2% | 73.3% |\n75.0% |\n\nRun a benchmark by name (`python scripts/run.py`\n\nwith no argument lists them):\n\n```\npython scripts/run.py terminal_bench\npython scripts/run.py swe_bench\npython scripts/run.py medagentbench\n```\n\nThe tournament defaults can be overridden on the command line:\n\n```\npython scripts/run.py swe_bench --pivots 2 --n-evaluations 8 --seed 0 --max-workers 50\n```\n\nBenchmarks are defined in `llm_verifier/benchmarks.py`\n\n— add or tweak one there.\n\nGiven a task and a pool of agent trajectories, pick the best one in a few lines of code.\n\n``` python\nimport llm_verifier\n\nproblem = \"Fix the failing test in utils.py.\"\ncandidates = [traj_1, traj_2, traj_3, traj_4, traj_5]\n\nresult = llm_verifier.select(\n    problem=problem,\n    candidates=candidates,\n    criteria={\"Root cause\": \"Did the agent fix the real cause?\",\n              \"Verification\": \"Did the agent confirm the fix?\"},\n    model=\"gemini-2.5-flash\",          # verifier model\n    n_evaluations=4,                 # repeated evaluations per criterion\n    pivots=2,                          # pivots < N; reduced verification cost\n)\n\nprint(\"Best candidate:\", result.index)            \nprint(\"Ranking:\", result.ranking)\n```\n\nUnder the hood, `select`\n\nruns the\n[Probabilistic Pivot Tournament](#probabilistic-pivot-tournament) to rank all\n`N`\n\ntrajectories using `O(Nk)`\n\npairwise verifications instead of a full\n`O(N²)`\n\nround-robin. `pivots`\n\ntrades cost for accuracy: more pivots = more\ncomparisons = higher accuracy.\n\nUse the verifier for your own task in three steps — Claude Code does the rest (generates the criteria, writes a runner, and selects the best-of-N for you):\n\n**Add your data.** Copy your agent trajectories into`data/task_name_trajs/`\n\n.**Update naming.** Replace every`task_name`\n\ninwith the name of your task.`add_new_benchmark.md`\n\n**Spin up Claude Code in this repo**(or Codex, or whatever you like — with permissions disabled) and paste the contents of`add_new_benchmark.md`\n\nto let it run.\n\nThe same fine-grained reward can score a trajectory *at every step* (see\n[ track in the Quickstart](#fine-grained-progress-tracking)). Below, we track two Terminus-2 runs of the Terminal-Bench task\n\n`pytorch-model-cli`\n\n. The successful trajectory exhibits consistently increasing verifier scores, whereas the failed trajectory is characterized by erroneous behaviors, resulting in lower scores throughout the execution. Reproduce it with:\n\n```\npython scripts/terminal_bench_progress.py    # scores both runs then plots\n```\n\n`track`\n\nscores a **finished** trajectory. To monitor an agent **while it\nruns**, use `ProgressTracker`\n\n: feed it each step as it happens and get a live\nprogress score back — e.g. to stop a hopeless rollout early or decide when to\nresample. Since the verifier only ever sees the steps so far, it cannot peek\nat the future.\n\n```\ntracker = llm_verifier.ProgressTracker(problem, n_evaluations=4)\n\nscore = tracker.update('Read the problem statement')            # 0.00002\nscore = tracker.update('Wrote def rev(s): return s')            # 0.00013\nscore = tracker.update('Changed to def rev(s): return s[::-1]') # 0.73938\nscore = tracker.update('Tested: rev(\"abc\") returned \"cba\"')     # 0.98604\n\nif score < 0.05:      # after any step: abandon a hopeless rollout early\n    ...\n```\n\nReplay the two Terminal-Bench trajectories step-by-step through\n`ProgressTracker`\n\n— printing a live score bar after every step, as an agent\nharness would see it:\n\n```\npython scripts/terminal_bench_progress.py --online\n```\n\nWith a multimodal verifier model (e.g. Gemini 2.5 Flash or\n`vllm serve Qwen/Qwen3.5-9B`\n\n), every\nentry point accepts `images`\n\n— a single image (`images=\"frame.png\"`\n\n) or a\nlist of images, each a local file path, an http(s) URL, or raw bytes:\n\n```\nresult = llm_verifier.select(problem, candidates, criteria=criteria,\n                             images=[\"before.png\", \"after.png\"])\n\ntracker = llm_verifier.ProgressTracker(problem)\nscore = tracker.update(step, images=\"camera_frame.png\")  # per-step frame\n```\n\nPer-step frames stay part of the trajectory for all later updates, so the\nverifier always sees the full visual history — e.g. camera frames while\ntracking a robot rollout. See the\n[multimodal documentation](https://llm-as-a-verifier.com/docs/multimodal/image_inputs.html) for accepted\ninput forms, backend notes, and verified examples.\n\n[TurboAgent](https://github.com/llm-as-a-verifier/TurboAgent) brings\nLLM-as-a-Verifier to [Claude Code](https://claude.com/claude-code) as a drop-in\nLLM API proxy. It sits between your client and the model provider, generating\nmultiple candidate responses in parallel and selecting the best one with a\n[Probabilistic Pivot Tournament](#probabilistic-pivot-tournament).\n\n```\npip install git+https://github.com/llm-as-a-verifier/TurboAgent\n```\n\nPoint Claude Code at the proxy and run as usual:\n\n```\nturbo-agent                                        # starts on port 8888\nANTHROPIC_BASE_URL=http://localhost:8888 claude\n```\n\nIt ships a built-in visualizer at\n`http://localhost:8888/visualizer`\n\nthat shows the pipeline DAG, progress scores, candidate\nresponses, and the final selection. See the\n[TurboAgent repository](https://github.com/llm-as-a-verifier/TurboAgent) for\nconfiguration and setup details.\n\n```\n.\n├── scripts/                     # command-line entry points\n│   ├── run.py                   #   registry-driven benchmark launcher\n│   ├── run_bo3.py               #   reproduce the best-of-3 self-verification run\n│   ├── run_bo5.py               #   reproduce the best-of-5 self-verification run\n│   └── terminal_bench_progress.py  # re-score + plot the progress-tracking example\n├── criteria/                    # verifier criteria + ground-truth notes\n│   ├── TEMPLATE.md              #   copy this to write your own\n│   ├── terminal_bench.md\n│   ├── swe_bench.md\n│   └── medagentbench.md\n├── llm_verifier/                # the reusable framework (import llm_verifier)\n│   ├── __init__.py              #   llm_verifier.select(...) / .compare(...)\n│   ├── __main__.py              #   python -m llm_verifier <file.md>: preview criteria\n│   ├── benchmarks.py            #   BENCHMARKS registry (one Benchmark / launch)\n│   ├── fine_grained_reward.py   #   R(x,τ): logprob scoring + score cache\n│   ├── progress.py              #   llm_verifier.track(...): per-step progress curve\n│   ├── pivot_tournament.py      #   PPT: O(Nk) selection (Bradley-Terry)\n│   ├── prompts.py               #   load criteria/*.md + normalize criteria args\n│   └── loaders.py               #   per-benchmark trajectory loaders\n└── data/                        # agent trajectories per benchmark\n```\n\nRuns write their verifier score caches to `cache/`\n\nand result tables to\n`results/`\n\n; both are created on demand and git-ignored.\n\nRather than reducing each distribution into a single discrete score (as in\nLLM-as-a-Judge), LLM-as-a-Verifier approximates the reward of a trajectory\n\n-\n$C$ = number of evaluation criteria -\n$K$ = number of repeated verifications -\n$G$ = number of score tokens (granularity level) -\n$p_{\\theta}(v_g \\mid x, c, \\tau)$ = probability assigned by model$\\theta$ to score token$v_g$ -\n$\\phi(v_g)$ = maps each scoring token to a scalar value -\n$V_{\\text{score}} = {v_1, \\ldots, v_G}$ = ordered set of discrete score tokens\n\nThis lives in `llm_verifier/fine_grained_reward.py`\n\n.\n\nTo pick the best of `N`\n\ncandidate trajectories, a round-robin tournament scores\nall `O(N²)`\n\n. Probabilistic Pivot Tournament (PPT) is a\ncost efficient ranking algorithm in which every candidate is compared only\nagainst a small set of pivots, reducing the budget from\n\n-\n**Candidates:** the pool${\\tau_1,\\dots,\\tau_N}$ to be ranked. -\n**Ring pass:** a random Hamiltonian cycle scores the$N$ adjacent pairs so every candidate appears once in the \"A\" slot and once in \"B\", canceling the model's positional bias. -\n**Pivot selection:** candidates are ranked by their ring-pass scores$w_{(i)}$ , and the top-$k$ candidates form the pivot set$\\mathcal{P}$ . -\n**Pivot tournament:** every*non-pivot–vs–pivot*and*pivot–vs–pivot*pair is scored via the pairwise preference$p(a \\succ b) = \\sigma(R_a - R_b)$ , concentrating the budget on uncertain top candidates and cutting cost from$\\mathcal{O}(N^2)$ to$\\mathcal{O}(Nk)$ . Repeated evaluations of a pair alternate the A/B prompt slots, so positional bias cancels here as well. -\n**Selection:** comparisons are aggregated into win mass$w_i$ and count$c_i$ , and the candidate with the highest normalized$w_i/c_i$ is returned.\n\nThis lives in `llm_verifier/pivot_tournament.py`\n\n.\n\n```\nYou are an expert [domain] reviewer. You will see a task description and two\ntrajectories.\n\nEvaluation Criteria: [domain specific criteria]\n\nTask: {task prompt}\nTrajectory A: {A}\nTrajectory B: {B}\n\nCarefully analyze each trajectory, then provide your final scores:\n<score_A> INTEGER_1_TO_20 </score_A>\n<score_B> INTEGER_1_TO_20 </score_B>\n\nRating Rules: Rate correctness on a 1-20 scale based on evaluation criteria\n(1 = incorrect, 10 = borderline, 20 = correct)\nYou are an evaluator of [domain] agent attempts. Trust observed output — NOT the agent's narration.\n\nTask: {task prompt}\nAgent trajectory ({N} steps): {trajectory}\n\nYou will score the trajectory at {N} checkpoints. Given everything the agent has done up to and including this step, would the agent's CURRENT state already complete the task?\n\nScore each checkpoint INDEPENDENTLY, then output exactly N lines:\n<c1> INTEGER_1_TO_20 </c1>\n...\n<cN> INTEGER_1_TO_20 </cN>\n\nRating Rules: Rate completion on a 1-20 scale (1 = certainly not complete,\n10 = uncertain, 20 = verified complete)\n```\n\nNote: we use a letter-based scale (A-T) instead of digits in the actual implementation to enable logprob extraction for granularity scaling.\n\nEach verification prompt carries two full trajectories (~80k tokens on\nTerminal-Bench 2.1) and is re-scored per criterion and repeat, so on a backend\nthat caches prompt prefixes almost all of that input can be reused. Two things\nmake it happen: the prompt keeps the criterion at the *tail*, so everything\nbefore it (task, both trajectories, rating scale) is a shared prefix, and\nscoring warms one request per distinct prefix to completion before fanning out\nthe rest. Together these take the cache hit rate from 5.2% to 78.4% on\n`terminal_bench_2.1`\n\n, cutting uncached input tokens by ~3.4×.\n\nEvery verifier call records what it was billed for, so the cache hit rate above\nis measured rather than assumed. `scripts/run.py`\n\nprints the totals under the\nresult table (and writes them to `results/<benchmark>.txt`\n\n):\n\n```\nVerifier tokens (4,320 verifier calls)\n  input                          272,551,552\n    cached input                 214,712,320  (78.8% hit rate)\n    uncached input                57,839,232\n  output                          32,441,600\n    reasoning                     26,102,144\n```\n\nOnly calls this run actually made are counted — comparisons served from the\nscore cache add nothing. Reasoning tokens are a subset of output tokens, and\ncached input is a subset of input. The counter is process-wide and\nthread-safe, so library users get the same numbers out of `select`\n\n/\n`compare`\n\n/ `track`\n\n:\n\n``` python\nimport llm_verifier\n\nllm_verifier.USAGE.reset()\nresult = llm_verifier.select(problem, trajectories, criteria=\"terminal_bench\")\nprint(llm_verifier.token_usage())\n# {'calls': 24, 'input_tokens': 1512480, 'cached_input_tokens': 1190208,\n#  'uncached_input_tokens': 322272, 'output_tokens': 180224,\n#  'reasoning_tokens': 145408, 'cache_hit_rate': 0.787}\n```\n\n`llm_verifier.USAGE`\n\nis a `TokenUsage`\n\n: `.snapshot()`\n\nfor the dict above,\n`.reset()`\n\nto zero it, and `format_usage(...)`\n\nfor the report block. Counts\ncome from the backend's own usage block; a backend that reports no usage\nsimply contributes zeros.\n\nIf you find this work useful, please cite:\n\n```\n@misc{kwok2026llmasaverifiergeneralpurposeverificationframework,\n      title={LLM-as-a-Verifier: A General-Purpose Verification Framework}, \n      author={Jacky Kwok and Shulu Li and Pranav Atreya and Yuejiang Liu and Yixing Jiang and Chelsea Finn and Marco Pavone and Ion Stoica and Azalia Mirhoseini},\n      year={2026},\n      eprint={2607.05391},\n      archivePrefix={arXiv},\n      primaryClass={cs.AI},\n      url={https://arxiv.org/abs/2607.05391}, \n}\n```\n\n", "url": "https://wpnews.pro/news/self-verification-with-deepseek-v4-flash-beats-claude-fable-5-on-terminal-bench", "canonical_source": "https://github.com/llm-as-a-verifier/llm-as-a-verifier", "published_at": "2026-08-18 16:30:33+00:00", "updated_at": "2026-08-18 16:41:42.512472+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "ai-tools", "ai-agents"], "entities": ["LLM-as-a-Verifier", "DeepSeek V4 Flash", "Claude Fable 5", "Terminal-Bench 2.1", "SWE-Bench Verified", "MedAgentBench", "RoboRewardBench", "Gemini 2.5 Flash"], "alternates": {"html": "https://wpnews.pro/news/self-verification-with-deepseek-v4-flash-beats-claude-fable-5-on-terminal-bench", "markdown": "https://wpnews.pro/news/self-verification-with-deepseek-v4-flash-beats-claude-fable-5-on-terminal-bench.md", "text": "https://wpnews.pro/news/self-verification-with-deepseek-v4-flash-beats-claude-fable-5-on-terminal-bench.txt", "jsonld": "https://wpnews.pro/news/self-verification-with-deepseek-v4-flash-beats-claude-fable-5-on-terminal-bench.jsonld"}}