Multi-model agent team vs a single frontier model: what the Memdoor claim measures, and the cost per task A manager-and-worker loop of Qwen3.8-27B calls scored 92.4% ± 2.3 on the 100 newest hard LiveCodeBench release_v6 problems versus 90.4% ± 1.4 for a single Claude Fable 5 call, at $0.48 versus $0.61 per problem, according to version 2 of the GVS5H paper by Victor Gao, Vida Khosrowshahi, Ali Khosrowshahi, Xihao Sun, Juhyun Lee and Simon Lee of Persis Capital, dated 21 September 2026. The 2-point lead is not statistically significant, and the Qwen3.8-27B team emitted 18.6 million output tokens against 1.2 million for Fable 5. Memdoor's 31 October replication ran only 10 problems in 5 passes and its own README calls the result parity, with the Fable 5 column taken from the paper's result files rather than measured by Memdoor. Multi-model agent team vs a single frontier model: what the Memdoor claim measures, and the cost per task On the 100 newest hard problems in LiveCodeBench, a manager-and-worker loop of Qwen3.8-27B https://stackness.dev/tools/qwen calls scored 92.4% against 90.4% for one call to Claude Fable 5 https://stackness.dev/tools/claude-fable , at $0.48 a problem against $0.61. The 2-point lead is not significant, and the team wrote about 15 times as many output tokens. Those are the numbers from version 2 of the GVS5H paper https://arxiv.org/abs/2608.26480 , dated 21 September 2026. Memdoor's "beats Fable 5" post of 31 October replicated that loop on 10 problems, and its own README calls the result parity. A team of small models pays off on problems a test can check, run on your own hardware; one frontier call wins on latency, on repo-level work and wherever there are no tests. Multi-model agent team vs single frontier model, side by side The deciding facts between a team of small open models and a single frontier call are score, cost per task, token volume, latency and where the model runs. Scores, costs and token counts below come from the GVS5H paper's five-pass runs on LiveCodeBench release v6; calls, time and the first-ten scores from Memdoor's run files; prices from OpenRouter and Anthropic on 1 November. The Fable 5 column was measured by the paper's authors, not by Memdoor. | | Qwen3.8-27B team GVS5H loop | Fable 5, one call | |---|---|---| | Score, 100 hard problems, 5 passes | 92.4% ± 2.3 | 90.4% ± 1.4 | | Score, first 10 problems | 9.8 Memdoor , 9.6 paper's harness | 8.0 | | Cost per problem, list price | $0.48 | $0.61 | | Output tokens per 100 problems | 18.6 million | 1.2 million | | Calls per problem | 5 to 22 in Memdoor's runs | 1 | | Time per problem | Median 40 min in Memdoor's runs, longest 4 h | Not published | | Runs on | Your GPU or any of 22 OpenRouter hosts | Anthropic's API | | Licence | Apache-2.0 weights | Proprietary | | Price per million tokens, in / out | $0.21 / $2.55 in the paper; $0.15 / $1.875 on OpenRouter https://stackness.dev/tools/openrouter today | $10 / $50 | Price and performance chart from "Zero-Shot Self-Orchestration with Ledger-Based Control Improves Coding in Language Models" GVS5H https://github.com/slee-persis/GVS5H by Victor Gao, Vida Khosrowshahi, Ali Khosrowshahi, Xihao Sun, Juhyun Lee and Simon Lee, Persis Capital, 2026, CC BY 4.0 https://creativecommons.org/licenses/by/4.0/ . The team of five is one Qwen3.8-27B model playing six roles The "team" in both GVS5H and Memdoor is a single model, Qwen3.8-27B, called again and again in a fresh context, each time in a different role. A manager writes a plan and seed tasks, a worker brainstorms approaches into notes.md , the manager picks one next task, a fresh worker rewrites solution.py , and a script runs the public sample tests. A finalizer writes the answer when the loop ends without a clean pass. There is no voting and no second model. The paper says "every role but the verifier is the same underlying model". The loop stops when the tests pass and the manager says done, or after 10 manager-worker rounds. The calls run one after another, not as five models side by side. Memdoor's team-solve README https://github.com/guregodevo/memdoor/tree/main/examples/workflows/team-solve shows a different workflow from the one that scored 9.8. The eight files in steps/ allow one repair round, a shape Memdoor's earlier 100-problem bench scored at 47 of 100 with thinking off, against 35 for a single call. The 9.8 came from a generated workflow, bench/results/parity-5pass/workflow/gen.py , that runs the paper's 10-round loop. Memdoor ran 10 problems and copied Fable 5's column Memdoor measured 10 problems in 5 passes, 50 graded runs on LiveCodeBench release v6, contests from spring 2025. It scored 9.8 out of 10 against 9.6 for the paper's own harness on the same problems. The Fable 5 score of 8.0 came from the paper's result files, one call per problem at effort: high per the authors' run config; the README says it "was not run in this harness". The README is plainer than the Hacker News title https://news.ycombinator.com/item?id=50320995 : "On a sample this small, 9.8 and 9.6 are the same number. The claim is parity with their harness, not a gain on it." The 50 runs cost $35.79 on OpenRouter, and the one lost run was a correct program that the manager talked the team into rewriting. The full-set result depends on fixes the paper made to the grading. LiveCodeBench's stock grader mis-scored programs reading input with sys.stdin.buffer.readline , which Qwen uses and Fable 5 does not, and the authors also corrected some answer keys. With the stock grader and the original keys, the team scores 84.4 and Fable 5 87.4. Contamination cannot be ruled out for either side: the problems date from December 2024 to April 2025, Fable 5 came out in June 2026, and Qwen3.8-27B in August 2026. The Qwen3.8-27B team writes 15 times the tokens for a 22% smaller bill The Qwen3.8-27B loop is cheaper per problem than Fable 5 only because Qwen's output tokens cost about a twentieth as much at the paper's prices. The paper prices one pass over 100 problems at $47.82 for the team and $61.11 for Fable 5; at OpenRouter's price today the team's tokens would cost about $35. The team wrote 18.6 million output tokens to Fable 5's 1.2 million, and used 1.5 million input tokens to Fable 5's 90,000. That ratio decides what the team costs you outside a benchmark. On a subscription with rate limits, on a host that caps output at 32,768 tokens like Cerebras, or on one local GPU, the team's volume is the bill. Memdoor's docs put the floor at "a 27B at Q5 or above on a 24 GB card", and a hard problem at about 300,000 output tokens a run; at 50 tokens a second, my arithmetic gives about 100 minutes. The price per task also moves with the host: the paper priced Qwen at $0.21 / $2.55 per million, while Memdoor paid $2.30 to $3.00 per million output tokens on Modal, Wafer and CoreWeave. The same loop on a stronger small model is the better buy. Qwen3.8-Flash-Next with the manager scored 93.0% for $5.76 per 100 problems, a tenth of Fable 5's cost. Against Claude Opus 5.5 https://stackness.dev/tools/claude-opus , $4 / $20 per million tokens and the model Anthropic says to start with "for most workloads", nobody has run the comparison. The loop beats best-of-five sampling on problems a test can check The Qwen loop's gain is not only extra sampling. I computed pass@k from the paper's stored run files, with an oracle that picks a passing answer whenever one exists: two or three independent Qwen single calls, at about the loop's budget, reach 81.6% to 87.1%. Five calls with a perfect picker reach 90.0% at $0.95 a problem, twice the loop's cost and still below its 92.4%. The paper finds the gain largest where the base model cannot organise its own reasoning: +25.6 points for Qwen3.8-27B, whose single calls ran past the 128,000-token cap in 150 of 500 attempts, against +3.6 for DeepSeek-V4.1-Flash. Some models gained nothing. The only outside signal is the public sample tests, so the 27 LeetCode problems, whose call-based tests the verifier does not run, got no feedback at all. Outside contest problems the evidence is mixed. Cohere's study https://cohere.com/blog/lessons-from-scaling-teams-of-frontier-coding-agents of 14 October found teams gained up to about 10 points on ProgramBench at three to seven times the cost, and that cheap workers need a strong lead: "A Terra lead with Luna workers still fell short of a Sol agent working alone 51.6% against 57.7% ". On the Artificial Analysis Intelligence Index v5, a composite with 60% private test sets, Qwen3.8-27B scores 20 https://artificialanalysis.ai/models/qwen3-8-27b against 58 for Opus 5.5, and no team result exists there. The YC Labs game swarms of 29 October ran frontier models only, so they say nothing about small against large. How do you verify a self-reported agent team benchmark? Verifying an agent team benchmark takes four checks, and the Memdoor case fails the first two as published. Ask who measured the frontier baseline and at what effort; compare against a compute-matched baseline, not one call; check that the problems postdate both models; and confirm that the workflow you would copy is the one that was scored. The GVS5H repository stores every run, so most of this takes minutes. 1. Regrade a stored answer. Memdoor's bench/results/parity-5pass/ holds each run's prompts, replies, per-call costs and solution.py ; bench/grade.py --private