{"slug": "memdoor-a-team-of-5-qwen3-8-27b-models-beats-fable-5-on-livecodebench-hard", "title": "Memdoor: A team of 5 Qwen3.8-27B models beats Fable 5 on LiveCodeBench Hard", "summary": "A five-role Memdoor multi-agent workflow running Qwen3.8-27B scored a mean of 9.8 out of 10 across five passes on the first ten LiveCodeBench Hard problems, edging the 9.6 mean of the original GVS5H harness (Lee, 2026) and beating Fable 5 alone at 8.0, according to results published in the Memdoor repository. The workflow's planner, two ideators, solver, manager and repair worker share only files on disk, and a shell script rather than a model grades success; 49 of 50 runs solved their problem at a total cost of $35.79 (15.3M output tokens), or $38.77 including 16 aborted attempts. The author cautions that only ten problems were tested rather than the authors' 100, that the 2025 contest problems may have appeared in training so contamination cannot be ruled out, and that Fable 5's column is a single call per problem from the authors' files rather than a run in this harness.", "body_md": "A multi-agent harness for programming problems, written as a\n[Memdoor](https://github.com/guregodevo/memdoor/blob/main/README.md) workflow. Five roles (planner, two ideators,\nsolver, manager) and a repair worker each start from a fresh context and talk\nonly through files on disk. A shell script that runs the samples decides\nsuccess, not a model. It is the method of\n[GVS5H](https://github.com/slee-persis/GVS5H) (Lee, 2026: five Qwen3.8-27B\nmodels match Fable 5 on LiveCodeBench Hard).\n\nRebuilt with the authors' exact prompts on Qwen3.8-27B, their loop scored a\nmean of **9.8 of 10** over five passes on the first ten LiveCodeBench Hard\nproblems, against 9.6 for their own harness and 8.0 for Fable 5 alone\n([below](#five-passes-on-the-authors-ten-2026-10-31), every run published).\n\nThe coder wrote these files itself from a description of the method (2026-10-25). Nobody hand-wrote them.\n\nGVS5H's own harness, rebuilt as a Memdoor workflow on the same model: Qwen3.8-27B, a 128K reply cap, the authors' prompts and loop. The first ten problems of their LiveCodeBench Hard list, five passes, every program graded on every public and hidden test with their corrected grader.\n\n| pass | Memdoor workflow | authors' 27B harness | Fable 5 alone | \n|---|---|---|---|\n| 1 | **10** / 10 | 9 | 7 | \n| 2 | **10** / 10 | 10 | 8 | \n| 3 | **10** / 10 | 10 | 10 | \n| 4 | **10** / 10 | 9 | 8 | \n| 5 | **9** / 10 | 10 | 7 | \n| mean | **9.8** | 9.6 | 8.0 | \n\nThe authors' columns are their repository's regraded result files\n(`q38_multiagent_pN`, `fable5_single_pN`), the numbers in their paper.\n[`bench/results/parity-5pass/`](https://github.com/guregodevo/memdoor/blob/main/examples/workflows/team-solve/bench/results/parity-5pass) holds all 50\nruns: every request and reply, the program, the grader's full output, each\ncall's cost and the host that served it, and every attempt that did not count.\n49 of 50 solved; the 50 runs cost $35.79 (15.3M output tokens), $38.77 with\nthe 16 attempts that were cut by a clock, stopped for a fix or paused.\n\n<sub>Note: ten problems, five passes each, not the authors' 100. The contests\ndate from 29 March to 6 April 2025, and the model may have seen them in\ntraining: contamination cannot be ruled out. Fable 5's column is a single call per\nproblem from the authors' files; it was not run in this harness. The loop\nsees only the public samples; the score is the hidden tests. A run took a\nmedian of 40 minutes of model time (the longest 4 hours) and $0.39. All on\nOpenRouter, not a local machine; Modal does not publish its precision.\nLocal paths in the run scripts and in four tracebacks a manager was shown\nread `<scratch>` and `$HOME`; nothing else in the runs is edited.</sub>\n\n**Same as the authors'.** Every message, byte for byte:\n[`lib/test_parity.py`](https://github.com/guregodevo/memdoor/blob/main/examples/workflows/team-solve/bench/results/parity-5pass/workflow/lib/test_parity.py)\nruns their `multiagent_solve` and our workflow against one scripted model and\ncompares each request (four cases, all identical; a one-character change makes\nthem differ). The temperatures (plan 0.3, ideation 0.4, manager and worker\n0.2), top_p 0.95 and top_k 20 (what their vLLM applied from the model's own\nconfig), the 128K cap, up to 10 rounds, a cut-off worker summarized, and a\nreply with no text answered by its reasoning.\n\n**Different.** The host. Theirs was their own vLLM in FP8. Ours is\nOpenRouter, asked in the order Modal, CoreWeave, Parasail, Wafer until Modal\nslowed to 14 tokens/s, then the FP8 hosts only (Wafer, Parasail, CoreWeave):\n203 calls were served by Modal, 27 by CoreWeave and 115 by Wafer, every one\nof the FP8 order's calls by Wafer. Three runs were moved mid-run, their\nfinished steps kept (`log/host-switch.txt`).\n\n**What it took.** The problem text was the mismatch: our first version wrote\nits own task file around each problem, and the 27B planned worse from it than\nfrom the authors' exact LiveCodeBench text (arc196_c's plan: 0 of 2 against 2\nof 2). Four Memdoor changes let a workflow ask what their harness asks: a task\nanswers with the model's reasoning when it wrote no text, `top_p`/` top_k` on a\nmodel line, `mark_cutoff` for a reply cut at the cap, and a call that runs on\nits own four-hour clock.\n\n**Where it lost.** Twice, the same two gaps in the authors' loop:\n\n- Pass 5, abc399_e (32 of 44 hidden): round 1's program was right; the manager invented a counterexample with a wrong expected answer, the team rewrote it, and the new program passed the samples, which do not cover the case.\n- Pass 4, arc196_c and arc196_d: a program passing every hidden test was overwritten by a later round because the manager did not say done; both got a correct program back before the loop's end (arc196_d at round 8, arc196_c through the final call), at $3.44 and $4.24, the two most expensive runs.\n\nBoth are fixable without changing the method (check a manager's counterexample against a brute force on small inputs; keep the last program that passed the samples). Neither is in these numbers.\n\n```\ncurl -fsSL https://memdoor.ai/install.sh | bash\nmemdoor connect openrouter                 # or anthropic, openai, gemini, chatgpt\ngit clone --depth 1 https://github.com/guregodevo/memdoor\n\nmkdir -p ~/.memdoor/workflows\ncp -r memdoor/examples/workflows/team-solve ~/.memdoor/workflows/   # the library: any project sees it\nmkdir my-problem && cd my-problem\ncp -r ../memdoor/examples/workflows/team-solve/sample/* .          # task.md + tests/1.in … 5.out\nmemdoor workflow run team-solve --wait   # progress on stderr, exit 0 when every sample passes\n```\n\n`/workflow` in `memdoor tui` draws the same run live. For your own problem,\nreplace `task.md` and put paired `tests/N.in` / `tests/N.out` files beside it.\nThe program is `solution.py`, read from stdin, compared to the samples with\n`diff`. It needs Python 3 and bash. To keep it with one repo instead, copy\nthe folder into that project's `.memdoor/workflows/`. The project's copy wins.\n\nThe steps name no model. Whatever `/model` is pinned to answers, a local one\nincluded (`memdoor connect local` for janis, Ollama or llama-server).\n\n```\nplan ──┬── ideate-a ──┬── solve ── test ── review ── repair ── verify\n       └── ideate-b ──┘\n```\n\n| step | kind | reads | writes | \n|---|---|---|---|\n| `plan` | agent | `task.md` | `plan.md` , and a`notes.md` ledger with one anchor per ideator | \n| `ideate-a` ,`ideate-b` | agent, in parallel | `task.md` ,`plan.md` ,`notes.md` | a proposal at its own anchor: A the most direct provable algorithm, B one from a different family of technique | \n| `solve` | agent | both proposals | the decision in `notes.md` ,`solution.py` | \n| `test` | command | `tests/` | `test-output.log` ,`test-status.txt` (never fails the run) | \n| `review` | agent | the captured test output, not a model's claim | `review.md` , repair instructions in`notes.md` | \n| `repair` | agent | the manager's instructions | a fixed `solution.py` , or nothing if the samples passed | \n| `verify` | command | `tests/` | `final-test-output.log` ; nonzero exit fails the run | \n\nEvery agent prompt says \"fresh context, read the files, do not rely on chat\". The paper credits that ledger discipline for the gain. There is one repair round and no human gate. The last step is the whole verdict:\n\n```\ntype: command\nrequires:\n  - table_pattern: repair\ncommand: |\n  check=.memdoor/workflows/team-solve/check-samples.sh\n  [ -f \"$check\" ] || check=\"$HOME/.memdoor/workflows/team-solve/check-samples.sh\"\n  bash \"$check\" > final-test-output.log 2>&1\n  result=$?\n  cat final-test-output.log\n  exit \"$result\"\n```\n\nEach step is one YAML file in `steps/`, the format of\n[mario](https://github.com/guregodevo/mario), the engine underneath.\n`requires:` is the DAG. Changing a role means changing its prompt.\n\nThe sample: count the substrings of a string over {a, b, c} with equal counts of each letter, up to 200,000 characters. Run on the default ladder through OpenRouter:\n\n- 8 of 8 steps, 5 of 5 samples. The solver took ideator A's prefix-state method, O(N) with a dictionary of (a−b, a−c) pairs. A brute force on 300 random strings found no mismatch afterwards, and it runs in 0.04 s at N = 200,000. The manager found nothing to repair.\n- One ideator's provider dropped the stream mid-answer. The coder had written\n`max_retries: 0` , so the run died there. The agent steps now carry`max_retries: 1` , and the run was resumed with`--partition` , which kept the\nplanner's and the other ideator's work.\n- The ideator prompts the coder wrote named the sample's methods (A: a streaming prefix-state count, B: sorting prefix states), which would steer any other problem the wrong way. They now ask A for the most direct provable algorithm and B for one from a different family of technique (2026-10-28).\n- The first attempts failed every step in 0 s with \"no agent named coder\": a run looked its agent up only in its own workspace. That was fixed in the gateway the same day (v1.692.0).", "url": "https://wpnews.pro/news/memdoor-a-team-of-5-qwen3-8-27b-models-beats-fable-5-on-livecodebench-hard", "canonical_source": "https://github.com/guregodevo/memdoor/tree/main/examples/workflows/team-solve", "published_at": "2026-10-31 22:46:12+00:00", "updated_at": "2026-10-31 22:51:05.560460+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-research", "ai-tools"], "entities": ["Memdoor", "Qwen3.8-27B", "Fable 5", "LiveCodeBench Hard", "GVS5H", "OpenRouter", "Modal", "Wafer"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/memdoor-a-team-of-5-qwen3-8-27b-models-beats-fable-5-on-livecodebench-hard", "markdown": "https://wpnews.pro/news/memdoor-a-team-of-5-qwen3-8-27b-models-beats-fable-5-on-livecodebench-hard.md", "text": "https://wpnews.pro/news/memdoor-a-team-of-5-qwen3-8-27b-models-beats-fable-5-on-livecodebench-hard.txt", "jsonld": "https://wpnews.pro/news/memdoor-a-team-of-5-qwen3-8-27b-models-beats-fable-5-on-livecodebench-hard.jsonld"}}