Five lead-worker AI model pairings were scored on six coding tasks. Here's which orchestration setup delivered the most usable code.
What is AI coding orchestration, and which team won? #
A recent hands-on benchmark tested five “lead and worker” AI model pairings across six coding projects, from an emulator to a Fitbit desktop app, scoring each out of 10 points per task. Opus paired as the lead with Sol as the worker came out on top, earning 44.2 out of 60 possible points (about 73.7%), narrowly ahead of Opus leading Sonnet at 43.7 (72.8%). The two Astra-led pairings scored far lower, partly because of incomplete planning and partly because of provider quota interruptions that cut runs short.
TL;DR #
- Opus leading Sol produced the highest overall score across six coding tasks, staying useful in every category tested, from emulators to health apps.
- Opus leading Sonnet came in a close second and actually produced the single strongest result of the whole test, a 7.1 out of 10 on a Rust-based Game Boy/DS/3DS emulator.
- Astra as a lead model repeatedly under-planned, assigning only partial work (like file inspection instead of execution) on tasks such as the emulator and keyboard configurator, tanking its scores even when its worker executed well.
- Quota and rate-limit failures hit several runs hard. One Astra-led Pokémon test returned a completely empty project folder after repeated HTTP 429 errors, and Opus with Muse full hit zero on the same task for similar reasons.
- No team built a true Docker alternative. All five pairings capped out around 5 out of 10 because none established real process isolation or resource controls, only partial image-handling tools.
- Token and cost accounting was incomplete. Only 8 of 30 task runs had fully captured usage data, so the roughly $405 in recorded spend across all teams can’t be used to declare a “cheapest” or best value-per-dollar winner.
- The benchmark explicitly isn’t a controlled experiment : each pairing got one run per task on a shared machine without matched compute budgets, solo baselines, or reversed lead/worker roles.
Seven tools to build an app. Or just Remy. #
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
How was the benchmark structured? #
The test used a lead model and a worker model for each team. The lead plans the work, assigns tasks, and reviews results; the worker (or workers, up to three running simultaneously) executes the assigned coding. This lead/worker split is what the source video calls orchestration, and the claim is that the lead’s planning decisions shape the entire outcome of the project, sometimes more than the raw coding skill of the worker model.
Five pairings were tested: Opus leading Sol, Astra leading Sonnet, Opus leading Sonnet, Astra leading Sol, and Opus leading Muse full. Everything ran through Claude Code, Codex, and a native Muse CLI, with each team getting up to 20 lead planning rounds and 50 total task assignments, with no hard time limit. Each pairing got exactly one high-effort run per task, on a shared machine, without equalized compute budgets across teams. There were no solo-model baselines and no tests with the lead/worker roles reversed, so the results show how these specific pairings performed, not how much the orchestration pattern itself added over a single model working alone.
What were the six coding tasks? #
The tasks were chosen to span different domains of difficulty:
- An emulator supporting Game Boy, DS, and 3DS systems in Rust, including save states and standard playback.
- A Docker alternative requiring container image handling plus process isolation and resource controls.
- A room designer (a 3D furniture/interior layout tool).
- A keyboard configurator for QMK keyboards, styled as a yellow-themed alternative to VIA.
- A Pokémon-style game opening playable through the first gym, with save/load support.
- A Fitbit-style desktop analytics app with a Rust back end.
Each task was worth 10 points, scored not just on whether code compiled but on whether the delivered feature actually worked under testing, including reproducing bugs, checking save/load integrity, and verifying real user flows rather than trusting the lead model’s own status reports.
Where did orchestration help the most? #
The clearest case of orchestration working as intended showed up in the room designer task. In the Opus-with-Sol run, a tester found that selecting a chair made the viewport go blank. The lead model initially assumed it was a stale screenshot artifact, but the tester reproduced the failure, a worker fixed the selection highlight bug, and the lead verified the rebuilt app. That loop, catching a real bug, assigning a fix, and confirming the fix, is what good orchestration is supposed to look like.
Room designer scores were strong across the board: Astra with Sonnet actually topped this category at 9.3, with the lead splitting ownership of the room model, editor, and renderer among separate workers and sending targeted corrections as the pieces connected. Opus with Sol and Opus with Sonnet both landed around 8.9-9.0, and even Opus with Muse full matched that with custom shapes and working project restoration, despite some leftover texture cleanup issues.
The Pokémon task showed a similar pattern: Astra with Sonnet caught a walkthrough bug where players were told to heal at a location with no actual healing center, and assigned a worker to build the missing door, NPC, and route. That attentiveness earned it the top Pokémon score at 9.1.
Where did orchestration fail or stall? #
Astra as a lead model was the weak point repeatedly. On the emulator task, both Astra-led teams planned full execution for Game Boy support but only wrote file inspection code for DS and 3DS, meaning those two systems never actually ran games. Astra with Sonnet scored 1.9 and Astra with Sol scored 2.2 on the emulator, a direct result of the lead failing to assign the right scope of work up front. The video’s framing is blunt about this: improving Game Boy support doesn’t make a DS file inspector run DS games.
The keyboard configurator task exposed a similar planning gap. The job required actually sending configuration changes to a connected QMK keyboard, not just building a pretty editor. Both Astra-led teams scored just 2.8 because their assignments explicitly skipped device communication, with Astra-with-Sonnet’s plan even excluding opening the device and sending configuration messages. The benchmark didn’t ban building that communication path, it only banned modifying real hardware during testing, so the low scores reflect a planning choice, not a technical constraint.
Provider quota and rate-limit failures caused outright task collapses in a few cases. Opus with Muse full scored zero on the Pokémon task because subscription quota errors hit all six workers used in that run, eventually blocking even a one-line read request. Astra with Sol also scored zero on Pokémon after HTTP 429 errors exhausted retries, leaving a completely empty project folder. These zeros reflect what got delivered under those specific failures, not a ceiling on what the underlying models can do with stable access.
Is any team the “best” AI coding pairing? #
Based on earned points, Opus leading Sol is the strongest all-around performer, staying functional across every one of the six tasks and finishing with the highest total score. Opus leading Sonnet is a close second and produced the single best result in the whole test, a 7.1 on the emulator task, suggesting it’s worth considering specifically for systems-level or performance-sensitive coding work. Astra leading Sonnet is worth a look for visual, UI-heavy work like the room designer and the Pokémon game, but its planning gaps on the emulator and keyboard tasks mean its initial task assignments need careful review rather than blind trust.
Two pairings, Opus with Muse full and Astra with Sol, had results too interrupted by quota and rate-limit failures to judge fairly. Both need clean, complete runs before any real conclusion can be drawn about their capability ceiling.
It’s also worth being clear about what this benchmark doesn’t establish. None of the five teams built a genuine Docker alternative; all were capped around 5 out of 10 because none demonstrated real process isolation. None of the emulators proved compatibility with most commercial games, only small homebrew tests. And the cost data is incomplete: only 8 of 30 task runs had fully captured token usage, so the roughly $405 total in recorded orchestration spend can’t be used to crown a cheapest or best-value setup.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
Frequently Asked Questions #
What does “lead and worker” mean in AI coding orchestration?
The lead model plans the project, breaks it into assignments, and reviews the results. The worker model (or models) executes those assignments. The lead’s planning quality, what it chooses to assign and what it leaves out, strongly shapes the final outcome, sometimes more than the worker’s raw coding ability.
Which AI model pairing scored highest in this benchmark?
Opus leading Sol scored highest overall, 44.2 out of 60 points (about 73.7%) across six coding tasks. Opus leading Sonnet was close behind at 43.7 points (72.8%).
Why did the Astra-led teams score so low on some tasks?
In several cases, Astra as the lead model wrote incomplete task plans, for example assigning only file inspection instead of full execution support for DS and 3DS emulation, or explicitly excluding device communication from the keyboard configurator task. The workers executed what they were assigned correctly, but the assignments themselves left out required functionality.
Did any team build a working Docker alternative?
No. All five pairings scored around 5 out of 10 or lower on the Docker alternative task. None demonstrated real process isolation or resource controls, only partial container image handling, so none qualifies as a functional Docker replacement.
Can this benchmark tell you the cheapest AI coding setup?
Not reliably. Only 8 of the 30 total task runs had complete token usage data captured. The reported total of roughly $405 across all teams is based on partial records, so it can’t be used to identify a true cheapest or best value-per-dollar pairing.