{"slug": "feeling-vs-data-my-multi-agent-workflow-measured", "title": "Feeling vs Data: My Multi-Agent Workflow measured", "summary": "A developer analyzed roughly two months of agent logs covering about 1,100 agents and 45 features to test whether a multi-agent coding workflow delivered real productivity gains. The data confirmed that orchestrator turn latency degraded sharply with context size before the switch — median turns slowed about 8x between small context and 600k tokens, with p90 at 188 seconds — and stayed flat at roughly 5 seconds per bucket after moving orchestration to Paseo with Claude Opus and 400k-token autocompaction. The developer credits the combined change of model, TUI and orchestration tool, noting the data cannot isolate which caused the effect.", "body_md": "*Two months of agent logs, about 1,100 agents and 45 features. What the data confirmed, what it corrected, and where the saved capacity went.*\n\nA few weeks ago I wrote on [LinkedIn](https://www.linkedin.com/pulse/eu-achava-que-produtividade-10x-com-ia-era-mito-bertacchi-dobmf/) (in Portuguese) that I used to think \"10x productivity with AI\" was a myth. Here is the short version for readers who missed it.\n\nI have used AI for coding for years. I went from autocomplete to chat, then IDE plugins (Continue, Cline, Roo Code, Kilo Code), then TUIs (Claude Code, Codex, OpenCode). For a long time, I felt 2x was the ceiling. Agents ignored my conventions, duplicated code and drifted from project patterns, even with commit hooks, an `AGENTS.md` and skills.\n\nSpec-driven development (SDD) changed that. Specs are bureaucratic, they cost time, and they drift from the code. But clear, verifiable goals let me give agents autonomy. I stopped babysitting them.\n\nAutonomy brought new problems: bigger and messier PRs, plan limits, and the wrong model or reasoning level for a task. So I built a process:\n\nOne orchestrator agent runs this process and dispatches other agents by task difficulty. My PRs got better. My reviews found fewer problems. But the orchestrator got slow as its context grew. A cache miss after lunch could burn my 5-hour plan limit in one message.\n\nSo I moved orchestration to [Paseo](https://paseo.sh/). Paseo lets me run each lab's native CLI and define one agent profile per kind of work. I went back to Claude Opus as the orchestrator, and I set Claude Code's autocompaction at 400k tokens. I work on three plans: $20 Claude, $20 ChatGPT and $10 OpenCode Go.\n\nI built [JOBE Vox](https://vox.jobe.works/), a transcription service, with this workflow in one month. It has Stripe billing, three environments, CI/CD, observability, backups and models from three AI labs.\n\nIn the LinkedIn post I described what I *felt*. This post checks those feelings against the data.\n\nThese are the agent profiles I use today. The orchestrator picks one for each task.\n\nThe workflow is driven by OpenSpec changes, one per feature. A change goes through these steps:\n\n`fixup!` commits until I approve, then the branch is squashed to a clean history.`main`.\nThe orchestrator also keeps a memory of rules. Most rules started as a correction from me: \"never merge without a reviewer\", \"dispatch a fresh agent after one fix round\", \"reject findings that cannot happen in this system\". The process is what is left after each mistake.\n\nSteps 4 and 5 did not exist before Paseo. The orchestrator never touched GitHub then: no PRs, no CI runs, no comment reviews. It felt impossible with the context I had. Keep that in mind, because it explains most of what follows.\n\nI used five sources:\n\nI used **OpenSpec changes as the proxy for features**. There were 16 before Paseo and 29 after.\n\n**Method notes**\n\n- The cutover is the first Paseo agent, on 2026-10-09.\n**Context tokens** = input + cache read + cache write, summed over requests. They are not unique tokens and not billed cost.- Each orchestrator session is mapped to the feature it mentions most.\n- The\n**model, the TUI and the orchestration tool all changed at the same time**. The data cannot say which one caused an effect. I credit the switch as a whole.\n\nThis was my strongest feeling: orchestrator sessions got slow once they crossed a context threshold. The recorded turn latency shows the same pattern.\n\nBefore Paseo, the median turn got about 8x slower between a small context and 600k. At 400–500k, the slowest 10% of turns took at least about 3 minutes (p90 188 s). The rise remains when I control for output length: seconds per output token went from 0.047 to 0.35. After Paseo, the median stays at about 5 seconds (4.5–6.1 s) in every bucket.\n\nOne caveat: the two tools log time differently. For OpenCode I measure from message creation to completion, minus tool execution. For Claude Code I measure from the previous tool result to the last block of the response. Both approximate the time I waited for the model, but they are not identical measurements.\n\nThe data also sharpened one claim. I wrote that the orchestrator \"frequently\" went above 600k. Only one feature before Paseo did that. The real threshold was 400k:\n\nSo my 600k figure was too high. The 400k threshold matches both the slowdown curve above and my autocompaction setting.\n\nFor the closest comparison, I split each post-Paseo feature at the moment its PR opened. I treat everything before that point as planning and implementation, the only work the old orchestrator ever did. PR creation is a proxy for the phase boundary, and features still differ in size.\n\nThe orchestrator now processes about half the tokens for the same phase. It also needs about a third of my messages. Five features have no recorded orchestrator work before their PR opened. Either the PR opened early, or the session mapping missed that work. Excluding them, the medians are 14.9M tokens and 12 messages, so the result holds.\n\nThis is the claim I made on LinkedIn, and it holds: the orchestrator's load dropped a lot.\n\nTotal orchestrator tokens per feature did *not* go down. They went from 26.4M to 47.3M. The difference is the new phase:\n\nThis changes how I read my \"1 in 8 features compacts\" estimate. In planning and development, the phase where it matters most, 5 of 28 features (18%, about 1 in 6) compacted before the PR opened. That is close to my estimate, and down from 4 of 16 (25%, 1 in 4) before Paseo. This phase is the most sensitive to compaction, because losing context there can change design and code decisions. Most compactions now happen later, in the PR and release phase. Over the whole lifecycle, 13 of 29 features (45%) had at least one compaction.\n\nThe data shows a lower load in the old phases and a large new phase after the PR. It cannot show why. My own account: once the orchestrator had room in its context, I used that room to add the review and release steps I could not afford before.\n\nWhere does the extra capacity go? Mostly to review.\n\nAfter Paseo, a typical feature used 7 developer agents, 8 code reviewers, 1 plan reviewer and 2 validators (medians). The roles come from agent titles, so the post-Paseo split is a range, not an exact count.\n\nReviews run on a model mix. DeepSeek flash and GLM flash, both on my $10 OpenCode Go plan, ran about 207 of the code reviews. GPT-Sol ran about 169, mostly for complex work.\n\nIs the quality visible? One independent signal is the Codex bot that reviews every PR push. Before October 25 (UTC), it raised 0.64 findings per review pass (63 in 99). From October 25 onward, it raised 0.18 (12 in 67). Fewer bot findings does not prove fewer defects, and these data do not show why the rate fell. The samples are small, and the bot had outages. It does match my impression that problems now get caught before the bot looks.\n\nEvery feature in this post shipped in [JOBE Vox](https://vox.jobe.works/). It is a transcription service I built with this workflow. You upload audio and get a clean, readable transcript, in the browser or through a public API. Every change in it went through the plan reviews, code reviews and release gates measured above.\n\nIf you need transcription, give it a try and tell me what you think. Your feedback goes into the same workflow.\n\n*Thanks for reading. If you run a multi-agent workflow and measure it differently, I'd like to hear how.*", "url": "https://wpnews.pro/news/feeling-vs-data-my-multi-agent-workflow-measured", "canonical_source": "https://dev.to/jebertacchi/feeling-vs-data-my-multi-agent-workflow-measured-3lmp", "published_at": "2026-11-20 21:37:39+00:00", "updated_at": "2026-11-20 21:45:04.637628+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "developer-tools", "mlops"], "entities": ["Paseo", "Claude Code", "Claude Opus", "OpenCode", "OpenSpec", "JOBE Vox", "Anthropic", "OpenAI"], "also_reported_by": [], "takeaway": "Orchestrator turn latency rose about 8x from small context to 600k tokens (p90 188s) before the switch, and stayed near 5 seconds per context bucket after moving orchestration to Paseo with 400k-token autocompaction.", "recommendation": {"advice": "Cap orchestrator context with autocompaction (e.g. 400k tokens) and offload orchestration to a tool that runs each lab's native CLI, since latency scaled sharply with context growth.", "playbook": "agent-architecture", "playbook_url": "https://wpnews.pro/playbooks/agent-architecture"}, "alternates": {"html": "https://wpnews.pro/news/feeling-vs-data-my-multi-agent-workflow-measured", "markdown": "https://wpnews.pro/news/feeling-vs-data-my-multi-agent-workflow-measured.md", "text": "https://wpnews.pro/news/feeling-vs-data-my-multi-agent-workflow-measured.txt", "jsonld": "https://wpnews.pro/news/feeling-vs-data-my-multi-agent-workflow-measured.jsonld"}}