In the last post, on 2026-08-18, I published what 14 MCP servers cost a context window before an agent does any work, and said the next thing was Tier 2: real clients, real tasks, every frame logged. That has now run. Ninety trials, three servers, two clients, fifteen scripted tasks, three trials each, through a proxy on the MCP stdio pipe.
The finding worth the post is not in the token table. It came out of the shakedown hours earlier that same day, on the client version the run then replaced: one of the clients was failing calls inside itself, before a byte reached the server, and on the wire that looked exactly like a model that tried very little and answered wrong.
The matrix is filesystem
, playwright
and github
, five scripted tasks each, three trials per task per client, suite version 1.0.1. Ninety trials, 87 successes. Servers pinned at @modelcontextprotocol/server-filesystem@2026.7.10
, @playwright/mcp@0.0.79
, and the ghcr.io/github/github-mcp-server
container, launched untagged, which reported itself as v1.9.0. Clients: Claude Code 2.1.235 on claude-sonnet-5
, Gemini CLI 0.55.1 on gemini-2.5-flash
, both model ids read back out of each trial's own client JSON rather than assumed from the flag. Claude Code also invokes claude-haiku-4-5
on every trial for its own bookkeeping, under a thousand input tokens a time. That never touches the MCP pipe and is in none of the figures below, but it is in the manifest, so it is worth knowing it is there.
The three Gemini failures are one per server and they are three different things: an off-by-one line count on FS-04, a response its own tool layer rejected on GH-05 after six calls reached the server, and a PW-01 final message that declared the task done without restating the price the check looks for.
Three trials per cell buys per-cell counts and nothing statistical, so these stay counts and never rates, per section 3.3 of the spec: Claude Code finished 45 of 45 trials, Gemini CLI 42 of 45. 87 of 90 state checks passed, and 87 of 90 trials also classify as tool_use_success
; the two are different fields that happen to agree this run. Three failures do not tell you one client is more reliable than the other.
Median call tokens per trial, meaning the tools/call
arguments plus the results the server sent back, counted with o200k_base
so the figure shares Tier 1's token basis. Each column is a median over 15 trials, five scripted tasks by three trials, so a different task mix moves these figures. They are not a per-server price:
| server | Claude Code | Gemini CLI |
|---|---|---|
| filesystem | 525 | 170 |
| playwright | 629 | 522 |
| github | 1,698 | 223 |
Both clients median one tool call on github. On GH-01 both called get_file_contents
with identical arguments, and the answer measured 1,561 result tokens for Claude Code against 161 for Gemini CLI.
Every response in the Claude Code session carried a _meta
block, io.modelcontextprotocol/serverInfo
, holding the server's name, version and two PNG icons inlined as base64: 2,215 characters wrapped around a 472-character answer. Across the 15 github trials on Claude Code, all 81 responses carried it. Across the 15 on Gemini CLI, none of the 93 did. The two sessions negotiated different protocol revisions, 2026-07-28
against 2025-06-18
. One server version against two clients does not prove the server keys on the revision. It does establish that a per-call cost figure is not a property of the server alone, which is the reason no row here publishes one number.
None of this is in the 90. The shakedown that preceded the run, one trial per task, ran on Gemini CLI 0.18.4, and its rows survive as superseded records in the same day's manifest. All five playwright tasks failed there, each trial with exactly one call on the wire. Read from the frames alone, the classifier bucketed it as a capability failure, once as a decline because the model apologised.
That reading is false. 0.18.4 validated each call's arguments against the schema the server advertised, and its bundled validator had no JSON Schema draft 2020-12 meta-schema registered. @playwright/mcp@0.0.79
declares 2020-12 on all 24 of its tools, so most of its calls died inside the client with no schema with key or ref "https://json-schema.org/draft/2020-12/schema"
and never reached the pipe: each trial got exactly one onto the wire and lost three or four more inside. The same client ran filesystem and github clean the same day, because @modelcontextprotocol/server-filesystem
declares draft-07 and github-mcp-server
declares no $schema
at all. The dialect decided it, not the server.
What caught it was the tool-call gap field, which exists for a different purpose (spec 4.2). The tool-call gap takes every tool name the server advertised, subtracts the wire frames the proxy logged from the calls the client's own usage output attributes to that tool, and sums the positive differences. Zero is the normal state. Those five trials read 3, 4, 4, 4 and 3, meaning the client formed calls to tools the server offered and those calls never left it. No classification could have said that, because a classification only sees what reached the wire.
Upstream had already fixed the validator in 0.28.0, before this shakedown ran (gemini-cli issue #14970, PR #15060): it dispatches a dedicated 2020-12 instance on the schema's own $schema
and falls back to skip-and-warn for unknown dialects. The stale client was mine. Upgrading to 0.55.1 and re-running the five tasks gave five passes, a zero gap on every trial, and no schema error in stderr.
The upgrade nearly broke the detector: the log keys the server's bare tool name, but the client's usage output moved to mcp_<server>_<tool>
between versions. Matched on the bare name alone, every difference came out negative, and the gap would have read a clean zero, a zero from a working detector and a zero from a broken one are the same character in the output. The runner now sums both spellings.
FS-04 asks the client to find the longest file in a fixture tree and write its path and line count to a file. The answer is logs/access.log
and 137. Gemini CLI passed two of three FS-04 trials. The third wrote logs/access.log
and 138, and reported the task complete.
The client made three calls, directory_tree
, read_multiple_files
, write_file
, all on the wire with a gap of zero, and the read_multiple_files
response carried the complete log: 137 lines, item-0001
through item-0137
, no truncation and no elision marker. That server concatenates files into one text block separated by a blank line and a marker line, so the last log line is followed by an empty line before the separator. The log cannot show which of those the model counted. It does show that nothing upstream lost or added a line.
Each of these produced a number that would have read as a capability result.
GH-03 came back from Claude Code with zero tool calls and this: "I have a standing instruction (from your global CLAUDE.md) that I never send outbound communications, including creating issues, myself." The trial measured my own memory file, and the classifier called it a hallucination. The runner now passes --setting-sources ""
and stamps the value in the run header.
FS-01 came back correct with zero frames on the wire, because Claude Code answered it with its own built-in Read
tool and the server never saw the task. Right answer, no measurement. That has a bucket of its own, answered_without_tools
, counted as a failure for every tool-use metric and never folded into a decline, and the runner now denies the built-in surface.
Gemini CLI answered a GitHub read task by posting the answer as a comment on the fixture issue, then saying only that it had completed the request. The check failed it on its merits, but the answer stayed in the repo and the fixture verifier still reported baseline, because nothing compared comment counts. The next trial would have found the previous one's answer waiting in the thread. The reset script now deletes comments and reports them as drift.
Eleven faults are enumerated in the spec: seven fixed on 2026-08-18, four more forced by the client upgrade the next day, two of which killed a run outright.
The proxy measures wire traffic on the MCP stdio pipe: the arguments of every tools/call
request plus the results the server sent back. That is all it can see.
It does not measure the tool-definition schema footprint any client loaded, because that happens inside the client and never crosses the pipe, and neither client exposes a session-start figure that isolates it: every token field they report contains the system prompt and the conversation alongside the tool definitions. The spec used to claim that metric and now records it as NOT CAPTURED
rather than dropping the row, so the hole stays visible.
So nothing here converts a MODELED label from Tier 1. Tool search stays modeled, because its total needs that per-session figure and a k
no client reports. Code mode stays modeled for a blunter reason: neither client has a code mode at all, so no trial was ever in one. The label rule used to make a Tier 2 run for a server the condition for relabelling it, which would have licensed exactly that mistake; it was corrected in methodology 0.3.1 before a run could trigger it.
Two more limits, both on the published file's face. Each figure is over five scripted tasks per server, so a different task mix moves the same server's call traffic. And the Tier 2 github pin is not the Tier 1 one: Tier 1 measured the remote endpoint, Tier 2 ran the ghcr.io
container untagged, so whatever latest
pointed at on 2026-08-18, with no image digest recorded; the server reported itself as v1.9.0
, and that row is reproducible to a self-reported version and not to an image, and the two github numbers are not the same artifact.
The run recorded one Gemini GH-03 trial as a client error, because Gemini reports an API-side failure by populating an error
object on an otherwise well-formed document. That trial's state check had already passed: exactly one open issue with the expected title, one tools/call
frame on the wire. The runner checked the error first and bucketed a trial that plainly succeeded as an infrastructure fault. The fix is a precedence rule: a passed success check outranks a client-reported error, and the error is kept as its own flag so neither hides the other. Every trial was then reclassified from its own stored fields rather than re-run, and exactly one moved, taking Gemini's tool_use_success
count from 41 of 45 to 42 of 45.
Post #6 promised one thing on the back of the fetch
correction: that the harness would record which dependency versions a resolve actually produced, so an unreachable
row could explain itself. That shipped in methodology 0.3.0. Every acquisition now records the resolved dependency set, the command that read it, and the environment it read from, plus the SDK version lifted out as its own field because that is what a reader of a broken row looks for first. The 2026-08-18 Tier 1 rows predate it; the next monthly run is the first to carry it.
Nothing here was metered: Claude Code rode plan quota, Gemini CLI an OAuth session on the free tier, and the servers ran local or in a container. Claude Code still reports a cost_usd
per trial, $4.85 across the 90, and that number is sitting in the manifest in the repo. It is what this work would have cost at API rates, not what it cost here. Both runners strip the provider API key variables out of the subprocess environment, so a stray key in a parent directory cannot quietly move a trial onto metered billing.
Repo: github.com/lopster568/loadline. Calculator: loadline-dev.netlify.app. The Tier 1 run is post #6.
slack
is still out; its operator credential has not landed. Three questions stay open in the spec:
Every trial in this run resolved to its pin. The container-digest gap in those github rows is closed from the next run on: since 2026-08-20 the runner resolves the image digest before the first trial and re-resolves it after the last, and a digest that moved mid-run stamps the run as drifted, the same rule already applied to client versions. The 2026-08-18 rows keep the tag they were recorded with. Nothing is republished.
If a number here is wrong, the frame logs, the per-trial client JSON, the manifest and the summary are in the repo, so disagreement can point at a line. Corrections go in the log with your name on them. I do this work for hire: auditing what your agent's tool surface costs before it does any work, and cutting it down without gutting what the search can still find. Scope and pricing at roshansingh.systems/#hire, or write to inbox@roshansingh.systems and tell me what your agents are .