{"slug": "i-measured-what-14-mcp-servers-cost-a-context-window-claude-counts-them-64-than", "title": "I measured what 14 MCP servers cost a context window. Claude counts them 64% higher than tiktoken", "summary": "A developer built a standing measurement tool called loadline to track how many context-window tokens MCP servers consume, running 72 trials across 14 servers. The measurements show that Claude's tokenizer counts schema text 47% to 70% higher than tiktoken's o200k_base, with a median increase of 64.1%, meaning published cost studies using tiktoken understate the context load for Claude agents by roughly 60%. The project publishes monthly reproducible runs and a stack calculator, with per-row auth scope and three counting modes.", "body_md": "Last month I [ran 72 trials](https://dev.to/lopster568/what-should-an-mcp-tool-return-i-ran-72-trials-instead-of-arguing-43b4) to settle what an MCP tool should return, because a maintainer would not take opinion for an answer. That left the other half open: before an agent does any work, how much of its context window have the attached servers eaten?\n\nVendors published numbers for this in 2026. I checked the six that get cited: exactly one is a real measurement study, StackOne's from 2026-03-31, which measured GitHub and Atlassian and worked through Cloudflare's code-mode case. Of the other five, one uses hypothetical unnamed servers, three contain no server-specific token study at all, and one I could not locate at any domain. So the published state of the art is a single snapshot, and it still shows March's numbers. A number that is not re-measured is a screenshot, and MCP servers change every few weeks. So I built a standing measurement, called loadline: 14 servers, methodology 0.2.0, one run dated 2026-08-18, monthly from here, every row reproducible from the repo with your own credentials. The front door is a stack calculator, not a table: the useful question is what my stack costs in the client I use.\n\nAll three count one fixed string, the canonical serialization from methodology section 1.5. Same bytes, three counters.\n\n| server | o200k_base | claude-opus-5 | premium |\n|---|---|---|---|\n| filesystem (14 tools) | 2,697 | 4,432 | +64.3% |\n| linear (53 tools) | 17,780 | 29,026 | +63.3% |\n| kubernetes (20 tools) | 4,430 | 7,552 | +70.5% |\n| github (47 tools) | 59,084 | 86,843 | +47.0% |\n\nAcross the 12 rows that produced counts the premium runs from 47.0 percent (github) to 70.5 percent (kubernetes), median 64.1, with 8 of the 12 between 60 and 66. Gemini stays within 11 percent of o200k on every row, so this is not a bigger-models-count-more effect. It is specific to Claude's tokenizer on schema text.\n\nEvery MCP cost study I have seen counts with tiktoken, the only one of the three that runs offline and free. If your agent runs on Claude, those studies describe a context load roughly 60 percent smaller than the one you pay for on the same schemas. They are not wrong about the bytes. They count them for a different model than the one in your loop.\n\nThe github row measures 47 tools at 59,084 tokens under o200k_base and 86,843 under claude-opus-5. That second figure is 43 percent of a 200k window, spent before the first user message.\n\nTwo qualifiers matter more than the number. First, 47 tools is what the server exposed to the classic personal access token this run used, and the spec lets a surface vary with the authorization presented, so the real GitHub surface is larger. Auth scope publishes per row, since that is the largest source of legitimate disagreement here.\n\nSecond, nobody on a tool-search client pays it. Claude Code has defaulted to progressive disclosure since January 2026, and on this surface that models out to roughly 4,300 to 6,800 tokens: a 500-token stub plus 3 to 5 tools at the measured 1,257-token average. A swing that size, driven entirely by the client, is why no row publishes one cost number. Three modes, always. And \"modeled\" is load-bearing there, which I get to below.\n\nCloudflare's aggregate endpoint answers `tools/list`\n\nwith three tools: `docs`\n\n(362 tokens), `search`\n\n(572), `execute`\n\n(660). Total 1,596 under o200k, 37 times smaller than github's naive load, and hygiene holds at B (85.19), so the compression did not come from deleting descriptions.\n\nThe qualifier: that is the aggregate endpoint measured as one server. Cloudflare also ships 16 product-scoped endpoints this run did not enumerate or sum.\n\nChrome DevTools MCP, official from Google and newly measured this run, exposes 52 tools, second only to linear's 53, at 7,984 tokens under o200k: 153 tokens per tool, the leanest average in the corpus. Playwright's 24 tools cost 167 each, filesystem's 14 cost 192, linear's 53 cost 335. Hygiene holds at B (80.28), so, as with Cloudflare, the leanness is not from deleting descriptions. A server can carry more surface than all but one other in the corpus and still land near the cheap end of it.\n\n`fetch`\n\n, the reference implementation, and `postgres`\n\nboth failed to launch from a clean install when the run executed, 2026-08-18 at 15:38 UTC. Each declares an unbounded dependency on the MCP Python SDK, `mcp>=1.1.3`\n\nfor fetch and `mcp[cli]>=1.5.0`\n\nfor postgres, so both take whatever the resolver hands them. A resolve that lands on `mcp`\n\n2.0.0 breaks both, in two different places.\n\nfetch imports a symbol that 2.0.0 renamed:\n\n``` python\nImportError: cannot import name 'McpError' from 'mcp.shared.exceptions'.\nDid you mean: 'MCPError'?\n```\n\npostgres dies one import earlier, on a module 2.0.0 no longer ships at that path:\n\n```\nModuleNotFoundError: No module named 'mcp.server.fastmcp'\n```\n\nThen, re-checking both rows about an hour later on the same machine with the resolver cache forced fresh, fetch came back. A clean resolve handed it `mcp`\n\n1.29.0 instead of 2.0.0, and it started. Nothing upstream was withdrawn: 2.0.0 is still the latest release and is not yanked. The same command, on the same machine, on the same afternoon, resolved differently. postgres still fails.\n\nSo the `fetch: unreachable`\n\nrow this post is built on went stale about an hour after it was written, and I am publishing it that way, labelled, with an entry in the corrections log, rather than quietly regenerating the dataset until it agrees with the prose. Next month's run will say whatever is true next month.\n\nThe more useful outcome is that it found a hole in my own instrument. The harness records whether the server package was pinned, but it never records which SDK version the resolve actually produced, so the artifact cannot explain why 15:38 and 16:45 disagree. That gets fixed before the next run.\n\nI could have pinned around the breakage in a minute and decided not to. Pinning changes what \"the fetch server\" means here, publishing a constrained old version while implying it describes what you would install today. Server rot is part of the subject matter, so it is part of the dataset.\n\n`figma`\n\ndoes not publish as a row at all this run: it failed gate 2 of the published selection rule, full surface enumerable under a free-tier credential, because its auth is OAuth-only and a reproducible harness has no credential to exercise that flow. It was excluded from the corpus rather than published as an auth row, and chrome-devtools was promoted to the freed slot.\n\nNotion's redesigned server is cheap: 24 tools, 5,180 tokens under o200k. It also scores hygiene D (54.79) and a top-3 retrievability fraction of 0.5417, the lowest measured here, with an MRR of 0.4527. What sinks it: `when_to_use_signal`\n\n0, `disambiguation`\n\n0, `parameter_descriptions`\n\n44.03. Context7, at 2 tools and 1,052 tokens, scores hygiene A (97.22), top-3 1.0, MRR 1.0.\n\nUnder a naive client that difference is invisible: both surfaces are just text in the prompt. Under a tool-search client it is the whole game, because a tool the search does not surface is a tool the agent does not have, however cheap it is. Rank on schema size alone and the winning move is gutting descriptions.\n\nEvery cell carries a MEASURED or MODELED label. Naive full-load is measured: the token count of the canonical serialization. The per-tool costs in the tool-search column are measured, but its total is modeled, because `k`\n\n, the number of tools a session pulls, is an assumption, which is why it publishes as a 3-to-5 range and not a point. Code mode is modeled end to end and keeps that label until a Tier 2 run validates it.\n\nRetrievability queries are derived from each tool's own description, so the metric measures within-server disambiguation, whether sibling tools shadow each other, not whether real phrasing finds a tool. It shares input text with the hygiene grade, so the two are not independent evidence.\n\nDollar figures are estimates, computed from a price table pending verification and shown only as a cold-write and cache-read pair. Tokens stay the primary unit: a single dollar figure is the cold-start number presented as though it recurs, overstating steady state by roughly 10x, since tool definitions are a near-perfect cache prefix. The rest is in `docs/methodology-v0.md`\n\n0.2.0, with the rejected alternative for each judgment call.\n\nNeutrality is not something to take on my word, so here are the checkable parts.\n\nSelection runs on a published rule: five binary gates (protocol compliance, full surface enumerable under a free-tier credential, verifiable adoption evidence, a release in the trailing six months, no official successor), plus a category spread and a 15-slot ceiling. Coverage is not a claim this project makes. Anyone can submit a server through a public issue and gets the gate result back, pass or fail, with the failing gate named.\n\nThe corrections log is public and append-only, and it has two entries before the first ranking: a mismatch between what the methodology said the harness did and what it did, caught in review before any release, and the `fetch`\n\nrow above. Before a server's numbers first publish, its maintainer gets 14 days with the rows, the artifacts, and the methodology version; a reply publishes verbatim beside the row, and non-response is recorded as \"no reply received\", not as agreement.\n\nEvery row ships three SHA-256 hashes of the tool surface plus the raw wire artifacts, so a disputed count is checked against the bytes it came from, not against my word. Recusal is published before the first ranking, and my own project is excluded from rankings by rule.\n\nRepo: [github.com/lopster568/loadline](https://github.com/lopster568/loadline). Calculator: [loadline-dev.netlify.app](https://loadline-dev.netlify.app).\n\nThe next run adds `slack`\n\n, once its operator credential lands, and starts Tier 2: scripted task runs through a call-logging proxy, measuring real call and response flows. Tier 2 is the capability axis, and if it stops running the leaderboard pauses, because a cost-only ranking is worse than none.\n\nIf a number here is wrong, tell me and bring the row. The harness, corpus file, artifacts, and derived query sets are in the repo, so disagreement can be specific, and corrections go in the log with your name on them.\n\nSTATUS: pre-release draft. Not published. All text pending owner review.\n\nloadline is a standing, versioned measurement of what MCP servers cost an\nagent's context window. It measures on a monthly cadence, run manually today\nper `docs/pc-sweep-runbook.md`\n\n, across a curated set of servers, counts\ntool-schema tokens with three tokenizer adapters, and reports the result\nthrough a stack calculator: pick servers, a client mode, and a model, and\nget total context footprint, window share, per-server attribution, and a\ncold-write/cache-read dollar pair. Published rows carry counts from all three\ntokenizers (OpenAI o200k locally, Claude and Gemini via their token-counting\nAPIs); a cell whose count could not be obtained publishes as\n`available: false`\n\n, never as an estimate. Costs\nare always reported per client mode (naive full-load, tool search /\nprogressive disclosure, code mode), never collapsed into one number\nbecause the same stack costs a different amount depending on how the\nclient…\n\nI do this work for hire: auditing what your agent's tool surface costs before it does any work, and cutting it down without gutting what the search can still find. Scope and pricing at [roshansingh.systems/#hire](https://roshansingh.systems/#hire), or write to [inbox@roshansingh.systems](mailto:inbox@roshansingh.systems) and tell me what your agents are loading.", "url": "https://wpnews.pro/news/i-measured-what-14-mcp-servers-cost-a-context-window-claude-counts-them-64-than", "canonical_source": "https://dev.to/lopster568/i-measured-what-14-mcp-servers-cost-a-context-window-claude-counts-them-64-higher-than-tiktoken-10pj", "published_at": "2026-08-18 17:11:54+00:00", "updated_at": "2026-08-18 17:43:17.373368+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence", "large-language-models"], "entities": ["Claude", "tiktoken", "StackOne", "Cloudflare", "Chrome DevTools MCP", "Google", "Playwright", "loadline"], "alternates": {"html": "https://wpnews.pro/news/i-measured-what-14-mcp-servers-cost-a-context-window-claude-counts-them-64-than", "markdown": "https://wpnews.pro/news/i-measured-what-14-mcp-servers-cost-a-context-window-claude-counts-them-64-than.md", "text": "https://wpnews.pro/news/i-measured-what-14-mcp-servers-cost-a-context-window-claude-counts-them-64-than.txt", "jsonld": "https://wpnews.pro/news/i-measured-what-14-mcp-servers-cost-a-context-window-claude-counts-them-64-than.jsonld"}}