{"slug": "a-copyable-mcp-tool-cost-check-for-ci-with-passing-and-failing-examples", "title": "A copyable MCP tool-cost check for CI, with passing and failing examples", "summary": "The maintainer of HISTOR, a tool-cost tracking project in the alexar76 ecosystem, published a copyable CI check that measures MCP server tool-definition token costs broken down by total, descriptions and input schemas rather than as a single number. The example reproduces HISTOR change 28848, a textokit remote MCP endpoint update observed on 24 October 2026 in which 17 of 20 tool definitions changed and full definitions fell from 5,565 to 3,776 tokens (-32.1%), driven mostly by a 1,375-token drop in descriptions while schemas shrank by only 414. The check pins aimarket-histor 0.3.1 and tiktoken 0.14.0 and fetches stored public tool lists without executing any server tools.", "body_md": "A tool's description can shrink while its schema grows. A single token total can hide that regression. Here is a complete example that checks the total, descriptions and input schemas separately, then shows how to use the same check in your existing CI job.\n\n**AI disclosure — Fully Autonomous:** AI agents prepared this example, its verification and this article under the project owner's direction. I maintain HISTOR in the alexar76 ecosystem. This is our own reproducibility check; it is not evidence that an external team has adopted the CI check.\n\n[Raju asked for a CI check](https://dev.to/alexar76/what-an-mcp-servers-tool-list-costs-we-counted-the-tokens-of-34374-of-them-3011#comment-3ij29) after our tool-definition cost survey. This page includes every command and the complete fetch script, so it can be used without a repository checkout.\n\n[HISTOR change 28848](https://histor.modelmarket.dev/api/v1/changes/28848/cost) records a change to textokit's remote MCP endpoint observed on 24 October 2026 at 19:52 UTC. Its before and after lists both have 20 tools; 17 definitions changed. We count compact `{name, description, input_schema}` definitions with `o200k_base`.\n\n| Tokens | Before | After | Change | \n|---|---|---|---|\n| Full definitions in the counted model-input form | 5,565 | 3,776 | -1,789 / -32.1% | \n| Descriptions | 2,158 | 783 | -1,375 | \n| Input schemas | 3,165 | 2,751 | -414 | \n| Names | 120 | 120 | 0 | \n| Keys and punctuation | 122 | 122 | 0 | \n\nThis is an observed remote definition change. It does not establish which npm release deployed it or imply an endorsement. The reproduction below downloads stored definition data. It does not run textokit code, call its tools or buy a service.\n\nUse Python 3.11. The example pins the published CLI and tokenizer versions rather than depending on the current latest release.\n\n```\nmkdir -p /tmp/histor-tool-cost-demo\ncd /tmp/histor-tool-cost-demo\npython3.11 -m venv .venv\n. .venv/bin/activate\npython -m pip install 'aimarket-histor[cost]==0.3.1' 'tiktoken==0.14.0'\n```\n\nInstallation, fetching the public lists and initially populating the tokenizer cache require network access. The recorded token-count validation itself ran without network access on a Linux server with Python 3.11.2 and tiktoken 0.14.0.\n\nFor precise provenance: those CLI checks ran in a HISTOR 0.4.0 container. We separately verified that its `histor/toolcost.py` and `histor/toolcost_cli.py` bytes are identical to the published 0.3.1 wheel used in the command above. This is not a claim of a separate fresh end-to-end installation test of that entire published wheel.\n\nCopy this shell block into the same directory. It writes `fetch_textokit.py` in full:\n\n```\ncat > fetch_textokit.py <<'PY'\n\"\"\"Fetch the immutable public tool lists of HISTOR change 28848; execute no server tools.\"\"\"\nfrom __future__ import annotations\n\nimport argparse\nimport json\nfrom pathlib import Path\nimport urllib.parse\nimport urllib.request\n\nfrom histor.subject import tool_set_digest\n\nSNAPSHOTS = {\n    \"before\": \"sha256-IP5W+kYoXogt7RDDPbG0FmUE97uEAqyIdxDoPjuocUA=\",\n    \"after\": \"sha256-zIV5QNaJd0IQp+5yFiYc4gtjQVAtMjfSc3Kq2tcE5uc=\",\n}\n\ndef main():\n    parser = argparse.ArgumentParser(__doc__)\n    parser.add_argument(\"--out\", type=Path, required=True)\n    args = parser.parse_args()\n    args.out.mkdir(parents=True, exist_ok=True)\n    for name, digest in SNAPSHOTS.items():\n        url = \"https://histor.modelmarket.dev/api/v1/toolsets/\" + urllib.parse.quote(digest, safe=\"\")\n        request = urllib.request.Request(url, headers={\"User-Agent\": \"HISTOR-cost-example/1.0\"})\n        with urllib.request.urlopen(request, timeout=30) as response:\n            raw = response.read(4 * 1024 * 1024 + 1)\n        if len(raw) > 4 * 1024 * 1024:\n            raise ValueError(\"tool list exceeds the example's size limit\")\n        tools = json.loads(raw)\n        if not isinstance(tools, list):\n            raise ValueError(\"expected a tool array\")\n        actual, _ = tool_set_digest(tools)\n        if actual != digest:\n            raise ValueError(f\"digest mismatch for {name}: {actual}\")\n        (args.out / f\"{name}-tools.json\").write_text(json.dumps(tools, ensure_ascii=False, indent=2) + \"\\n\")\n        print(f\"{name}: {len(tools)} tools, {actual}\")\n\nif __name__ == \"__main__\":\n    main()\nPY\n```\n\nThen fetch the pair:\n\n```\npython fetch_textokit.py --out /tmp/textokit-cost\n```\n\nThe script checks HISTOR's canonical tool-set digest for each list, not the SHA-256 of a particular JSON serialization. It refuses a changed digest or an oversized response. Both lists should contain 20 tools. No API key is required.\n\n```\nhistor tool-cost snapshot \\\n  --tools /tmp/textokit-cost/before-tools.json \\\n  -o /tmp/textokit-cost/baseline.json\n\nhistor tool-cost check \\\n  --baseline /tmp/textokit-cost/baseline.json \\\n  --tools /tmp/textokit-cost/after-tools.json \\\n  --max-increase 10%\n```\n\nExpected exit code: **0**. The total, descriptions and input schemas all became smaller. They are gated separately: shorter descriptions cannot conceal a large schema increase in the aggregate.\n\nDeliberately compare the same lists in the opposite direction:\n\n```\nhistor tool-cost snapshot \\\n  --tools /tmp/textokit-cost/after-tools.json \\\n  -o /tmp/textokit-cost/reduced-baseline.json\n\nhistor tool-cost check \\\n  --baseline /tmp/textokit-cost/reduced-baseline.json \\\n  --tools /tmp/textokit-cost/before-tools.json \\\n  --max-increase 10%\n```\n\nExpected exit code: **1**, with three reported problems:\n\nThis simulates returning to the older, larger definitions. It does not claim that a later textokit release regressed. Exit **2** means the input could not be obtained or counted, rather than a successful comparison. Growth below the default 50-token noise floor does not fail the gate; `--noise` can change that policy.\n\nAfter building your own server, create a baseline once and commit the JSON file:\n\n```\nhistor tool-cost snapshot --stdio 'node dist/index.js' -o tool-cost.json\n```\n\nUse your actual launch command. Unlike the stored-list example, `--stdio` launches your server and requests `initialize` and `tools/list`.\n\nIn a CI job that already installs dependencies and builds your project, add these steps:\n\n```\n- name: Install the pinned tool-cost checker\n  run: |\n    python3.11 -m venv .venv-tool-cost\n    .venv-tool-cost/bin/python -m pip install 'aimarket-histor[cost]==0.3.1' 'tiktoken==0.14.0'\n\n- name: Check tool-definition growth\n  run: |\n    .venv-tool-cost/bin/histor tool-cost check \\\n      --baseline tool-cost.json \\\n      --stdio 'node dist/index.js' \\\n      --max-increase 10%\n```\n\nThe runner needs Python 3.11 and the runtime your server uses. If your build already exports `tools/list`, replace `--stdio 'node dist/index.js'` with `--tools current-tools.json`. The command also writes its Markdown table to the job summary when `GITHUB_STEP_SUMMARY` is present.\n\nReview changes to the committed baseline when accepting growth. Do not regenerate it immediately before every CI check: that would compare the new list with itself. Keep tokenizer and CLI versions fixed on both sides of a baseline comparison; review and regenerate the baseline explicitly when changing them.\n\nThe check counts complete definitions in one documented input form. Clients that load tools on demand may send only a subset. Provider wrappers add tokens, and prompt-cache writes and reads have different billing rules. Actual invoices require the client's usage records.\n\nThe useful CI signal is bounded: a reviewed definition baseline changed enough to exceed your chosen limits. It does not judge tool quality, security or whether a larger schema is justified.", "url": "https://wpnews.pro/news/a-copyable-mcp-tool-cost-check-for-ci-with-passing-and-failing-examples", "canonical_source": "https://dev.to/alexar76/a-copyable-mcp-tool-cost-check-for-ci-with-passing-and-failing-examples-3hl5", "published_at": "2026-10-25 08:02:01+00:00", "updated_at": "2026-10-25 08:12:29.509817+00:00", "lang": "en", "topics": ["ai-agents", "agent-protocols", "ai-tools", "developer-tools", "mlops"], "entities": ["HISTOR", "alexar76", "textokit", "tiktoken", "aimarket-histor", "Raju"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-copyable-mcp-tool-cost-check-for-ci-with-passing-and-failing-examples", "markdown": "https://wpnews.pro/news/a-copyable-mcp-tool-cost-check-for-ci-with-passing-and-failing-examples.md", "text": "https://wpnews.pro/news/a-copyable-mcp-tool-cost-check-for-ci-with-passing-and-failing-examples.txt", "jsonld": "https://wpnews.pro/news/a-copyable-mcp-tool-cost-check-for-ci-with-passing-and-failing-examples.jsonld"}}