{"slug": "does-your-webmcp-actually-save-time-and-tokens-run-an-on-off-experiment", "title": "Does your WebMCP actually save time and tokens? Run an ON/OFF experiment", "summary": "A developer added a WebMCP ON/OFF ablation benchmark to DeepDeck, an MIT-licensed macOS desktop project built on DeepSeek Harness, to measure whether exposing structured browser tools actually reduces task time and token use. In a local batch of 49 tasks across 8 sites, the 46 tasks passed by all three models in both arms showed token reductions of 35.6%, 26.1%, and 22.7% for GPT-5.6 Terra, DeepSeek v4.1 Flash, and Hy3 respectively, with agent time falling 33.5%, 27.9%, and 22.9%. The runner freezes model configuration, uses clean browser profiles, alternates ON/OFF order, and explicitly leaves correctness unscored unless an independently checked expected answer is supplied.", "body_md": "I added a WebMCP benchmark to [DeepDeck](https://github.com/jo32/DeepDeck), my MIT-licensed macOS desktop project built on DeepSeek Harness. The question behind it is simple: **does exposing a structured browser tool reduce the time and tokens needed to finish a task correctly?**\n\nSuccessful tool registration only proves availability. It doesn't tell you whether the agent used the tool, whether the answer was right, or whether the extra tool description and interaction actually paid off.\n\nHere is a small experiment you can run on your own site.\n\nUse a DeepDeck source checkout and the repository's Node/pnpm setup:\n\n```\npnpm install\npnpm build:desktop\n```\n\nConfigure a working model in DeepDeck first. The runner uses an isolated snapshot of that configuration. See the [benchmark guide](https://github.com/jo32/DeepDeck/blob/main/docs/webmcp-benchmark.md) for provider overrides and setup details.\n\nThe URL-based experiment below does **not** require Docker, a registered benchmark site, or a task YAML file. The separate bundled 49-task corpus does require its site infrastructure.\n\nStart with a public, read-only page and a question whose answer you can check independently. Avoid tasks that mutate a remote account or depend on changing data: the runner starts clean browsers but cannot reset your remote site's backend.\n\nFor a minimal example, save this as `heading-webmcp.js`:\n\n```\nglobalThis.__deepdeckWebMCP.registerTool({\n  name: 'read_heading',\n  description: 'Read the main heading from the current page.',\n  inputSchema: {\n    type: 'object',\n    properties: {},\n    additionalProperties: false,\n  },\n  execute: async () => ({\n    content: [{\n      type: 'text',\n      text: JSON.stringify({\n        heading: document.querySelector('h1')?.textContent ?? null,\n        url: location.href,\n      }),\n    }],\n  }),\n});\n```\n\nThis is DeepDeck's local registration bridge for a custom tool bundle. The tool reads the page; it doesn't contain the expected answer. For a site that already exposes native WebMCP tools, omit the custom file in the next command.\n\n```\npnpm benchmark:webmcp ablate \\\n  --url https://example.com \\\n  --query 'What is the main heading?' \\\n  --webmcp-file ./heading-webmcp.js \\\n  --n 3\n```\n\nReplace the URL, question and tool with your real workflow after the smoke-sized example. This command makes model calls through your configured provider.\n\nBoth arms keep ordinary browser tools. The runner freezes the selected model configuration, creates fresh browser profiles and sessions, and alternates ON/OFF order across repetitions. The custom bundle is installed only in the ON arm; the OFF arm disables WebMCP and checks that its tool list is empty.\n\nIf no tools are discovered, it produces a **not applicable** report and exits without model calls. Existing signed-in cookies are not copied into these clean profiles.\n\nThe output includes `comparison.md` and `report.json`, with:\n\nWithout an expected answer, correctness is explicitly **unscored**. A completed run is not automatically a correct run. Check both answers before claiming that an efficiency change is useful.\n\nYou can add `--expected-answer 'your independently checked answer'`. That check ignores case and normalizes whitespace before testing text containment. It is useful for simple facts, but it is not a semantic evaluator. The expected answer stays with the runner and is not sent to the agent.\n\nFor richer tasks, use the editable corpus and independent answer/state predicates described in the guide.\n\nMy initial local batch covered 49 tasks across 8 sites. The efficiency comparison below uses the **same 46 tasks that all three models passed in both arms**, so it doesn't reward a model for skipping a difficult failed task.\n\n| Model | Token reduction with WebMCP | Agent time reduction | Agent steps OFF → ON | \n|---|---|---|---|\n| GPT-5.6 Terra | 35.6% | 33.5% | 413 → 268 | \n| DeepSeek v4.1 Flash | 26.1% | 27.9% | 708 → 572 | \n| Hy3 | 22.7% | 22.9% | 694 → 608 | \n\nAll-task pass counts, ON / OFF, were 49/49 / 49/49 for Terra, 48/49 / 47/49 for DeepSeek, and 49/49 / 48/49 for Hy3.\n\nThese are exploratory observations: one run per task per arm, fixed ON-then-OFF order in that published batch, and provider-default reasoning with the effective level unrecorded. The current repeated URL runner's alternating order should not be retroactively attributed to that batch. Providers, tokenizers and environments differ, and the results page documents environment and scorer limitations.\n\nTotal tokens include input, output, cache reads and cache writes. They are not a dollar-cost comparison; a provider can consume more tokens and still cost less.\n\nThe insight I want to investigate further is that **fast individual inference does not guarantee fast task completion**. Extra agent rounds, repeated page reading and tool waits can dominate. In this batch, Terra used fewer agent steps and finished sooner. That observation does not establish why a model was trained that way, or prove a general model ranking.\n\nStructured tools also make text-model browser interaction worth testing: many basic tasks can be expressed as useful tool calls. The right next experiment is your own task set, with independent correctness checks and repeated runs.\n\n[Full results and methodology](https://deepdeck.getmegaportal.com/benchmarks) · [Source and usage guide](https://github.com/jo32/DeepDeck/blob/main/docs/webmcp-benchmark.md)\n\nCredit to [nekuda-ai/WindTunnel](https://github.com/nekuda-ai/WindTunnel) for the starter corpus, site recipes and WebMCP patches. These are DeepDeck-local adaptations, not WindTunnel leaderboard results.\n\nIf you try this with a tool that makes a task slower, that is a useful result too. I'd like to hear what the tool exposed, whether the agent used it, and where the extra time went.", "url": "https://wpnews.pro/news/does-your-webmcp-actually-save-time-and-tokens-run-an-on-off-experiment", "canonical_source": "https://dev.to/jo32/does-your-webmcp-actually-save-time-and-tokens-run-an-onoff-experiment-3n9l", "published_at": "2026-09-17 08:02:19+00:00", "updated_at": "2026-09-17 08:23:31.715087+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "agent-protocols", "mlops"], "entities": ["DeepDeck", "DeepSeek Harness", "WebMCP", "GPT-5.6 Terra", "DeepSeek v4.1 Flash", "Hy3", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/does-your-webmcp-actually-save-time-and-tokens-run-an-on-off-experiment", "markdown": "https://wpnews.pro/news/does-your-webmcp-actually-save-time-and-tokens-run-an-on-off-experiment.md", "text": "https://wpnews.pro/news/does-your-webmcp-actually-save-time-and-tokens-run-an-on-off-experiment.txt", "jsonld": "https://wpnews.pro/news/does-your-webmcp-actually-save-time-and-tokens-run-an-on-off-experiment.jsonld"}}