Does your WebMCP actually save time and tokens? Run an ON/OFF experiment

I added a WebMCP benchmark to DeepDeck, my MIT-licensed macOS desktop project built on DeepSeek Harness. The question behind it is simple: does exposing a structured browser tool reduce the time and tokens needed to finish a task correctly?

Successful tool registration only proves availability. It doesn't tell you whether the agent used the tool, whether the answer was right, or whether the extra tool description and interaction actually paid off.

Here is a small experiment you can run on your own site.

Use a DeepDeck source checkout and the repository's Node/pnpm setup:

pnpm install
pnpm build:desktop

Configure a working model in DeepDeck first. The runner uses an isolated snapshot of that configuration. See the benchmark guide for provider overrides and setup details.

The URL-based experiment below does not require Docker, a registered benchmark site, or a task YAML file. The separate bundled 49-task corpus does require its site infrastructure.

Start with a public, read-only page and a question whose answer you can check independently. Avoid tasks that mutate a remote account or depend on changing data: the runner starts clean browsers but cannot reset your remote site's backend.

For a minimal example, save this as heading-webmcp.js:

globalThis.__deepdeckWebMCP.registerTool({
  name: 'read_heading',
  description: 'Read the main heading from the current page.',
  inputSchema: {
    type: 'object',
    properties: {},
    additionalProperties: false,
  },
  execute: async () => ({
    content: [{
      type: 'text',
      text: JSON.stringify({
        heading: document.querySelector('h1')?.textContent ?? null,
        url: location.href,
      }),
    }],
  }),
});

This is DeepDeck's local registration bridge for a custom tool bundle. The tool reads the page; it doesn't contain the expected answer. For a site that already exposes native WebMCP tools, omit the custom file in the next command.

pnpm benchmark:webmcp ablate \
  --url https://example.com \
  --query 'What is the main heading?' \
  --webmcp-file ./heading-webmcp.js \
  --n 3

Replace the URL, question and tool with your real workflow after the smoke-sized example. This command makes model calls through your configured provider.

Both arms keep ordinary browser tools. The runner freezes the selected model configuration, creates fresh browser profiles and sessions, and alternates ON/OFF order across repetitions. The custom bundle is installed only in the ON arm; the OFF arm disables WebMCP and checks that its tool list is empty.

If no tools are discovered, it produces a not applicable report and exits without model calls. Existing signed-in cookies are not copied into these clean profiles.

The output includes comparison.md and report.json, with:

Without an expected answer, correctness is explicitly unscored. A completed run is not automatically a correct run. Check both answers before claiming that an efficiency change is useful.

You can add --expected-answer 'your independently checked answer'. That check ignores case and normalizes whitespace before testing text containment. It is useful for simple facts, but it is not a semantic evaluator. The expected answer stays with the runner and is not sent to the agent.

For richer tasks, use the editable corpus and independent answer/state predicates described in the guide.

My initial local batch covered 49 tasks across 8 sites. The efficiency comparison below uses the same 46 tasks that all three models passed in both arms, so it doesn't reward a model for skipping a difficult failed task.

All-task pass counts, ON / OFF, were 49/49 / 49/49 for Terra, 48/49 / 47/49 for DeepSeek, and 49/49 / 48/49 for Hy3.

These are exploratory observations: one run per task per arm, fixed ON-then-OFF order in that published batch, and provider-default reasoning with the effective level unrecorded. The current repeated URL runner's alternating order should not be retroactively attributed to that batch. Providers, tokenizers and environments differ, and the results page documents environment and scorer limitations.

Total tokens include input, output, cache reads and cache writes. They are not a dollar-cost comparison; a provider can consume more tokens and still cost less.

The insight I want to investigate further is that fast individual inference does not guarantee fast task completion. Extra agent rounds, repeated page reading and tool waits can dominate. In this batch, Terra used fewer agent steps and finished sooner. That observation does not establish why a model was trained that way, or prove a general model ranking.

Structured tools also make text-model browser interaction worth testing: many basic tasks can be expressed as useful tool calls. The right next experiment is your own task set, with independent correctness checks and repeated runs.

Full results and methodology · Source and usage guide

Credit to nekuda-ai/WindTunnel for the starter corpus, site recipes and WebMCP patches. These are DeepDeck-local adaptations, not WindTunnel leaderboard results.

If you try this with a tool that makes a task slower, that is a useful result too. I'd like to hear what the tool exposed, whether the agent used it, and where the extra time went.