{"slug": "show-hn-maverik-benchmark-compare-and-cost-predict-your-mcp-agents", "title": "Show HN: Maverik – Benchmark, compare and cost-predict your MCP agents", "summary": "Maverik, an open-source tool for benchmarking and cost-predicting MCP agents, was released on Hacker News. It applies JMeter-style load testing to agent configurations, measuring correctness, speed, and cost across tunable parameters like system prompts and models. The tool aims to help developers answer questions such as whether a new prompt improves answers or just increases cost, and what 10,000 questions will cost at a given customer.", "body_md": "**Benchmark, compare, and cost-predict your MCP agents.**\n\n*Think JMeter — but for agents built on the Model Context Protocol.*\n\n[Quick start](#-quick-start) · [Frontend](#-frontend) · [Define a suite](#-defining-a-test-suite) ·\n[Run a benchmark](#-running-a-benchmark) · [Metrics](#-metrics) · [API](#-api-reference) [ ·\n](https://camo.githubusercontent.com/8320bd405869f93aeb889bb1453d3ede02c97c9519062cc25192e5b90b955cb1/68747470733a2f2f6465762e746f2f6c616e67656e736a6f6e617468616e2f646f2d756e757365642d6d63702d746f6f6c732d636f73742d796f752d6d6f6e65792d32676e62)[Roadmap](#-roadmap)\n\nJMeter exists because \"is my system fast enough?\" is a question you answer by measuring, not\nguessing — you define a test plan, throw load at your system, and read off the numbers that\ntell you whether a change made things better or worse. MAVERIK applies the same idea to MCP\nagents: instead of a system's throughput and latency under load, you're measuring an *agent\nconfiguration's* correctness, speed, and cost as you tweak it.\n\nAn agent configuration has a set of **tunable parameters** — the system prompt, the LLM\nmodel, which MCP servers/tools it can reach, the tool-loop strategy, the iteration cap, and\n(down the road) context-reduction strategies — and a set of **outcome parameters** you judge\nit on: whether it reaches an acceptable answer, how long that takes, how many input/output\ntokens it burns, and which tools it reaches for (some are far more expensive to call than\nothers). The MAVERIK workflow is to model the outcome parameters *first* — what counts as a\ncorrect answer, what latency and token budget is acceptable, which tools are \"free\" and which\nshould be used sparingly — and only then sweep the tunable parameters and compare.\n\nConcretely: you define *Agent Configurations* (system prompt, model, MCP servers, loop\nstrategy, iteration cap) and *Test Suites* of questions with pass criteria. MAVERIK fires\nevery question at every agent configuration and records:\n\n⏱️ Time |\nwall-clock duration of the full agent turn, tools included |\n🔢 Tokens |\ninput & output tokens, summed across every LLM round-trip in the loop |\n✅ Correctness |\ndeterministic checks or LLM-as-judge (judge cost tracked separately) |\n💰 Cost |\nestimated per-question and total cost from per-model pricing |\n\nSo instead of guessing, you can *measure* questions like:\n\n- Does the new version of my system prompt answer better — or just cost more?\n- Is Claude Haiku good enough for this scenario, or do I need Sonnet?\n- Does running tool calls in parallel actually make my agent faster?\n- What will 10,000 of these questions cost at customer X?\n\nIf you're new to the term: an **LLM agent** is a language model wired up to a loop and a set\nof tools. On its own, an LLM just turns text into text — it can't look anything up or take\naction. Give it a set of tools (functions it can call — read a file, hit an API, query a\ndatabase) and a loop that feeds each tool's result back to it, and it can work multi-step\nproblems: decide it needs information, call a tool, read what came back, decide whether\nthat's enough or it needs another tool, and eventually answer. That loop — *call a tool or\nanswer, repeat until done* — is what makes something an agent rather than a one-shot\ncompletion.\n\nMAVERIK's MCP host implements exactly that loop (see `maverik/src/chat/ChatWorker.cs`\n\n/\n`maverik/src/loop/LoopStrategy.cs`\n\n), and the tools it hands the model come from one or more\n**MCP servers** — servers exposing typed, discoverable functions over the\n[Model Context Protocol](https://modelcontextprotocol.io).\n\nMAVERIK's specific definition of \"agent\" is an **Agent Configuration** — the `AgentConfig`\n\nobject (`maverik/src/agents/AgentConfig.cs`\n\n) defined in `config/agents.json`\n\n. It's a named\nbundle of everything that determines how the loop behaves for a given use case:\n\n| Field | Meaning |\n|---|---|\n`systemPrompt` |\nhow the agent is primed/instructed (inline or `config/prompts/agent/<id>.md` ) |\n`model` |\nwhich LLM answers, from `config/llm-models.json` |\n`mcpServers` |\nwhich MCP servers' tools it's allowed to reach for |\n`loopType` |\nwhich `ILoopStrategy` drives its tool loop (`manual` , `parallel-tools` , ...) |\n`maxIterations` |\nhow many LLM round-trips it gets before MAVERIK gives up on it |\n\nThe important part: **an agent is data, not code.** Two agents that differ only in their\nsystem prompt, or only in their model, are two entries in `config/agents.json`\n\n— not two\ncodebases. That's what makes A/B testing possible: point the same test suite at, say,\n`github-helper`\n\nand `github-helper-v2-prompt`\n\n, and any difference in the results is\nattributable to the one thing you changed.\n\n**Agent configurations as data**— prompt, model, MCP servers, loop type, and iteration cap live in`config/agents.json`\n\n; comparing two agents is a config edit, not a code change.**Multi-provider model registry**— Anthropic and any OpenAI-compatible endpoint (OpenAI, Ollama, LM Studio, vLLM, …) side by side in`config/llm-models.json`\n\n.**Pluggable host-loop strategies**—`manual`\n\n(sequential tool calls) and`parallel-tools`\n\n(concurrent tool calls per turn) out of the box, behind a small`ILoopStrategy`\n\nseam.**Four criterion types**—`exact`\n\n,`contains`\n\n,`regex`\n\n, and`llm-judge`\n\nwith a free-text rubric. Judge tokens are measured but**never** pollute the agent's metrics.**Repetitions**— run each case N times to see through LLM nondeterminism.** Results that persist**— every run writes`results/{runId}/run.json`\n\n+`summary.csv`\n\n, ready for Excel, pandas, or your BI tool of choice.**Cost prediction**— attach per-MTok pricing to models and get estimated cost per question, per agent, per run.** Interactive chat REPL included**— poke at any agent configuration by hand, in the dashboard's`Chat`\n\ntab, before you benchmark it.**Wire-level debug logging (\"dev mode\")**— toggle at runtime (dashboard header, or`POST /api/dev-mode`\n\n) and every raw LLM HTTP exchange (including judge traffic) is logged with timings and token counts — off by default, see[Debugging / Dev mode](#-debugging--dev-mode).**Docker-first deployment**— one`docker compose up`\n\nwith secrets mounted, never baked in.\n\n```\n                     ┌──────────────────────────────────────────────────────┐\n POST /api/maverik/runs ─► run queue ─► MaverikRunner                       │\n                     │                     │  for each agent × question × N │\n                     │                     ▼                                │\n                     │           ILoopStrategy ◄──── config/agents.json     │\n                     │             (manual / parallel)                      │\n                     │                 │         │                          │\n                     │       LLM models│         │MCP tool calls            │\n                     │ (config/llm-models.json)  (config/mcp-servers.json)  │\n                     │                     │                                │\n                     │                     ▼                                │\n                     │             CriterionEvaluator (exact/contains/      │\n                     │                     │           regex/llm-judge)     │\n                     │                     ▼                                │\n GET /api/maverik/runs/{id} ◄── run store + results/{runId}/ (json + csv)   │\n                     └──────────────────────────────────────────────────────┘\n```\n\nThe `maverik-frontend`\n\ndashboard is just a browser-side client of this same API — it calls\n`GET /api/maverik/suites/{id}`\n\nto render a test plan, `POST /api/maverik/runs`\n\nto execute it, and\npolls `GET /api/maverik/runs/{id}`\n\n+ `.../summary`\n\nto show progress and the side-by-side\ncomparison. It runs in its own container and talks to `maverik`\n\nover HTTP from the browser, not\ncontainer-to-container — see [Frontend](#-frontend) below.\n\nEvery question runs in **complete isolation**: a fresh conversation seeded with the agent's\nsystem prompt, no shared history, no sessions. The runner executes cases **sequentially** so\ntiming numbers stay clean. And critically, the benchmark runner and the interactive chat mode\nshare the *same* loop code — what you measure is what you ship.\n\nMAVERIK is Docker/Compose-only — there's no supported bare-metal path, just config and a container.\n\n[Docker](https://docs.docker.com/get-docker/)with Compose- At least one reachable MCP server (HTTP / streamable-HTTP transport)\n- An API key for Anthropic and/or any OpenAI-compatible endpoint\n\nEvery real config file lives in `./config/`\n\nand has a committed `*.example.*`\n\ntemplate next to\nit — copy each one and edit the copy:\n\n```\ngit clone <this-repo>\ncd maverik\n\ncp config/.env.example config/.env                             # secrets: API keys, tokens\ncp config/llm-models.example.json config/llm-models.json       # LLM models, referencing .env by name\ncp config/mcp-servers.example.json config/mcp-servers.json     # MCP servers to connect to\ncp config/agents.example.json config/agents.json               # agent configurations under test\ncp docker-compose.example.yml docker-compose.yml               # ports, mounts, env_file\n```\n\nSecrets never go in JSON config files — put real values in `config/.env`\n\n(gitignored), then\nreference them by name as `${VAR_NAME}`\n\nin `mcp-servers.json`\n\nheaders or `llm-models.json`\n\n's\n`apiKey`\n\nfield. `docker-compose.yml`\n\nloads `config/.env`\n\nstraight into the container via\n`env_file:`\n\n, so any variable you add there is available for expansion without touching the\ncompose file. **Never paste real secrets into a chat/terminal session that gets logged — edit\nconfig/.env directly in a file editor.**\n\n```\ndocker compose up -d --build\n```\n\n`./config`\n\nis bind-mounted read-write (so the dashboard's config editor — see below — can save\nchanges) and `results/`\n\n/`logs/`\n\nare mounted read-write so your benchmark data survives the\ncontainer. Remember that *inside* the container,\n`localhost`\n\nis the container — MCP servers running on your host machine are reached via\n`http://host.docker.internal:...`\n\n(already wired up in the compose template).\n\nThrough the dashboard: open a test plan, pick agents/repetitions, click **Start run**, watch it\nprogress and compare results. Or drive the API directly:\n\n```\ncurl -X POST http://localhost:5088/api/maverik/runs \\\n     -H \"Content-Type: application/json\" \\\n     -d '{ \"suiteId\": \"github-basics\", \"repetitions\": 3 }'\n# → { \"runId\": \"github-basics-20260709-141502\" }\n\ncurl http://localhost:5088/api/maverik/runs/github-basics-20260709-141502/summary\n```\n\nWant a slower, hands-on walkthrough instead? [ TUTORIAL.md](/langens-jonathan/maverik/blob/master/TUTORIAL.md) takes you from a fresh\nclone all the way through duplicating an agent, running the built-in smoke suite against both\nversions, and building a comparison report — the full loop, step by step.\n\n`./frontend/`\n\nis a small React (Vite) dashboard, shipped as its own `maverik-frontend`\n\nCompose\nservice — the `maverik`\n\ncontainer stays backend-only. `docker compose up -d --build`\n\nbuilds and\nruns both; the frontend is a static build served by nginx, and it talks to `maverik`\n\n's API\ndirectly from your browser (not container-to-container), so no MCP-style networking concerns\napply here.\n\nIt gives you: **Test plans** (a suite's questions and criteria — the test plan itself), a **run\nform** on each test plan (pick agents/repetitions, start a run), **Runs** (live progress while a\nrun executes, then the per-agent comparison — pass rate, duration, tokens, estimated cost — plus\na per-case results table once it's done), **Chat** (a REPL for talking to one agent by hand — see\n[Interactive chat REPL](#-interactive-chat-repl) below), **Config** (structured editors for\n`agents.json`\n\n, `llm-models.json`\n\n, `mcp-servers.json`\n\n, MAVERIK suites, tool costs, and per-agent\nprompt files — see Configuration below), and **Reporting** (visualizations, composed into\ndashboards, filtered and selected via reports — see [Reporting](#-reporting) below).\n\nBecause the API URL is baked into the static build at image-build time (Vite's `VITE_API_BASE_URL`\n\n,\nset via the `args:`\n\nblock in `docker-compose.yml`\n\n), if you change `maverik`\n\n's published port from\nthe default `5088`\n\n, update both that `args:`\n\nvalue *and* the `maverik`\n\nservice's\n`MAVERIK_FRONTEND_ORIGIN`\n\n(used for CORS) to match your `maverik-frontend`\n\nport, then rebuild.\n\nEverything user-editable lives under `./config/`\n\n, bind-mounted read-write into the container.\nEvery real JSON file below is gitignored and has a matching committed `config/*.example.*`\n\ntemplate to copy from — and if you skip that step, the app copies it for you: a missing\n`agents.json`\n\n/`llm-models.json`\n\n/`mcp-servers.json`\n\nis bootstrapped from its `.example`\n\nsibling on\nfirst read, whether that read happens at container startup or from the dashboard's **Config**\ntab. The dashboard also lets you edit all three of those files, plus per-agent prompt files,\nthrough structured forms (`GET`\n\n/`PUT /api/config/agents`\n\n, `/llm-models`\n\n, `/mcp-servers`\n\n,\n`/prompts/{agentId}`\n\n— see API reference). Saves write straight to the files below.\n`agents.json`\n\n/prompts and `llm-models.json`\n\nedits **apply immediately, no restart needed** — the\nunderlying registries rebuild themselves in place and swap in atomically, so an in-flight chat or\nbenchmark run keeps using whatever it started with while new requests pick up the change. If the\nnew config doesn't actually work (e.g. an agent with no system prompt anywhere, or a model with a\nbad endpoint), the file is still saved but the bad part is reported back and the previous\nin-memory config keeps serving — nothing crashes. `mcp-servers.json`\n\nis the one exception:\n`McpServerRegistry`\n\nalso owns live connections to each server, and reconnecting safely while\nrequests may be in flight is more involved, so **that one still needs docker compose restart maverik** after a save (the dashboard says so explicitly).\n\n| File | What it defines |\n|---|---|\n`config/.env` |\nReal secret values (API keys, tokens). Copy from `config/.env.example` . Loaded into the container by `docker-compose.yml` 's `env_file:` . |\n`config/llm-models.json` |\nLLM models across providers, plus optional per-MTok pricing; `apiKey` supports `${VAR_NAME}` expansion against `.env` . Copy from `config/llm-models.example.json` . |\n`config/mcp-servers.json` |\nMCP servers (name, HTTP endpoint, headers with `${VAR_NAME}` expansion against `.env` ). Copy from `config/mcp-servers.example.json` . |\n`config/agents.json` |\nThe agent configurations under test. Copy from `config/agents.example.json` . |\n`config/maverik-suites/*.json` |\nTest suites: questions, criteria, default agent set, judge model. |\n`config/prompts/agent/<id>.md` |\nAn agent's system prompt, when not defined inline. |\n`config/reporting/visualizations/<id>.js` |\nVisualization functions, authored directly on disk (no PUT/UI editor) — see\n|\n\n`config/reporting/dashboards/<id>.json`\n\n`Reporting > Dashboards`\n\n— see [Dashboards](#dashboards).`config/reporting/reports/<id>.json`\n\n`Reporting > Reports`\n\n— see [Reports](#reports).Both `mcp-servers.json`\n\nheaders and `llm-models.json`\n\n's `apiKey`\n\nshare the same `${VAR_NAME}`\n\nexpansion (`maverik/src/config/EnvExpansion.cs`\n\n): a variable referenced but not defined in\n`config/.env`\n\nfails startup with a clear error rather than sending an empty credential.\n\n```\n{\n  \"id\": \"github-helper-v2\",\n  \"name\": \"GitHub Helper (v2 prompt)\",\n  \"description\": \"Tighter prompt; should reduce tool-call count.\",\n  \"model\": \"claude-sonnet\",           // an id from config/llm-models.json\n  \"loopType\": \"parallel-tools\",       // \"manual\" (default) or \"parallel-tools\"\n  \"mcpServers\": [ \"github\" ],         // names from config/mcp-servers.json — the agent only sees these tools\n  \"maxIterations\": 8                  // cap on LLM round-trips per question\n}\n```\n\nThe system prompt is either inline (`\"systemPrompt\": \"...\"`\n\n) or in\n`config/prompts/agent/github-helper-v2.md`\n\n— perfect for versioning prompt experiments in git.\n\n```\n{\n  \"id\": \"claude-sonnet\",\n  \"provider\": \"anthropic\",\n  \"model\": \"claude-sonnet-5\",\n  \"apiKey\": \"${ANTHROPIC_API_KEY}\",   // expanded against config/.env at model-load time\n  \"inputPricePerMTok\": 3.00,          // optional — enables cost estimation\n  \"outputPricePerMTok\": 15.00\n}\n```\n\nOne file per suite in `config/maverik-suites/`\n\n:\n\n```\n{\n  \"id\": \"github-basics\",\n  \"name\": \"GitHub basics\",\n  \"description\": \"Sanity checks against the GitHub MCP server.\",\n  \"agents\": [ \"github-helper\", \"github-helper-v2\" ],   // default set; overridable per run\n  \"judgeModel\": \"claude-haiku\",                        // used by llm-judge criteria\n  \"questions\": [\n    {\n      \"id\": \"default-branch\",\n      \"text\": \"What is the default branch of repo X?\",\n      \"criterion\": { \"type\": \"exact\", \"expected\": \"main\", \"caseSensitive\": false }\n    },\n    {\n      \"id\": \"open-issues\",\n      \"text\": \"How many open issues does repo X have?\",\n      \"criterion\": { \"type\": \"regex\", \"pattern\": \"\\\\b12\\\\b\" }\n    },\n    {\n      \"id\": \"release-summary\",\n      \"text\": \"Summarize the latest release notes of repo X.\",\n      \"criterion\": {\n        \"type\": \"llm-judge\",\n        \"rubric\": \"PASS if the answer mentions the 2.0 release and at least two of its features.\"\n      }\n    }\n  ]\n}\n```\n\n| Type | Fields | Passes when… |\n|---|---|---|\n`exact` |\n`expected` , `caseSensitive?` |\nthe trimmed final answer equals `expected` |\n`contains` |\n`expected` , `caseSensitive?` |\nthe final answer contains `expected` |\n`regex` |\n`pattern` |\nthe final answer matches `pattern` |\n`llm-judge` |\n`rubric` , `judgeModel?` |\nthe judge model returns `PASS` against the rubric |\n\nThe judge runs on a fresh, tool-less conversation at temperature 0 and must answer in strict\nJSON (`{\"verdict\": \"PASS\", \"reasoning\": \"...\"}`\n\n). Its token usage is recorded — but as\n*testing overhead*, never as part of the agent's score.\n\nSuites are validated at startup: unknown agent ids, unknown judge models, invalid regexes, or missing criterion fields fail fast with a clear message.\n\nThrough the dashboard, a suite's page shows its questions/criteria plus a run form — pick which\nagents to include, set repetitions, hit **Start run**:\n\n...and the run's page compares them live as results come in — pass rate, duration, tokens, and cost, per agent, side by side:\n\nOr drive the same thing over the API directly:\n\n```\n# Start a run (agents defaults to the suite's list; repetitions defaults to 1)\nPOST /api/maverik/runs\n{ \"suiteId\": \"github-basics\", \"agentIds\": [\"github-helper\", \"github-helper-v2\"], \"repetitions\": 3 }\n\n# Poll progress + per-case results\nGET /api/maverik/runs/{runId}\n\n# The payoff: per-agent aggregates, side by side\nGET /api/maverik/runs/{runId}/summary\n```\n\nA summary looks like:\n\n```\n{\n  \"runId\": \"github-basics-20260709-141502\",\n  \"agents\": [\n    {\n      \"agentId\": \"github-helper\",\n      \"passRate\": 0.89,\n      \"avgDurationMs\": 6420,\n      \"avgInputTokens\": 3812, \"avgOutputTokens\": 402,\n      \"avgIterations\": 2.6, \"avgToolCalls\": 1.8,\n      \"estCostPerQuestion\": 0.0175, \"estCostTotal\": 0.157,\n      \"errors\": 0, \"casesWithoutUsage\": 0\n    },\n    {\n      \"agentId\": \"github-helper-v2\",\n      \"passRate\": 0.89,\n      \"avgDurationMs\": 4110,                     // ← the v2 prompt is faster…\n      \"avgInputTokens\": 2954, \"avgOutputTokens\": 371,\n      \"avgIterations\": 1.9, \"avgToolCalls\": 1.2, // ← …because it calls fewer tools\n      \"estCostPerQuestion\": 0.0124, \"estCostTotal\": 0.112,\n      \"errors\": 0, \"casesWithoutUsage\": 0\n    }\n  ],\n  \"judgeOverhead\": { \"inputTokens\": 5210, \"outputTokens\": 640, \"estCost\": 0.006 }\n}\n```\n\nEvery run is also written to disk:\n\n```\nresults/\n└── github-basics-20260709-141502/\n    ├── run.json       # full per-case detail\n    ├── summary.json   # the aggregate above\n    └── summary.csv    # one row per case — Excel/pandas ready\n```\n\nCaptured per **case** (agent × question × repetition):\n\n| Metric | Notes |\n|---|---|\n`durationMs` |\nfull turn: LLM round-trips and MCP tool time |\n`inputTokens` / `outputTokens` |\nsummed over every LLM call in the loop; `null` (not 0) when a provider reports no usage |\n`iterations` |\nLLM round-trips used |\n`toolCallCount` / `toolNames` |\nwhich tools the agent actually reached for |\n`hitIterationLimit` |\nthe loop was cut off before a final answer |\n`passed` + `evaluationDetail` |\ncriterion outcome (judge reasoning for `llm-judge` ) |\n`judgeInputTokens` / `judgeOutputTokens` |\ntracked separately from agent metrics |\n`error` |\na failing case is recorded and the run continues |\n\nThe MCP host loop is hand-driven (no SDK auto-invocation), which is what makes it measurable and swappable:\n\n`loopType` |\nBehavior |\n|---|---|\n`manual` |\nThe classic loop: model responds → requested tools run sequentially → results fed back → repeat until a final answer (or `maxIterations` ). |\n`parallel-tools` |\nSame loop, but when the model requests several tools in one turn they run concurrently — often a big latency win on I/O-heavy MCP servers. |\n\nNew strategies implement one small interface (`ILoopStrategy`\n\n) and become available as a\n`loopType`\n\nvalue — comparing loop designs is then just another benchmark run.\n\nBefore benchmarking an agent, talk to it. `Chat`\n\nin the dashboard is a minimal REPL: pick an\nagent from a dropdown, type, watch progress lines (`(calling get_issues ...)`\n\n) arrive as the\nagent works, then the final answer. Switching the agent dropdown — or hitting \"New\nconversation\" — starts a clean conversation; nothing carries over.\n\nUnder the hood it's the same simple polling chat API the backend has always had, unchanged in shape:\n\n```\nPOST /api/chat                # { \"message\": \"...\", \"agent\": \"github-helper-v2\" } → accepted\nGET  /api/messages             # poll: progress lines + final answer\n```\n\nBoth require an `X-Session-Id`\n\nheader — an opaque id the frontend mints itself (a UUID), not a\ncookie. There's no `/api/session`\n\nendpoint to call first: an id nobody's used before just starts\na fresh conversation the moment you send the first message under it. This is deliberate — the\nbackend is **API-only** (no static front end, no `wwwroot/`\n\n, no server-side session state tied\nto a browser cookie), so the same client-supplied-id approach works identically whether you're\ndriving it from the dashboard, a script, or a CI job (see\n[CI/CD integration](#-cicd-integration)).\n\n| Method | Route | Purpose |\n|---|---|---|\n| GET | `/api/agents` |\nList agent configurations (id, name, description, model, loop type, servers). |\n| GET | `/api/tools` |\nThe aggregated MCP tool catalog, grouped by server. |\n| GET/POST | `/api/dev-mode` |\nView/toggle wire-level LLM logging at runtime — see\n|\n\n`/api/config/agents`\n\n`agents.json`\n\n(bootstraps from example if missing). PUT body/response is `AgentsFile`\n\n.`/api/config/llm-models`\n\n`llm-models.json`\n\n.`/api/config/mcp-servers`\n\n`mcp-servers.json`\n\n.`/api/config/prompts/{agentId}`\n\n`config/prompts/agent/{agentId}.md`\n\n.`/api/maverik/suites`\n\n`/api/maverik/suites/{id}`\n\n`/api/maverik/runs`\n\n`{ suiteId, agentIds?, repetitions? }`\n\n→ `{ runId }`\n\n.`/api/maverik/runs`\n\n`/api/maverik/runs/{id}`\n\n`/api/maverik/runs/{id}/summary`\n\n`/api/maverik/suite-runs?suiteIds=&from=&to=`\n\n[Reporting](#-reporting)visualizations consume.`/api/reporting/visualizations`\n\n`/api/reporting/visualizations/{id}`\n\n[Reporting](#-reporting).`/api/reporting/dashboards`\n\n`/api/reporting/dashboards/{id}`\n\n`/api/reporting/dashboards[/{id}]`\n\n[Dashboards](#dashboards).`/api/reporting/reports`\n\n`/api/reporting/reports/{id}`\n\n`/api/reporting/reports[/{id}]`\n\n[Reports](#reports).`/api/chat`\n\n`X-Session-Id`\n\nheader (any client-minted opaque id — see [Interactive chat REPL](#-interactive-chat-repl)).`/api/messages`\n\n`X-Session-Id`\n\nheader required.**Dev mode** logs every raw LLM HTTP exchange — agent *and* judge traffic, covering both\ninteractive chat and MAVERIK runs — to `logs/{sessionId|runId}.log`\n\non the host (via the\n`./logs:/app/logs`\n\nmount), with method, endpoint, full request/response bodies, round-trip time,\nand token usage. In effect, this is the full conversation history for every session/run, in raw\nform, persisted to disk.\n\nOff by default, zero overhead when off. It's a **runtime toggle**, not just a startup setting:\nflip it via the dashboard header's \"Dev mode\" button, or directly:\n\n```\ncurl -X POST http://localhost:5088/api/dev-mode -H \"Content-Type: application/json\" -d '{\"enabled\": true}'\n```\n\n`config/.env`\n\n's `MCPHOST_LLM_DEBUG=1`\n\nstill works — it just sets the *starting* value at\ncontainer boot; the toggle can change it from there without a restart.\n\n** ⚠️ Turning dev mode on mid-run only logs what happens from that point on.** If a MAVERIK run\nis already in progress when you flip the switch, cases that already completed before the toggle\nwere never logged —\n\n`logs/{runId}.log`\n\nfor that run will be missing their exchanges, i.e. an\n**incomplete conversation record** for that run. Toggle it on\n\n*before*starting a run if you need the full picture.\n\nThere's no UI yet to browse what's been logged — for now, read the files directly under `logs/`\n\n.\nA viewer is planned.\n\nThe dashboard's **Reporting** tab lets you define reusable **visualizations** — small JavaScript\nfunctions you write yourself, rendering either a chart or a table (there's no separate \"table\"\ntype — both have the identical contract, so it's just a matter of what a given function draws) —\nand preview them against real MAVERIK results. This is the data-visualization layer for comparing\nruns. **Dashboards** compose several visualizations into titled sections — see below.\n**Reports** filter/select suite-runs and feed them into a dashboard — also below. A saved report\nresolved against a dashboard is the screenshot at the top of this README — the same one, again:\n\nA visualization is a file under `config/reporting/visualizations/<id>.js`\n\n, id = filename. Unlike\nevery other editable config in MAVERIK, **there's no PUT endpoint or in-app editor for these** —\nwrite/edit them directly on disk (`config/`\n\nis already bind-mounted read-write), then refresh the\nReporting tab. Every file's default export follows one fixed contract:\n\n```\nexport default function (container, data, { d3, fullWidth, halfWidth }) {\n  // container: an empty <div> the dashboard created for this visualization — render into it.\n  // data: an array of SuiteRunRecord (results/suite-runs/*.json).\n  // { d3 }: the library the host injects — don't `import` anything yourself.\n  // fullWidth/halfWidth: pixel budgets for a full-row vs. half-row slot, where the host lays\n  // visualizations out in a grid (see \"Sizing & layout\" in config/reporting/README.md).\n}\n```\n\nA file can also export `const layout = \"full\"`\n\n(default `\"half\"`\n\n) so a two-column host knows\nwhether to give it the whole row or share it with a neighbor — tables set this and otherwise don't\nneed to do anything (`table { width: 100% }`\n\nalready fills the slot), while charts size their own\nSVG off `halfWidth`\n\n. The report \"Open\" screen (`/reporting/reports/:id`\n\n) is the one page that lays\nthings out this way, on a wider 1250px layout; every other page still renders one visualization\nper row at the usual width.\n\nExecution happens entirely in the browser (fetch the raw source, dynamic-`import()`\n\nit from a\n`Blob`\n\nURL) and is **deliberately not sandboxed** — MAVERIK is a self-hosted, single-user tool,\nso this is trusted the same way system prompts and suite definitions already are. The one rule to\nfollow anyway: only touch `container`\n\nand the arguments you're given — no `window`\n\n/`document`\n\nreach-out, no `fetch`\n\n. That discipline is what would keep a future move to iframe-sandboxed\nexecution a plumbing change instead of a rewrite of every visualization you've written. See\n`config/reporting/README.md`\n\nand the two hand-written examples (`avg-duration-by-agent.js`\n\n, a D3\nbar chart, and `results-table.js`\n\n, a plain `<table>`\n\n) for the full contract and a working starting\npoint.\n\nBeyond those two examples, MAVERIK ships **21 default visualizations** covering the 9 outcome\nparameters every `AgentSummary`\n\ntracks (pass rate, duration, input/output tokens, tool calls, peak\ncontext tokens, token/tool/overall cost): a `runs-over-time-<metric>.js`\n\nand an\n`agent-average-<metric>.js`\n\nline chart per metric, plus three tables — `metrics-by-agent.js`\n\n,\n`metrics-by-run.js`\n\n, and `question-details.js`\n\n(per-question detail, sourced from a `results`\n\nfield\neach `SuiteRunRecord`\n\nnow carries — that agent's slice of the source run's per-case results, copied\nin at write time so no visualization ever needs to `fetch`\n\nback to `run.json`\n\n). Four more lean on\nthat same `results`\n\ndata for comparative views the single-metric grid can't do:\n`question-pass-rate-matrix.js`\n\n(a question × agent pass-rate heatmap, worst questions first),\n`cost-vs-correctness.js`\n\n(a cost-vs-accuracy scatter, one point per agent), `tool-call-frequency.js`\n\n(call count per tool name), and `reliability-by-agent.js`\n\n(error rate and iteration-limit-hit rate\nper agent). See `config/reporting/README.md`\n\nfor the full list.\n\nA dashboard has a title and an ordered list of **sections**, each with its own title and a list\nof visualization references (a visualization id plus an optional per-instance caption). Unlike\nevery other config type in MAVERIK, dashboards **aren't editable-config-with-a-registry** — they\nhave full CRUD through the dashboard (`Reporting > Dashboards`\n\n, or `GET/POST/PUT/DELETE /api/reporting/dashboards[/{id}]`\n\ndirectly), one JSON file per dashboard under\n`config/reporting/dashboards/<id>.json`\n\n, but there's no hot-reload machinery behind it because\ndashboards are never read on a hot path — every request just reads the file straight off disk.\nSaving validates that every visualization a dashboard references actually exists.\n\nThe dashboard editor doubles as a preview: pick one or more suite-runs and every section renders live via the same visualization-execution mechanism as the Visualizations tab — no need to save first, since a dashboard is just a client-side composition of files and run data you already have locally while editing.\n\nThree general-purpose dashboards ship out of the box, so you don't have to build one before you can look at your first runs:\n\n**Agent Comparison**(`agent-comparison`\n\n) — cross-agent snapshot: the per-agent summary table, a cost-vs-accuracy scatter, and the full`agent-average-*`\n\nmetric grid. Use this to pick a winner among several agents/configs run against the same suite.**Trends Over Time**(`trends-over-time`\n\n) — single-agent/suite history: the per-run summary table and the full`runs-over-time-*`\n\nmetric grid. Use this to see whether a prompt or config change helped.**Reliability & Diagnostics**(`reliability-diagnostics`\n\n) — root-cause view: the question × agent pass-rate heatmap, error/iteration-limit rates, tool-call frequency, and full per-question detail. Use this to find out*why*a pass rate is what it is.\n\nA report is a saved `{ title, filter: { suiteIds, from?, to? }, dashboardId }`\n\n— the thing that\nactually answers \"how did this suite's runs look over the last two weeks?\" `Reporting > Reports`\n\nlists saved reports, each with two separate, deep-linkable screens:\n\n**Open**(`/reporting/reports/:id`\n\n) — the plain result. Resolves the report's filter against every currently-matching run and renders the dashboard against all of them. Read-only. Has an**Export** button (top-right) with two options, each starting a download immediately — no dialog, no backend round-trip, everything runs client-side against data already on the page:**To CSV**— one row per matching run, all 9 outcome parameters plus suite/agent/timestamp.** To PDF**— a rasterized, paginated snapshot of the report exactly as rendered (title, section headers, every chart/table), built with`html2canvas`\n\n+`jsPDF`\n\n.\n\n**Configure**(`/reporting/reports/:id/configure`\n\n, or`/reporting/reports/new`\n\nfor a blank one) — filter + dashboard picker with a live preview:- Pick one or more suites (or leave empty for \"any\") and an optional date range.\n**Find runs**— matches against the same persisted`SuiteRunRecord`\n\ns (`results/suite-runs/*.json`\n\n) the Visualizations/Dashboards previews use, all pre-selected; deselect any you don't want included.- Pick a dashboard. It renders immediately against the selected runs.\n**Save report** is optional, at any point — you don't need to save anything just to look at a result.\n\nSaving only persists the filter and dashboard choice, **not** which runs matched at save time —\nopening a saved report's Configure screen re-resolves the filter fresh (new runs may have\nappeared since), and Open always reflects current matches too, never a stale snapshot. One file\nper report under `config/reporting/reports/<id>.json`\n\n, same no-registry CRUD pattern as\ndashboards; the only thing validated at save time is that the chosen dashboard actually exists.\n\nJMeter's core idea is a feedback loop: define what you're testing, run it, read the numbers, adjust, run again. MAVERIK runs the same loop, just aimed at agent configurations instead of HTTP endpoints.\n\n| JMeter concept | MAVERIK equivalent |\n|---|---|\n| Test plan | Test suite (`config/maverik-suites/*.json` ) |\n| Sampler (one request) | Question (one prompt + criterion) |\n| Assertion | Criterion (`exact` / `contains` / `regex` / `llm-judge` ) |\n| Thread group / loop count | `repetitions` — run the same case N times to see through LLM nondeterminism |\n| Target under test | Agent configuration (`config/agents.json` ) |\n| Listener / results table | `GET /api/maverik/runs/{id}` + `results/{runId}/summary.csv` |\n\n**Tunable parameters** — the levers you pull between runs:\n\n`systemPrompt`\n\n— how the agent is instructed`model`\n\n— which LLM answers, and at what price`mcpServers`\n\n— which tools it's allowed to reach for`loopType`\n\n— how it drives the tool-call loop (sequential vs. parallel tool calls today; more strategies land as`ILoopStrategy`\n\nimplementations)`maxIterations`\n\n— how much rope it gets before MAVERIK calls it a failure- the question wording itself, if you're testing prompt phrasing rather than the agent\n- context-reduction strategies (summarizing/trimming history as it grows) — a natural future lever, not yet implemented\n\n**Outcome parameters** — what you judge a configuration on, captured per case:\n\n- correctness (\n`passed`\n\n, via the case's criterion) - speed (\n`durationMs`\n\n— the full turn, LLM and tool time both) - cost (\n`inputTokens`\n\n/`outputTokens`\n\n, and`estCostPerQuestion`\n\nwhen the model has pricing) - tool usage (\n`toolCallCount`\n\n/`toolNames`\n\n— some tools are far cheaper to call than others, so*which*tools an agent reaches for is itself a signal, not just how many)\n\nThe order matters. **Model the outcome parameters first**: write down what a correct answer\nlooks like (the criterion), what latency is acceptable, what token budget you're willing to\nspend, and which tools are \"free\" versus ones you want the agent to avoid unless necessary.\nOnly once that's pinned down do you sweep the tunable parameters — try a tighter system\nprompt, a cheaper model, a parallel-tools loop — and let MAVERIK tell you, in the same units\nyou defined up front, whether the change actually helped or just moved the cost around.\n\nEverything above — starting a run, reading results, editing agents and MCP servers, building\ndashboards and reports, even toggling wire-level logging — is reachable over the plain HTTP API\nin the [reference table](#-api-reference), plus a `config/`\n\n/`results/`\n\ndirectory that's just\nfiles (bind-mounted read-write). None of it requires the dashboard UI. That combination is what\nmakes MAVERIK straightforward to wire into an existing pipeline — TeamCity, Jenkins, GitHub\nActions, anything that can run a scheduled job and speak HTTP — rather than a separate tool\nsomeone has to babysit by hand.\n\nThe loop a CI job needs is the same one a person uses interactively, just scripted:\n\n`POST /api/maverik/runs`\n\nwith`{ suiteId, agentIds?, repetitions? }`\n\n→`{ runId }`\n\nimmediately (the run is queued, not executed synchronously in the request).- Poll\n`GET /api/maverik/runs/{runId}`\n\nuntil`state`\n\nis`completed`\n\nor`failed`\n\n. - Read\n`GET /api/maverik/runs/{runId}/summary`\n\nfor the per-agent aggregate — or, if the CI runner shares the results volume, read`results/{runId}/run.json`\n\n/`summary.csv`\n\nstraight off disk.\n\nEvery finished run also drops one `SuiteRunRecord`\n\nper agent under `results/suite-runs/*.json`\n\n—\nself-contained, timestamped, independently addressable — which is exactly what\n`GET /api/maverik/suite-runs?suiteIds=&from=&to=`\n\nlists back out. A nightly job doesn't need any\nextra bookkeeping to build up history: run the suite every night and the comparable-over-time\ndataset accumulates on its own, ready for the [Trends Over Time](#dashboards) dashboard to\nrender without further setup.\n\nA `ReportConfig`\n\n(`{ title, filter: { suiteIds, from?, to? }, dashboardId }`\n\n) is a saved,\nreusable *question* — \"how did suite X look over the last N days?\" — not a snapshot frozen at\ncreation time. Wire one up once (`POST /api/reporting/reports`\n\n, or drop a JSON file under\n`config/reporting/reports/`\n\ndirectly) pointing at a fixed suite and a rolling window, and it\nanswers freshly every time it's opened — including by tonight's job, tomorrow's, and a person\nchecking in from the dashboard a week later, all reading the same live-resolved answer.\n\nFor pulling result data into another system (a build dashboard, a chat digest, a gate that fails\nthe build on a pass-rate drop), consume `GET /api/maverik/suite-runs`\n\ndirectly rather than\nround-tripping through the rendered report — it's the same numbers the report's CSV export\nproduces, without needing a browser. **PDF export is the one piece that's client-rendered**\n(`html2canvas`\n\nrasterizes the page, `jsPDF`\n\npaginates it — see [Reports](#reports)); there's no\nserver-side \"give me a PDF\" endpoint. A pipeline that wants the actual PDF file (e.g. to attach\nto a release) needs a small headless-browser step — open `/reporting/reports/{id}`\n\n, click\nExport → To PDF, pick up the download — while a pipeline that just wants the numbers should read\nthe API/CSV path instead. PDF is built for a person to read, not for automation to parse.\n\n`POST /api/dev-mode`\n\nwith `{ enabled: true }`\n\nturns on full wire-level LLM request/response\nlogging before a run, and `{ enabled: false }`\n\nturns it back off — useful as an occasional\n\"investigate this regression\" pipeline stage without paying the logging cost (and disk usage) on\nevery ordinary nightly run. The toggle only affects calls made after it flips, so turn it on\n*before* starting the run you want captured, and remember a run already in flight when you flip\nit off ends up with a partial log (see [Debugging / Dev mode](#-debugging--dev-mode)).\n\nAn agent config only names which MCP servers it's allowed to use —\n`\"mcpServers\": [\"my-server\"]`\n\n— it never records that server's tools, descriptions, or schemas.\nThose are resolved live, at connect time, from whatever the server currently advertises. That has\na useful consequence: **a MAVERIK suite re-exercises the live state of your MCP server on every\nrun, not a frozen snapshot of it.** If you're developing an MCP server alongside the agents that\nuse it, the same nightly suite that catches an agent-prompt regression also catches an\nMCP-server regression — rename a tool, tighten a description, add a required parameter, and the\nvery next run's pass rate and tool-call counts move, with nothing in `agents.json`\n\nhaving\nchanged at all. That's a second, independent test surface a CI pipeline gets for free just by\npointing a suite at the tools in question — genuinely useful for catching the kind of change that\nlives entirely outside the agent spec and would otherwise only surface when a user hits it.\n\nOne operational caveat worth planning around: unlike agents/models/suites, `mcp-servers.json`\n\nis\n**not** hot-reloadable — `PUT /api/config/mcp-servers`\n\nalways returns `restartRequired: true`\n\n(see [Configuration](#-configuration)). A pipeline that also deploys a new version of the MCP\nserver under test needs to restart the MAVERIK container (`docker compose restart`\n\n) as its own\npipeline step, before the next suite run picks up the new tool catalog — otherwise the\nalready-connected session just keeps serving the old one.\n\nSee [ CI-CD Tutorial.md](/langens-jonathan/maverik/blob/master/CI-CD%20Tutorial.md) for a full worked-out example of how this could be\ndone — a Jenkins pipeline that restarts MAVERIK and re-runs a suite on every commit to an MCP server's\nown repo, then fails the build if pass rate drops or cost regresses against the previous run.\n\nNone of this is authenticated — MAVERIK is a self-hosted, single-user tool by design, and every endpoint above assumes whoever can reach it is trusted. That's fine for a CI runner on a private network talking to a container it manages itself; it is not something to expose on the open internet.\n\n**Parameterized agent sweeps**— generate a matrix of`AgentConfig`\n\ns from a base config plus a set of variations (e.g. 3 system prompts × 2 models = 6 agents) instead of hand-authoring every combination in`config/agents.json`\n\n— the direct analog of JMeter's CSV-driven parameterization.**Context-reduction strategies**— history-trimming/summarization as a loop-level lever (a new`ILoopStrategy`\n\nconcern), sitting alongside`manual`\n\nand`parallel-tools`\n\nas something you can A/B like any other tunable parameter.\n\n**Statistical rigor**— percentiles and std-dev over repetitions, flakiness detection.\n\n**More loop strategies**— SDK-driven function invocation, retry-on-tool-error, reflection loops.** Configurable concurrency**— JMeter-style parallel case execution for load testing.** Judge quality controls**— second-opinion judging, self-consistency checks.\n\nIssues and pull requests are welcome. See [ CONTRIBUTE.md](/langens-jonathan/maverik/blob/master/CONTRIBUTE.md) for the four kinds of\ncontribution (a new visualization, a new dashboard, backend changes, frontend changes), a worked\nexample of each, and what to check before opening a PR — there's no CI test suite, so \"actually\nrun it\" is the bar. The two invariants that matter most if you touch the backend:\n\n- The chat clients stay registered\n**without** automatic function invocation — the loop strategies own the tool loop, and that's what makes it measurable. - Benchmark runner and chat mode must keep sharing the same loop code.\n\nIt starts as an acronym: **M** CP **A** gent **V** alidator — *MAV* — which naturally wants to\nbe completed to *maverick*. But a maverick is an unorthodox character who refuses to conform\nto accepted standards, and this software is the opposite: it exists to hold agents *to* a\nstandard. So the *ck* had to go. **MAVERIK** — almost a maverick, but standards-compliant.\n\nApache 2.0 License.", "url": "https://wpnews.pro/news/show-hn-maverik-benchmark-compare-and-cost-predict-your-mcp-agents", "canonical_source": "https://github.com/langens-jonathan/maverik", "published_at": "2026-08-02 13:35:56+00:00", "updated_at": "2026-08-02 13:52:39.809548+00:00", "lang": "en", "topics": ["ai-tools", "ai-agents", "developer-tools"], "entities": ["Maverik", "Model Context Protocol", "JMeter", "Claude Haiku", "Sonnet"], "alternates": {"html": "https://wpnews.pro/news/show-hn-maverik-benchmark-compare-and-cost-predict-your-mcp-agents", "markdown": "https://wpnews.pro/news/show-hn-maverik-benchmark-compare-and-cost-predict-your-mcp-agents.md", "text": "https://wpnews.pro/news/show-hn-maverik-benchmark-compare-and-cost-predict-your-mcp-agents.txt", "jsonld": "https://wpnews.pro/news/show-hn-maverik-benchmark-compare-and-cost-predict-your-mcp-agents.jsonld"}}