Part 2 of 3 — building and testing MCP agents
Every AI agent is a bundle of decisions, most of which get made once, informally, and never revisited: which model, what system prompt, which tools it's allowed to touch, how many steps it gets before you give up on it. Nobody writes these down as a config. They just happen, in code, and then everyone forgets they were choices at all.
In Part 1 I argued those decisions have a shape: an agent definition (the class — loop, system context, capability boundary, model, limits) and an agent instance (a running occurrence — goal, context window, credentials, consumed budget). That split turns out to be exactly what you need for tuning, because the two kinds of parameters people mix up when "tuning an agent" live on opposite sides of it: you tune the definition, and you measure the instances.
These are the levers — change any one and you have a different agent, even if everything else stays the same:
That's the agent definition from Part 1, field for field — with one deliberate absence. The task contract is the only definition field that isn't a lever. You don't tune it; you tune everything else against it. Hold that thought.
The consequence of seeing these as fields is simple but real: if you don't track which field you changed, you can't attribute why the results changed.
This is the half people skip. An agent isn't "good" in the abstract — it's good or bad against criteria you define yourself, and every one of them is a measurement taken from agent instances after they run:
Here's the part that's easy to get backwards: you have to define the outcome parameters before you start sweeping the tunable ones. Decide what a correct answer looks like, what latency is acceptable, what token budget you're willing to spend, and which tools should be used sparingly — write it down as a criterion, a rubric, a budget. Only then start trying a tighter system context, a cheaper model, a different loop strategy.
Skip that step and "tuning" becomes vibes — you nudge a prompt, the answer looks a little better in the three examples you tried, you ship it, and you have no idea whether it's actually better or just different. Do it in the right order and every change becomes a testable hypothesis: does this specific change to one definition field move the numbers I said mattered, in the direction I wanted, without moving the ones I didn't touch?
The reason this framing pays off in practice is that once the system context, model, capability boundary, and loop settings are configuration rather than something baked into code — an agents.json
, in my host — comparing two agents stops being a rewrite and becomes a diff. Point the same set of test questions at two definitions that differ in exactly one field, run a batch of instances against each, and any difference in the measured outcomes is attributable to that one field.
That's the whole game: tune the class, measure the instances. And it's exactly the setup you need for what comes next — running these comparisons systematically instead of by hand.
The link to part 1 of this series: https://dev.to/langensjonathan/mcp-agents-explained-what-actually-makes-an-llm-an-agent-12a0