Regression Tests for kagent Agents with agentevals

A developer built kagent-agentevals, an open-source tool that converts kagent session records into the trajectory format scored by LangChain's agentevals, plus a golden-suite runner that exits non-zero when an agent's behavior changes. The converter addresses a key mismatch: kagent stores agent activity as Google ADK events, where human-in-the-loop confirmation round-trips carry the agent's own function calls and results under a user content role, so naive role mapping misattributes tool calls to the person and corrupts trajectory comparisons. The tool also adds a summary view that explains why each event was dropped, since ADK serializes roughly forty null optional fields per event.

Originally published at webofmike.com https://webofmike.com/kagent-trajectory-evals/?utm source=devto&utm medium=syndication&utm campaign=kagent-trajectory-evals on 2026-09-25. The demo repo and every command in it were run before publishing. An AI agent that stops calling its tools does not throw an error. It answers from the model's memory, confidently and plausibly, and the output still looks fine in the UI. I wanted a regression test that catches that, so I wrote kagent-agentevals https://github.com/themsquared/kagent-agentevals : a converter from kagent https://kagent.dev/ session records into the trajectory format agentevals https://github.com/langchain-ai/agentevals scores, plus a golden-suite runner that exits non-zero when an agent's behavior changes. The conversion is where the work turned out to be. agentevals wants a flat list of OpenAI-format chat messages. kagent records what its agents do as Google ADK https://google.github.io/adk-docs/ events. Getting from one to the other is not a field rename, and three of the reasons only showed up when I ran it against real sessions instead of reading the schema. kagent writes one row per ADK event, with the event JSON in event.data . A single event carrying a tool call looks like this: {"author": "github assistant", "invocation id": "e-c51ee408-...", "partial": null, "content": {"role": "model", "parts": {"function call": {"id": "toolu 01...", "name": "search users", "args": {"query": "themsquared"}}} }} That is the readable version. ADK serializes every optional field, so the actual row has around forty keys set to null around the four that matter. Dumping a session's raw events to a terminal is unusable, which is why the first thing the tool grew was a view that accounts for each event and says why anything was dropped: kagent-evals extract <session-id --summary AUTHOR ROLE CONTRIBUTED WHY NOT 0 system no content 1 user user text "Who is themsquared?" 2 github assistant model call ask user 3 github assistant user filtered: adk request confirmation 4 user user filtered: adk request confirmation 5 github assistant user result ask user 6 github assistant model text "I don't currently have a tool to look … 7 github assistant no content 8 system no content 9 user user text "Try again" 10 github assistant model call search users 11 github assistant user result search users 12 github assistant model text "Here's what I found for themsquared … 13 github assistant no content 14 events, 8 contributed → 8 messages, tools: ask user, search users Fourteen rows, eight of which contribute anything. That is a real session: someone asked an agent about a GitHub user, the agent asked a clarifying question first, then retried and searched. Everything you would want to assert on is in there. So is a lot that you do not. Look at events 3, 5 and 11 above. Their content.role is user , and every one of them carries the agent's own function call or function result. That is not a bug in kagent, it is how the human-in-the-loop confirmation round-trip is modeled: the approval request and its answer flow back through the user side of the conversation. But if you map content.role onto the OpenAI role field, the agent's tool calls end up attributed to the person, and every comparison you write after that scores the wrong speaker. The trajectory looks plausible and is wrong. The fix is to stop reading the role at all and decide from the part type: php if isinstance response, dict : function response - tool message flush trajectory.append {"role": "tool", "content": ..., "name": name} elif isinstance call, dict : function call - assistant tool calls buffer.tool calls.append entry elif isinstance text, str and text.strip : if author == "user": only plain text consults the author trajectory.append {"role": "user", "content": text} else: buffer.text.append text A function call part is always an assistant action. A function response part is always a tool message. Only plain text needs to ask who was speaking, and for that author is reliable in a way content.role is not. Events 3 and 4 in that table are adk request confirmation . The ADK runtime injects that tool to run the approval handshake. The agent never chose to call it. Leave it in and it appears as a tool call in the trajectory, which means every reference trajectory you write has to include a piece of runtime plumbing that has nothing to do with the agent's behavior. Worse, it appears conditionally, only on turns where a confirmation happened, so references written against one session break against the next. So anything matching the adk prefix is dropped by default. The distinction is the prefix, not a judgment about the tool: ask user is a real kagent tool the agent genuinely decided to call, and it stays in. kagent's builtin tools return {"result": ...} . MCP tools return the richer MCP shape: {"content": {"type": "text", "text": "{\"total count\":1,...}"} , "isError": false} Hand either of those to an LLM-as-judge evaluator and the judge spends its attention on the envelope. Both get unwrapped, and the isError flag is preserved as a tool error prefix on the message content, so a failed tool call still reads as a failure rather than as data. This one bit me in a way worth mentioning, because it is the kind of bug that ships. My first unwrap only handled single-key dictionaries. The MCP envelope has two keys, content and isError , so it sailed straight through unwrapped and I only noticed because the tool message in the output still had isError in it. This is the part I would have gotten wrong from the documentation, and it changed how I wrote the suite. agentevals offers four trajectory match modes, and their names oversell what they inspect. I read the scorers to be sure: subset , superset and unordered extract tool calls from both trajectories and compare strict adds message count, roles, and per-message tool calls with argument matching. It still never compares assistant prose. So none of the match modes grade what the agent actually said. That is not a criticism, it is the right design for a deterministic check, but it means two things in practice. First, the content fields in a strict reference trajectory are documentation for whoever reads the suite next, not assertions. Second, if you need the wording graded, you need a trajectory LLM-as-judge https://github.com/langchain-ai/agentevals trajectory-llm-as-judge evaluator, which is a different tool with a different cost profile. Knowing that let me add a third evaluator type that asserts on tool names alone, with no reference trajectory to transcribe: evaluators: - type: tools used mode: superset expected: get-weather-by-city get-weather-by-city It builds a synthetic reference containing exactly those tool calls and delegates to agentevals. Because it only ever supplies names, it forces argument matching off. I learned that the hard way: I had set that default in the YAML parser rather than in the evaluator, so the two unit tests that constructed the spec directly in Python failed against a session they should have passed. The invariant belonged in the code, not in a config convention. Here is what makes it a test rather than a demo. The assertions never change. Only the session does. The suite says three things about a weather agent: the answer has to come from the weather tool, the agent stays inside its tool budget, and it looks up the city the user actually asked about. Against the real captured session: kagent-evals run demo/suite.yaml --fixture demo/fixtures/weather-grounded.jsonl answer-is-grounded-in-a-tool-call session fixture 4 messages, tools: get-weather-by-city get-weather-by-city PASS called-the-weather-tool trajectory superset match true stayed-within-its-tool-budget session fixture 4 messages, tools: get-weather-by-city get-weather-by-city PASS no-unexpected-tools trajectory subset match true looked-up-the-city-the-user-asked-about session fixture 4 messages, tools: get-weather-by-city get-weather-by-city PASS strict-with-args trajectory strict match true 3/3 cases passed Now the same suite against a session where the agent skipped the tool and answered from memory: answer-is-grounded-in-a-tool-call session fixture 2 messages, tools: none FAIL called-the-weather-tool trajectory superset match false stayed-within-its-tool-budget session fixture 2 messages, tools: none PASS no-unexpected-tools trajectory subset match true looked-up-the-city-the-user-asked-about session fixture 2 messages, tools: none FAIL strict-with-args trajectory strict match false 1/3 cases passed Caught, exit code 1, CI stops. The answer that agent produced was a fluent, specific, entirely invented weather report. Notice the middle case still passed on the ungrounded session. That is worth sitting with, because it is the most useful thing the demo taught me. A subset check asks whether the agent called anything outside the approved set. An agent that called nothing satisfies that trivially, since the empty set is a subset of everything. A tool allowlist cannot catch an agent that did no work. Catching that needs a superset check asserting the tool was called at least once. A suite that only allowlists is half a suite. The second one has the same shape. Point the suite at a session where the agent did call the weather tool, but for Paris when the user asked about London, and only the strict case fails. Both tool-name checks pass, because they compare names and ignore arguments. If the arguments carry the meaning, and for a lookup tool they usually do, you have to say so explicitly. Both of those are pinned by tests in the repo, so if the behavior ever changes the narration fails in CI rather than going stale on me in front of someone. The most expensive mistake I nearly made had nothing to do with agentevals. I started by reading a months-old checkout of kagent to work out the storage format, and that version serialized session events as A2A protocol.Message objects, with tool calls encoded as data parts carrying kagent type metadata. I had most of a converter written against that shape before I queried the running cluster and found it storing raw ADK events instead. Different envelope, different field names, and a much closer fit to what agentevals wants. Reading the source told me what some version once did. Only the database told me what the cluster in front of me was actually writing. For anything that consumes another system's persisted records, the stored bytes are the contract, and the fixture in the repo is a real captured session precisely so the converter stays pinned to data the runtime produces rather than to my reading of it. There is a related caveat I have written about before. The trajectory being scored is the runtime's own record of what the agent did, and an agent's self-reported record is not automatically trustworthy https://webofmike.com/agent-audit-log-integrity/ . These evals catch behavior drift in a record produced by the system under test. That is genuinely useful for regressions and it is not the same thing as a witness. No cluster and no API key. The sessions are JSONL fixtures in the repo, which is also what CI runs on every push. git clone https://github.com/themsquared/kagent-agentevals cd kagent-agentevals pip install -e . make demo make demo walks the whole story with pauses so you can talk over it. make demo-fast runs straight through. Against a live cluster it is the same suite with one flag changed: kagent-evals sessions --agent my-agent kagent-evals extract <session-id --summary kagent-evals extract <session-id --raw -o demo/fixtures/mine.jsonl kagent-evals run demo/suite.yaml --fixture demo/fixtures/mine.jsonl Capture a session once and it becomes a permanent regression test for that agent's behavior. One thing to do before you commit a capture: read it. A raw session record can carry request metadata from whoever was chatting, so the CLI prints a reminder on every --raw , and there is a test that fails if a fixture in the repo contains a JWT. The LLM-as-judge path is wired and unit-tested but I have not run it against a real model yet, so the repo says so rather than implying otherwise. Two evaluators there are worth having: reference-free grading of whether a trajectory is coherent and efficient, which the deterministic modes cannot express, and grading of the final answer's wording, which no match mode touches. The other direction is the boring, valuable one. A capture from every agent, a suite per agent, run nightly, with the tool budget asserted in both directions. That is a small amount of YAML and it is the difference between finding out an agent stopped grounding its answers from your eval suite versus finding out from a user. Code, the demo, and the talk track are at github.com/themsquared/kagent-agentevals https://github.com/themsquared/kagent-agentevals . How do I evaluate a kagent agent's tool calls with agentevals? Read the session's ADK events from kagent's event table, convert each event's content.parts into OpenAI-format chat messages, then pass that list to an agentevals evaluator as outputs. function call parts become an assistant message with tool calls, function response parts become tool messages. kagent-agentevals does the conversion and ships a suite runner that exits non-zero on failure, so it drops into CI. Why do my kagent tool calls show up with the wrong role? Because kagent sets content.role to user on some events that carry the agent's own function calls, notably during human-in-the-loop confirmation round-trips. If you map role directly onto the OpenAI role field, the agent's tool calls get attributed to the person. Decide the role from the part type instead: a function call part is always assistant, a function response part is always a tool message. What does trajectory match subset mode actually compare in agentevals? Only tool calls. The subset, superset, and unordered modes extract tool calls from both trajectories and compare names and arguments, ignoring roles, message order, and all text. Strict mode adds message count, roles, and per-message tool calls, but still never compares assistant prose. No match mode grades wording, so use a trajectory LLM-as-judge evaluator for that. Can a tool allowlist check catch an agent that stopped calling tools? No. A subset check asks whether the agent called anything outside the approved set, and an agent that called nothing satisfies that trivially, because the empty set is a subset of everything. It passes. Catching an agent that stopped doing work needs a superset check asserting the tool was called at least once, which is why a useful suite asserts in both directions. Canonical version, with machine-readable markdown at https://webofmike.com/kagent-trajectory-evals/index.md : https://webofmike.com/kagent-trajectory-evals/ https://webofmike.com/kagent-trajectory-evals/