{"slug": "regression-tests-for-kagent-agents-with-agentevals", "title": "Regression Tests for kagent Agents with agentevals", "summary": "A developer built kagent-agentevals, an open-source tool that converts kagent session records into the trajectory format scored by LangChain's agentevals, plus a golden-suite runner that exits non-zero when an agent's behavior changes. The converter addresses a key mismatch: kagent stores agent activity as Google ADK events, where human-in-the-loop confirmation round-trips carry the agent's own function calls and results under a user content role, so naive role mapping misattributes tool calls to the person and corrupts trajectory comparisons. The tool also adds a summary view that explains why each event was dropped, since ADK serializes roughly forty null optional fields per event.", "body_md": "*Originally published at [webofmike.com](https://webofmike.com/kagent-trajectory-evals/?utm_source=devto&utm_medium=syndication&utm_campaign=kagent-trajectory-evals) on 2026-09-25. The demo repo and every command in it were run before publishing.*\n\nAn AI agent that stops calling its tools does not throw an error. It answers from the model's memory, confidently and plausibly, and the output still looks fine in the UI. I wanted a regression test that catches that, so I wrote [kagent-agentevals](https://github.com/themsquared/kagent-agentevals): a converter from [kagent](https://kagent.dev/) session records into the trajectory format [agentevals](https://github.com/langchain-ai/agentevals) scores, plus a golden-suite runner that exits non-zero when an agent's behavior changes.\n\nThe conversion is where the work turned out to be. agentevals wants a flat list of OpenAI-format chat messages. kagent records what its agents do as [Google ADK](https://google.github.io/adk-docs/) events. Getting from one to the other is not a field rename, and three of the reasons only showed up when I ran it against real sessions instead of reading the schema.\n\nkagent writes one row per ADK event, with the event JSON in `event.data`. A single event carrying a tool call looks like this:\n\n```\n{\"author\": \"github_assistant\",\n \"invocation_id\": \"e-c51ee408-...\",\n \"partial\": null,\n \"content\": {\"role\": \"model\",\n             \"parts\": [{\"function_call\": {\"id\": \"toolu_01...\",\n                                          \"name\": \"search_users\",\n                                          \"args\": {\"query\": \"themsquared\"}}}]}}\n```\n\nThat is the readable version. ADK serializes every optional field, so the actual row has around forty keys set to `null` around the four that matter. Dumping a session's raw events to a terminal is unusable, which is why the first thing the tool grew was a view that accounts for each event and says why anything was dropped:\n\n```\nkagent-evals extract <session-id> --summary\n#  AUTHOR               ROLE   CONTRIBUTED                                    WHY NOT\n  0  system                                                                     no content\n  1  user                 user   text \"Who is themsquared?\"\n  2  github_assistant     model  call ask_user\n  3  github_assistant     user                                                  filtered: adk_request_confirmation\n  4  user                 user                                                  filtered: adk_request_confirmation\n  5  github_assistant     user   result ask_user\n  6  github_assistant     model  text \"I don't currently have a tool to look …\n  7  github_assistant                                                           no content\n  8  system                                                                     no content\n  9  user                 user   text \"Try again\"\n 10  github_assistant     model  call search_users\n 11  github_assistant     user   result search_users\n 12  github_assistant     model  text \"Here's what I found for **themsquared*…\n 13  github_assistant                                                           no content\n\n14 events, 8 contributed → 8 messages, tools: ask_user, search_users\n```\n\nFourteen rows, eight of which contribute anything. That is a real session: someone asked an agent about a GitHub user, the agent asked a clarifying question first, then retried and searched. Everything you would want to assert on is in there. So is a lot that you do not.\n\nLook at events 3, 5 and 11 above. Their `content.role` is `user`, and every one of them carries the agent's own function call or function result.\n\nThat is not a bug in kagent, it is how the human-in-the-loop confirmation round-trip is modeled: the approval request and its answer flow back through the user side of the conversation. But if you map `content.role` onto the OpenAI `role` field, the agent's tool calls end up attributed to the person, and every comparison you write after that scores the wrong speaker. The trajectory looks plausible and is wrong.\n\nThe fix is to stop reading the role at all and decide from the part type:\n\n``` php\nif isinstance(response, dict):          # function_response -> tool message\n    flush()\n    trajectory.append({\"role\": \"tool\", \"content\": ..., \"name\": name})\nelif isinstance(call, dict):            # function_call -> assistant tool_calls\n    buffer.tool_calls.append(entry)\nelif isinstance(text, str) and text.strip():\n    if author == \"user\":                # only plain text consults the author\n        trajectory.append({\"role\": \"user\", \"content\": text})\n    else:\n        buffer.text.append(text)\n```\n\nA `function_call` part is always an assistant action. A `function_response` part is always a tool message. Only plain text needs to ask who was speaking, and for that `author` is reliable in a way `content.role` is not.\n\nEvents 3 and 4 in that table are `adk_request_confirmation`. The ADK runtime injects that tool to run the approval handshake. The agent never chose to call it.\n\nLeave it in and it appears as a tool call in the trajectory, which means every reference trajectory you write has to include a piece of runtime plumbing that has nothing to do with the agent's behavior. Worse, it appears conditionally, only on turns where a confirmation happened, so references written against one session break against the next.\n\nSo anything matching the `adk_` prefix is dropped by default. The distinction is the prefix, not a judgment about the tool: `ask_user` is a real kagent tool the agent genuinely decided to call, and it stays in.\n\nkagent's builtin tools return `{\"result\": ...}`. MCP tools return the richer MCP shape:\n\n```\n{\"content\": [{\"type\": \"text\", \"text\": \"{\\\"total_count\\\":1,...}\"}], \"isError\": false}\n```\n\nHand either of those to an LLM-as-judge evaluator and the judge spends its attention on the envelope. Both get unwrapped, and the `isError` flag is preserved as a `[tool error]` prefix on the message content, so a failed tool call still reads as a failure rather than as data.\n\nThis one bit me in a way worth mentioning, because it is the kind of bug that ships. My first unwrap only handled single-key dictionaries. The MCP envelope has two keys, `content` and `isError`, so it sailed straight through unwrapped and I only noticed because the tool message in the output still had `isError` in it.\n\nThis is the part I would have gotten wrong from the documentation, and it changed how I wrote the suite. agentevals offers four trajectory match modes, and their names oversell what they inspect. I read the scorers to be sure:\n\n`subset`, `superset` and `unordered` extract tool calls from both trajectories and compare `strict` adds message count, roles, and per-message tool calls with argument matching. It still never compares assistant prose.\nSo none of the match modes grade what the agent actually said. That is not a criticism, it is the right design for a deterministic check, but it means two things in practice. First, the `content` fields in a `strict` reference trajectory are documentation for whoever reads the suite next, not assertions. Second, if you need the wording graded, you need a [trajectory LLM-as-judge](https://github.com/langchain-ai/agentevals#trajectory-llm-as-judge) evaluator, which is a different tool with a different cost profile.\n\nKnowing that let me add a third evaluator type that asserts on tool names alone, with no reference trajectory to transcribe:\n\n```\nevaluators:\n  - type: tools_used\n    mode: superset\n    expected: [get-weather-by-city_get-weather-by-city]\n```\n\nIt builds a synthetic reference containing exactly those tool calls and delegates to agentevals. Because it only ever supplies names, it forces argument matching off. I learned that the hard way: I had set that default in the YAML parser rather than in the evaluator, so the two unit tests that constructed the spec directly in Python failed against a session they should have passed. The invariant belonged in the code, not in a config convention.\n\nHere is what makes it a test rather than a demo. The assertions never change. Only the session does.\n\nThe suite says three things about a weather agent: the answer has to come from the weather tool, the agent stays inside its tool budget, and it looks up the city the user actually asked about. Against the real captured session:\n\n```\nkagent-evals run demo/suite.yaml --fixture demo/fixtures/weather-grounded.jsonl\nanswer-is-grounded-in-a-tool-call  [session fixture]\n  4 messages, tools: get-weather-by-city_get-weather-by-city\n  PASS  called-the-weather-tool  trajectory_superset_match    true\n\nstayed-within-its-tool-budget  [session fixture]\n  4 messages, tools: get-weather-by-city_get-weather-by-city\n  PASS  no-unexpected-tools      trajectory_subset_match      true\n\nlooked-up-the-city-the-user-asked-about  [session fixture]\n  4 messages, tools: get-weather-by-city_get-weather-by-city\n  PASS  strict-with-args         trajectory_strict_match      true\n\n3/3 cases passed\n```\n\nNow the same suite against a session where the agent skipped the tool and answered from memory:\n\n```\nanswer-is-grounded-in-a-tool-call  [session fixture]\n  2 messages, tools: (none)\n  FAIL  called-the-weather-tool  trajectory_superset_match    false\n\nstayed-within-its-tool-budget  [session fixture]\n  2 messages, tools: (none)\n  PASS  no-unexpected-tools      trajectory_subset_match      true\n\nlooked-up-the-city-the-user-asked-about  [session fixture]\n  2 messages, tools: (none)\n  FAIL  strict-with-args         trajectory_strict_match      false\n\n1/3 cases passed\n```\n\nCaught, exit code 1, CI stops. The answer that agent produced was a fluent, specific, entirely invented weather report.\n\nNotice the middle case still passed on the ungrounded session. That is worth sitting with, because it is the most useful thing the demo taught me.\n\nA `subset` check asks whether the agent called anything outside the approved set. An agent that called nothing satisfies that trivially, since the empty set is a subset of everything. **A tool allowlist cannot catch an agent that did no work.** Catching that needs a `superset` check asserting the tool was called at least once. A suite that only allowlists is half a suite.\n\nThe second one has the same shape. Point the suite at a session where the agent did call the weather tool, but for Paris when the user asked about London, and only the `strict` case fails. Both tool-name checks pass, because they compare names and ignore arguments. If the arguments carry the meaning, and for a lookup tool they usually do, you have to say so explicitly.\n\nBoth of those are pinned by tests in the repo, so if the behavior ever changes the narration fails in CI rather than going stale on me in front of someone.\n\nThe most expensive mistake I nearly made had nothing to do with agentevals.\n\nI started by reading a months-old checkout of kagent to work out the storage format, and that version serialized session events as A2A `protocol.Message` objects, with tool calls encoded as data parts carrying `kagent_type` metadata. I had most of a converter written against that shape before I queried the running cluster and found it storing raw ADK events instead. Different envelope, different field names, and a much closer fit to what agentevals wants.\n\nReading the source told me what some version once did. Only the database told me what the cluster in front of me was actually writing. For anything that consumes another system's persisted records, the stored bytes are the contract, and the fixture in the repo is a real captured session precisely so the converter stays pinned to data the runtime produces rather than to my reading of it.\n\nThere is a related caveat I have written about before. The trajectory being scored is the runtime's own record of what the agent did, and [an agent's self-reported record is not automatically trustworthy](https://webofmike.com/agent-audit-log-integrity/). These evals catch behavior drift in a record produced by the system under test. That is genuinely useful for regressions and it is not the same thing as a witness.\n\nNo cluster and no API key. The sessions are JSONL fixtures in the repo, which is also what CI runs on every push.\n\n```\ngit clone https://github.com/themsquared/kagent-agentevals\ncd kagent-agentevals\npip install -e .\nmake demo\n```\n\n`make demo` walks the whole story with pauses so you can talk over it. `make demo-fast` runs straight through.\n\nAgainst a live cluster it is the same suite with one flag changed:\n\n```\nkagent-evals sessions --agent my-agent\nkagent-evals extract <session-id> --summary\nkagent-evals extract <session-id> --raw -o demo/fixtures/mine.jsonl\nkagent-evals run demo/suite.yaml --fixture demo/fixtures/mine.jsonl\n```\n\nCapture a session once and it becomes a permanent regression test for that agent's behavior. One thing to do before you commit a capture: read it. A raw session record can carry request metadata from whoever was chatting, so the CLI prints a reminder on every `--raw`, and there is a test that fails if a fixture in the repo contains a JWT.\n\nThe LLM-as-judge path is wired and unit-tested but I have not run it against a real model yet, so the repo says so rather than implying otherwise. Two evaluators there are worth having: reference-free grading of whether a trajectory is coherent and efficient, which the deterministic modes cannot express, and grading of the final answer's wording, which no match mode touches.\n\nThe other direction is the boring, valuable one. A capture from every agent, a suite per agent, run nightly, with the tool budget asserted in both directions. That is a small amount of YAML and it is the difference between finding out an agent stopped grounding its answers from your eval suite versus finding out from a user.\n\nCode, the demo, and the talk track are at [github.com/themsquared/kagent-agentevals](https://github.com/themsquared/kagent-agentevals).\n\n**How do I evaluate a kagent agent's tool calls with agentevals?**\n\nRead the session's ADK events from kagent's event table, convert each event's content.parts into OpenAI-format chat messages, then pass that list to an agentevals evaluator as outputs. function_call parts become an assistant message with tool_calls, function_response parts become tool messages. kagent-agentevals does the conversion and ships a suite runner that exits non-zero on failure, so it drops into CI.\n\n**Why do my kagent tool calls show up with the wrong role?**\n\nBecause kagent sets content.role to user on some events that carry the agent's own function calls, notably during human-in-the-loop confirmation round-trips. If you map role directly onto the OpenAI role field, the agent's tool calls get attributed to the person. Decide the role from the part type instead: a function_call part is always assistant, a function_response part is always a tool message.\n\n**What does trajectory_match subset mode actually compare in agentevals?**\n\nOnly tool calls. The subset, superset, and unordered modes extract tool calls from both trajectories and compare names and arguments, ignoring roles, message order, and all text. Strict mode adds message count, roles, and per-message tool calls, but still never compares assistant prose. No match mode grades wording, so use a trajectory LLM-as-judge evaluator for that.\n\n**Can a tool allowlist check catch an agent that stopped calling tools?**\n\nNo. A subset check asks whether the agent called anything outside the approved set, and an agent that called nothing satisfies that trivially, because the empty set is a subset of everything. It passes. Catching an agent that stopped doing work needs a superset check asserting the tool was called at least once, which is why a useful suite asserts in both directions.\n\n*Canonical version, with machine-readable markdown at `https://webofmike.com/kagent-trajectory-evals/index.md`: [https://webofmike.com/kagent-trajectory-evals/](https://webofmike.com/kagent-trajectory-evals/)*", "url": "https://wpnews.pro/news/regression-tests-for-kagent-agents-with-agentevals", "canonical_source": "https://dev.to/webofmike/regression-tests-for-kagent-agents-with-agentevals-28op", "published_at": "2026-09-25 16:10:54+00:00", "updated_at": "2026-09-25 16:31:01.950132+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "mlops"], "entities": ["kagent", "agentevals", "kagent-agentevals", "Google ADK", "LangChain", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/regression-tests-for-kagent-agents-with-agentevals", "markdown": "https://wpnews.pro/news/regression-tests-for-kagent-agents-with-agentevals.md", "text": "https://wpnews.pro/news/regression-tests-for-kagent-agents-with-agentevals.txt", "jsonld": "https://wpnews.pro/news/regression-tests-for-kagent-agents-with-agentevals.jsonld"}}