Build an AI Agent Evaluation with JEV A developer walkthrough published in the VersusControl devops-ai-guidelines repo shows how to build an AI agent evaluation harness that separates what an incident agent did (tool calls) from what it said (its explanation), using plain Python for the work checks and the Jev structured evaluation model as the judge. In the worked example, an agent running on gpt-5.4-mini-2026-03-17 correctly traced a checkout p95 latency alert to a 14:02 deploy that cut DB_MAX_CONNECTIONS from 50 to 5, calling all four tools (get_metrics, get_deploys, get_db_status, get_logs) and rejecting the later payment_provider_latency signal, while a shorter system prompt produced an answer that read almost the same but cited a tool it never called and never saw the warning it claimed to rule out. The harness requires Python 3.10 or newer, an OpenAI API key, and a Cloudflare account for the judge step, with tool responses drawn from a recorded JSON file that also holds a hidden answer key the agent never receives. One run of my incident agent told me a checkout slowdown was caused by a config deploy that shrank the database pool from 50 connections to 5. It was right. The explanation was clear; it cited four tools, and it even ruled out a payment-provider warning that showed up later in the logs. Then I ran it with a shorter system prompt and got an answer that read almost the same. That one was wrong in two ways. It cited a tool it never called, and it never saw the warning it was supposed to rule out. If I had only read the paragraph, I would have shipped it. That is the problem with grading an agent by reading its answer. A good paragraph and a good investigation are two different things. This post builds a small harness that checks both. Plain Python checks the work. Jev, a fast structured evaluation model, judges the explanation. You can clone it and run it in a few minutes. I think the fastest way to understand an eval harness is to watch it grade something. So let’s start there. You need Python 3.10 or newer, an OpenAI API key, and for the judge step a Cloudflare account. The code is in the devops-ai-guidelines repo https://github.com/VersusControl/devops-ai-guidelines/tree/main/07-evaluating-ai-agents/code/chapter-08 : git clone https://github.com/VersusControl/devops-ai-guidelines.gitcd devops-ai-guidelines/07-evaluating-ai-agents/code/chapter-08python -m pip install -r requirements.txtcp .env.example .env Open .env and fill in OPENAI API KEY. Leave the Cloudflare fields empty for now. OPENAI MODEL defaults to gpt-4.1-mini; I used gpt-5.4-mini. Now run the agent once: python run agent.py Before you look at the output, here is what that command does. It takes about ten seconds, and nothing in it touches a real system. The answer key is in the same JSON file, but the agent never gets it. That matters later: it’s the only reason a grade means anything. Here is what one run printed for me on September 24, 2026: Scenario: checkout-latency-after-pool-changeAgent model: gpt-5.4-mini-2026-03-17Alert: checkout-service p95 latency 2sTool calls: get metrics, get deploys, get db status, get logsRoot cause: A deploy at 14:02 changed DB MAX CONNECTIONS 50 - 5, which immediatelyreduced database pool capacity and caused checkout requests to queue and time out.Category: deployCause change: DB MAX CONNECTIONS 50 - 5Cause effect: pool exhaustedCited evidence: get metrics, get logs, get deploys, get db statusRejected signals: payment provider latency, checkout image v1 9 2, general capacity limitSteps: 5 Your run will not match this word for word. The tool data is frozen, but the model can pick a different tool order or different wording every time. That’s normal, and it’s exactly why we need a grader instead of eyeballs. It’s worth slowing down here, because every line in that output is either something the agent did or something the agent said . The whole harness is built on keeping those two apart. Read the figure from top to bottom: So in this run, the agent did the work: it called all four tools, including get logs, so it really could have seen the payment line. The paragraph also sounds right. But "sounds right" is the part we can't check with code, and that is where Jev comes in. The example is an incident agent for a checkout service. The alert says p95 latency crossed two seconds. The real cause is a config deploy one minute earlier that cut DB MAX CONNECTIONS from 50 to 5. The pool fills up, requests wait for a connection, and checkout times out. The order on that timeline is the whole puzzle. The deploy comes before the alert, so it can be the cause. The payment-provider line comes after , so it can’t be, even though “payment provider slow” sounds like a checkout problem. A good agent notices the order. A lucky one just picks the scariest line. Every tool response comes from a recorded JSON file, not from production. The same file holds a hidden answer key: the true cause, the evidence that proves it, the planted distraction, and the step budget. The runner never gives the answer key to the agent. It loads it only after the agent has committed to an answer. Incidents are just my example. The pattern works for any agent that uses tools and then explains itself: a support agent, a SQL agent, a code-review agent. An agent eval has to answer two separate questions: The first question has exact answers, so code should answer it. I call these hard gates . The second question is about meaning, so it needs a model. The four hard gates live in grade.py https://github.com/VersusControl/devops-ai-guidelines/blob/main/07-evaluating-ai-agents/code/chapter-08/grade.py : Jev https://docs.typesafe.ai/introduction.md is a model from TypeSafe AI. TypeSafe calls it a System One model: instead of generating text, it answers typed questions about a piece of content you give it. You send a state the thing to judge and a map of questions. Each question is one of three types: Noul, Choice, Score. The answers come back as JSON your code can read directly. There is no paragraph to parse, and no chance of a missing field or a stray sentence before the JSON. That alone removes a whole class of bugs I’ve hit with LLM-as-judge setups. You can use any chat model as a judge. I’ve done it. Here is why I picked Jev for this one. It’s built for this shape of task. An eval judge doesn’t need to write anything. It needs to answer narrow questions like “does this explanation connect the change to the impact?” and give you a number. TypeSafe’s own docs list “score, judge, verify, guardrail” as core use cases. It’s cheap enough to run on every commit. TypeSafe’s published price for jev-1.13.0 is $0.042 per million $42 per billion input tokens, and output tokens are free. Their launch post puts typical chat-model input prices at $0.20 to $10 per million tokens, with output around five times that. To make that concrete, here is a small, made-up but realistic eval budget: 10 cases, 5 runs each, on 20 pull requests a day. That’s 1,000 judge calls a day. Say each call sends about 1,000 input tokens, and a chat judge writes about 100 tokens back. The bars use a log scale, because a normal one would make Jev’s bar invisible. The chat numbers are ranges from TypeSafe’s post, not quotes for a specific model, so plug in your own. The point doesn’t change much: a judge that runs on every case, several times each, on every pull request adds up fast. With Jev it mostly doesn’t. Cloudflare lists its own price for Jev in the dashboard, and it may not match TypeSafe’s direct price. Check before a large run. It’s fast. TypeSafe reports 70 to 500 ms end to end, because it produces all answers in one parallel pass instead of token by token. Their headline “193.6x faster, 444.6x cheaper” comes from their own workflow evals, and they say themselves that it’s on the high end of real-world gains. I’d plan for less, but even a tenth of that is a big deal inside CI. Confidence comes with every answer. Jev is trained to return calibrated probabilities. That gives you a natural review needed lane: when the judge isn't sure, a person looks, instead of the pipeline guessing. It isn’t magic, and TypeSafe is honest about that in its Jev 1.13 jaggedness page https://docs.typesafe.ai/model-jaggedness/jev-1.13.md . It reads instructions literally. It’s weak at counting, arithmetic, and comparing dates. Accuracy drops when the state is full of irrelevant detail. And it doesn’t treat the state as hostile by default, so text inside it can try to steer the answer. That list turned into my design rules. Anything numeric or time-based stays in Python: step counts, “did the deploy happen before the alert”, “was the payment line after 14:03”. Jev only gets the part code can’t do, which is reading the paragraph. Workers AI https://developers.cloudflare.com/workers-ai/ is Cloudflare’s serverless inference service. You call models on Cloudflare’s GPUs with one account ID and one API token, over plain HTTPS. You don’t deploy a Worker for this; the REST API is enough. Jev is listed in the Workers AI catalog as a third-party model. I used Cloudflare for one practical reason: we already had credit on our Cloudflare account, and it kept every model call on one bill. There is a free daily allocation 10,000 Neurons a day at the time of writing , but some models, including third-party ones, need a paid plan or prepaid AI Gateway credits. You don’t have to use Cloudflare. TypeSafe offers Jev directly at https://api.typesafe.ai/v1/systemone with its own API key, plus Python and JavaScript SDKs https://docs.typesafe.ai/sdk.md . The request is the same idea: a state and a map of questions. Pick whichever fits your billing. The grader only cares that it sends a state and questions and gets typed answers back. Before any code, here’s the map. The code folder https://github.com/VersusControl/devops-ai-guidelines/tree/main/07-evaluating-ai-agents/code/chapter-08 has a handful of small files, and each one does one job: Two objects travel through these files. If you understand them, the rest of the code is easy to follow. The Conclusion is what the agent hands back. Some fields are written by the model, and some by the runner. For the run above, it looks roughly like this: Conclusion written by the model root cause="A deploy at 14:02 changed DB MAX CONNECTIONS 50 - 5, which ...", category="deploy", cause change="DB MAX CONNECTIONS 50 - 5", cause effect="pool exhausted", evidence= "get metrics", "get logs", "get deploys", "get db status" , rejected signals= "payment provider latency", "checkout image v1 9 2", ... , written by the runner, which the model can't edit trajectory= {"type": "call tool", "tool": "get metrics"}, ..., {"type": "conclude"} , observations={"get db status": {"pool size": 5, "in use": 5, "waiting": 40}, ...}, trajectory is the list of actions in order. observations is what each tool actually returned. These are the same two groups from the output figure earlier: what the agent did and what it said . The answer key is the truth, stored in the same JSON file but never given to the agent: "answer key": { "true category": "deploy", "true cause": "The 14:02 deploy reduced DB MAX CONNECTIONS from 50 to 5, exhausting the connection pool.", "cause change": "DB MAX CONNECTIONS 50 - 5", "cause effect": "pool exhausted", "required evidence": "get deploys", "get db status" , "distraction id": "payment provider", "max steps": 5} I trimmed a few fields that describe where the distraction appears. Now the whole judged run fits in a few lines. This is the core of run judge.py https://github.com/VersusControl/devops-ai-guidelines/blob/main/07-evaluating-ai-agents/code/chapter-08/run judge.py , simplified: agent input, conclusion = run agent settings, case path=case the agent investigates and commitsanswer = load answer key case only now load the truthresult = evaluate conclusion, answer, send to jev, DEMO POLICY print result "status" pass, fail, review needed, or incomplete And evaluate, in jev judge.py https://github.com/VersusControl/devops-ai-guidelines/blob/main/07-evaluating-ai-agents/code/chapter-08/jev judge.py , does three things: hard = grade conclusion, answer 1. the four hard gates, in plain Pythonpayload = build request conclusion, answer, ... 2. a short state plus two questions for Jevresponse = send payload 3. HTTPS to Cloudflare, typed answers back then check the reply and combine hard gates + Jev into one status The four steps below walk through that flow: what goes into Jev, what we ask it, how the call works, and how to run the full grade. Jev can’t read a Python object, and it shouldn’t see everything anyway. So build request takes the Conclusion and the answer key and makes one small dictionary, called the state . It's the only thing Jev reads about this run: state = { "case id": answer.scenario id, "expected cause": answer.true cause, from the hidden answer key "agent explanation": conclusion.root cause, the paragraph the agent wrote "alert started at": conclusion.alert "started at" , "observed tool responses": observed, short facts, only from tools that ran} Think of it as a note to a reviewer: “Here’s what really happened, here’s what the agent wrote, and here’s what its tools showed. Does the paragraph hold up?” Two details matter here. expected cause is fine to send to the judge , because the agent has already answered and can't change its answer. And observed is a set of short facts computed from conclusion.observations, like {"pool size": 5, "in use": 5, "waiting": 40}, not raw log lines. Only tools the agent really called show up there. That keeps the state small Jev likes that and limits what leaves your machine. Before anything goes out, an approval step checks the state against the recorded case and rejects explanations that look like they contain a token, URL, or email address. That’s a guard for these synthetic cases, not a real redaction policy. For production incidents, you’d need a reviewed allowlist. The rubric is two questions in one request: QUESTIONS = { "explanation quality": { "type": "score", "instructions": "How well does the agent explanation connect the expected change to the observed checkout impact?", "criteria": "The explanation contradicts the observations or gives a wrong mechanism", "The explanation names the change but leaves out how it caused checkout impact", "The explanation connects the observed change, its supported mechanism, and checkout timeouts", , }, "unsupported claim": { "type": "noul", "instructions": "Does the agent explanation make a material claim unsupported by the observed tool responses?", "criteria": { "true": "A material claim has no support in the returned tool responses", "false": "The material claims follow from the returned tool responses", }, },} The Score places the explanation on three described levels, 0 to 2. It comes back as a probability for each level, and the score is the weighted average. For example, probabilities of 0, 0.25, and 0.75 give a score of 0 0 +1 0.25 +2 0.75 =1.750 0 +1 0.25 +2 0.75 =1.75. The Noul asks one specific thing: did the agent claim something no tool showed? Watch the direction. A high value is bad here, because “true” means unsupported. I kept these as two questions on purpose. An explanation can describe the right mechanism and still invent a rollback that never happened. One blended “quality” number would hide that. Add three values to .env: CLOUDFLARE ACCOUNT ID, CLOUDFLARE API TOKEN create it under Workers AI, "Use REST API" , and JEV MODEL. The documented request looks like this: url = f"https://api.cloudflare.com/client/v4/accounts/{account id}/ai/run"body = {"model": "typesafe/jev", "input": {"state": state, "questions": QUESTIONS}} POST it with "Authorization: Bearer