# Build an AI Agent Evaluation with JEV > Source: > Published: 2026-09-25 05:27:14+00:00 One run of my incident agent told me a checkout slowdown was caused by a config deploy that shrank the database pool from 50 connections to 5. It was right. The explanation was clear; it cited four tools, and it even ruled out a payment-provider warning that showed up later in the logs. Then I ran it with a shorter system prompt and got an answer that read almost the same. That one was wrong in two ways. It cited a tool it never called, and it never saw the warning it was supposed to rule out. If I had only read the paragraph, I would have shipped it. That is the problem with grading an agent by reading its answer. A good paragraph and a good investigation are two different things. This post builds a small harness that checks both. Plain Python checks the work. Jev, a fast structured evaluation model, judges the explanation. You can clone it and run it in a few minutes. I think the fastest way to understand an eval harness is to watch it grade something. So let’s start there. You need Python 3.10 or newer, an OpenAI API key, and (for the judge step) a Cloudflare account. The code is in the [devops-ai-guidelines repo](https://github.com/VersusControl/devops-ai-guidelines/tree/main/07-evaluating-ai-agents/code/chapter-08): ``` git clone https://github.com/VersusControl/devops-ai-guidelines.gitcd devops-ai-guidelines/07-evaluating-ai-agents/code/chapter-08python -m pip install -r requirements.txtcp .env.example .env ``` Open .env and fill in OPENAI_API_KEY. Leave the Cloudflare fields empty for now. OPENAI_MODEL defaults to gpt-4.1-mini; I used gpt-5.4-mini. Now run the agent once: ``` python run_agent.py ``` Before you look at the output, here is what that command does. It takes about ten seconds, and nothing in it touches a real system. The answer key is in the same JSON file, but the agent never gets it. That matters later: it’s the only reason a grade means anything. Here is what one run printed for me on September 24, 2026: ``` Scenario: checkout-latency-after-pool-changeAgent model: gpt-5.4-mini-2026-03-17Alert: checkout-service p95 latency > 2sTool calls: get_metrics, get_deploys, get_db_status, get_logsRoot cause: A deploy at 14:02 changed DB_MAX_CONNECTIONS 50 -> 5, which immediatelyreduced database pool capacity and caused checkout requests to queue and time out.Category: deployCause change: DB_MAX_CONNECTIONS 50 -> 5Cause effect: pool_exhaustedCited evidence: get_metrics, get_logs, get_deploys, get_db_statusRejected signals: payment_provider_latency, checkout_image_v1_9_2, general_capacity_limitSteps: 5 ``` Your run will not match this word for word. The tool data is frozen, but the model can pick a different tool order or different wording every time. That’s normal, and it’s exactly why we need a grader instead of eyeballs. It’s worth slowing down here, because every line in that output is either something the agent *did* or something the agent *said*. The whole harness is built on keeping those two apart. Read the figure from top to bottom: So in this run, the agent did the work: it called all four tools, including get_logs, so it really could have seen the payment line. The paragraph also sounds right. But "sounds right" is the part we can't check with code, and that is where Jev comes in. The example is an incident agent for a checkout service. The alert says p95 latency crossed two seconds. The real cause is a config deploy one minute earlier that cut DB_MAX_CONNECTIONS from 50 to 5. The pool fills up, requests wait for a connection, and checkout times out. The order on that timeline is the whole puzzle. The deploy comes *before* the alert, so it can be the cause. The payment-provider line comes *after*, so it can’t be, even though “payment provider slow” sounds like a checkout problem. A good agent notices the order. A lucky one just picks the scariest line. Every tool response comes from a recorded JSON file, not from production. The same file holds a hidden answer key: the true cause, the evidence that proves it, the planted distraction, and the step budget. The runner never gives the answer key to the agent. It loads it only after the agent has committed to an answer. Incidents are just my example. The pattern works for any agent that uses tools and then explains itself: a support agent, a SQL agent, a code-review agent. An agent eval has to answer two separate questions: The first question has exact answers, so code should answer it. I call these *hard gates*. The second question is about meaning, so it needs a model. The four hard gates live in [grade.py](https://github.com/VersusControl/devops-ai-guidelines/blob/main/07-evaluating-ai-agents/code/chapter-08/grade.py): [Jev](https://docs.typesafe.ai/introduction.md) is a model from TypeSafe AI. TypeSafe calls it a *System One* model: instead of generating text, it answers typed questions about a piece of content you give it. You send a state (the thing to judge) and a map of questions. Each question is one of three types: Noul, Choice, Score. The answers come back as JSON your code can read directly. There is no paragraph to parse, and no chance of a missing field or a stray sentence before the JSON. That alone removes a whole class of bugs I’ve hit with LLM-as-judge setups. You can use any chat model as a judge. I’ve done it. Here is why I picked Jev for this one. **It’s built for this shape of task.** An eval judge doesn’t need to write anything. It needs to answer narrow questions like “does this explanation connect the change to the impact?” and give you a number. TypeSafe’s own docs list “score, judge, verify, guardrail” as core use cases. **It’s cheap enough to run on every commit.** TypeSafe’s published price for jev-1.13.0 is $0.042 per million ($42 per *billion*) input tokens, and output tokens are free. Their launch post puts typical chat-model input prices at $0.20 to $10 per *million* tokens, with output around five times that. To make that concrete, here is a small, made-up but realistic eval budget: 10 cases, 5 runs each, on 20 pull requests a day. That’s 1,000 judge calls a day. Say each call sends about 1,000 input tokens, and a chat judge writes about 100 tokens back. The bars use a log scale, because a normal one would make Jev’s bar invisible. The chat numbers are ranges from TypeSafe’s post, not quotes for a specific model, so plug in your own. The point doesn’t change much: a judge that runs on every case, several times each, on every pull request adds up fast. With Jev it mostly doesn’t. (Cloudflare lists its own price for Jev in the dashboard, and it may not match TypeSafe’s direct price. Check before a large run.) **It’s fast.** TypeSafe reports 70 to 500 ms end to end, because it produces all answers in one parallel pass instead of token by token. Their headline “193.6x faster, 444.6x cheaper” comes from their own workflow evals, and they say themselves that it’s on the high end of real-world gains. I’d plan for less, but even a tenth of that is a big deal inside CI. **Confidence comes with every answer.** Jev is trained to return calibrated probabilities. That gives you a natural review_needed lane: when the judge isn't sure, a person looks, instead of the pipeline guessing. It isn’t magic, and TypeSafe is honest about that in its [Jev 1.13 jaggedness page](https://docs.typesafe.ai/model-jaggedness/jev-1.13.md). It reads instructions literally. It’s weak at counting, arithmetic, and comparing dates. Accuracy drops when the state is full of irrelevant detail. And it doesn’t treat the state as hostile by default, so text inside it can try to steer the answer. That list turned into my design rules. Anything numeric or time-based stays in Python: step counts, “did the deploy happen before the alert”, “was the payment line after 14:03”. Jev only gets the part code can’t do, which is reading the paragraph. [Workers AI](https://developers.cloudflare.com/workers-ai/) is Cloudflare’s serverless inference service. You call models on Cloudflare’s GPUs with one account ID and one API token, over plain HTTPS. You don’t deploy a Worker for this; the REST API is enough. Jev is listed in the Workers AI catalog as a third-party model. I used Cloudflare for one practical reason: we already had credit on our Cloudflare account, and it kept every model call on one bill. There is a free daily allocation (10,000 Neurons a day at the time of writing), but some models, including third-party ones, need a paid plan or prepaid AI Gateway credits. You don’t have to use Cloudflare. TypeSafe offers Jev directly at https://api.typesafe.ai/v1/systemone with its own API key, plus [Python and JavaScript SDKs](https://docs.typesafe.ai/sdk.md). The request is the same idea: a state and a map of questions. Pick whichever fits your billing. The grader only cares that it sends a state and questions and gets typed answers back. Before any code, here’s the map. The [code folder](https://github.com/VersusControl/devops-ai-guidelines/tree/main/07-evaluating-ai-agents/code/chapter-08) has a handful of small files, and each one does one job: Two objects travel through these files. If you understand them, the rest of the code is easy to follow. **The** **Conclusion** is what the agent hands back. Some fields are written by the model, and some by the runner. For the run above, it looks roughly like this: ``` Conclusion( # written by the model root_cause="A deploy at 14:02 changed DB_MAX_CONNECTIONS 50 -> 5, which ...", category="deploy", cause_change="DB_MAX_CONNECTIONS 50 -> 5", cause_effect="pool_exhausted", evidence=["get_metrics", "get_logs", "get_deploys", "get_db_status"], rejected_signals=["payment_provider_latency", "checkout_image_v1_9_2", ...], # written by the runner, which the model can't edit trajectory=[{"type": "call_tool", "tool": "get_metrics"}, ..., {"type": "conclude"}], observations={"get_db_status": {"pool_size": 5, "in_use": 5, "waiting": 40}, ...},) ``` trajectory is the list of actions in order. observations is what each tool actually returned. These are the same two groups from the output figure earlier: what the agent *did* and what it *said*. **The answer key** is the truth, stored in the same JSON file but never given to the agent: ``` "answer_key": { "true_category": "deploy", "true_cause": "The 14:02 deploy reduced DB_MAX_CONNECTIONS from 50 to 5, exhausting the connection pool.", "cause_change": "DB_MAX_CONNECTIONS 50 -> 5", "cause_effect": "pool_exhausted", "required_evidence": ["get_deploys", "get_db_status"], "distraction_id": "payment_provider", "max_steps": 5} ``` (I trimmed a few fields that describe where the distraction appears.) Now the whole judged run fits in a few lines. This is the core of [run_judge.py](https://github.com/VersusControl/devops-ai-guidelines/blob/main/07-evaluating-ai-agents/code/chapter-08/run_judge.py), simplified: ``` agent_input, conclusion = run_agent(settings, case_path=case) # the agent investigates and commitsanswer = load_answer_key(case) # only now load the truthresult = evaluate(conclusion, answer, send_to_jev, DEMO_POLICY)print(result["status"]) # pass, fail, review_needed, or incomplete ``` And evaluate, in [jev_judge.py](https://github.com/VersusControl/devops-ai-guidelines/blob/main/07-evaluating-ai-agents/code/chapter-08/jev_judge.py), does three things: ``` hard = grade(conclusion, answer) # 1. the four hard gates, in plain Pythonpayload = build_request(conclusion, answer, ...) # 2. a short state plus two questions for Jevresponse = send(payload) # 3. HTTPS to Cloudflare, typed answers back# then check the reply and combine hard gates + Jev into one status ``` The four steps below walk through that flow: what goes into Jev, what we ask it, how the call works, and how to run the full grade. Jev can’t read a Python object, and it shouldn’t see everything anyway. So build_request takes the Conclusion and the answer key and makes one small dictionary, called the *state*. It's the only thing Jev reads about this run: ``` state = { "case_id": answer.scenario_id, "expected_cause": answer.true_cause, # from the hidden answer key "agent_explanation": conclusion.root_cause, # the paragraph the agent wrote "alert_started_at": conclusion.alert["started_at"], "observed_tool_responses": observed, # short facts, only from tools that ran} ``` Think of it as a note to a reviewer: “Here’s what really happened, here’s what the agent wrote, and here’s what its tools showed. Does the paragraph hold up?” Two details matter here. expected_cause is fine to send to the *judge*, because the agent has already answered and can't change its answer. And observed is a set of short facts computed from conclusion.observations, like {"pool_size": 5, "in_use": 5, "waiting": 40}, not raw log lines. Only tools the agent really called show up there. That keeps the state small (Jev likes that) and limits what leaves your machine. Before anything goes out, an approval step checks the state against the recorded case and rejects explanations that look like they contain a token, URL, or email address. That’s a guard for these synthetic cases, not a real redaction policy. For production incidents, you’d need a reviewed allowlist. The rubric is two questions in one request: ``` QUESTIONS = { "explanation_quality": { "type": "score", "instructions": "How well does the agent explanation connect the expected change to the observed checkout impact?", "criteria": [ "The explanation contradicts the observations or gives a wrong mechanism", "The explanation names the change but leaves out how it caused checkout impact", "The explanation connects the observed change, its supported mechanism, and checkout timeouts", ], }, "unsupported_claim": { "type": "noul", "instructions": "Does the agent explanation make a material claim unsupported by the observed tool responses?", "criteria": { "true": "A material claim has no support in the returned tool responses", "false": "The material claims follow from the returned tool responses", }, },} ``` The Score places the explanation on three described levels, 0 to 2. It comes back as a probability for each level, and the score is the weighted average. For example, probabilities of 0, 0.25, and 0.75 give a score of 0(0)+1(0.25)+2(0.75)=1.750(0)+1(0.25)+2(0.75)=1.75. The Noul asks one specific thing: did the agent claim something no tool showed? Watch the direction. A *high* value is *bad* here, because “true” means unsupported. I kept these as two questions on purpose. An explanation can describe the right mechanism and still invent a rollback that never happened. One blended “quality” number would hide that. Add three values to .env: CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_API_TOKEN (create it under Workers AI, "Use REST API"), and JEV_MODEL. The documented request looks like this: ``` url = f"https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/run"body = {"model": "typesafe/jev", "input": {"state": state, "questions": QUESTIONS}}# POST it with "Authorization: Bearer ". The answer is under result.answers. ``` The response has model, answers, and usage. The grader checks every field before using it: the types, the ranges, that the Score probabilities add up to 1, and that model is the version you calibrated against (jev-1.13.0). If Cloudflare or TypeSafe moves the model behind the name, the grade becomes incomplete until you re-check your thresholds. A silent model upgrade should never shift your benchmark. ``` python run_judge.py ``` This starts a new agent run, prints it in the same format as before, then prints the four gates, Jev’s model and two answers, and the final status. It exits nonzero for anything but pass, so it drops straight into CI. The thresholds live in plain Python, in DEMO_POLICY: a minimum score of 1.5, a minimum confidence of 0.6, and an unsupported-claim value of at most 0.2 to pass (0.8 or more is a clear failure; in between goes to review). I picked those to show the wiring. They are not measured numbers. More on that below. One grade for one run tells you very little. The real value of an eval is comparison: change one thing, run again, see which check moved. The repo has ten small simulation files in [scenarios/simulations](https://github.com/VersusControl/devops-ai-guidelines/tree/main/07-evaluating-ai-agents/code/chapter-08/scenarios/simulations). Each names a recorded case and the agent settings to use: ``` { "id": "checkout-latency-01", "case": "pool", "change": "Baseline: one vague instruction", "instructions": "Find out why checkout is slow. Call submit_diagnosis when you are done.", "max_steps": 5} ``` The tool data, the answer key, the gates, and the judge questions never change between files. Only the agent does. So when a result moves, you know what moved it. ``` python run_simulations.py # all ten, in parallelpython run_simulations.py --only checkout-latency-01python run_simulations.py --no-judge # hard gates only, no Cloudflare needed ``` Here are the hard-gate results from my run on the pool case, one prompt change at a time: What happened, in order: Then I pushed on version 04. With only three steps allowed, it ran out before answering, and every gate failed. That’s the eval catching a budget that’s too tight. With an alert that wrongly blamed the payment provider, it still found the deploy. And when I ran the *same* prompt against the other two incidents, it failed the distraction gate on both: it named the rejected signals after tools (get_db_status) instead of the component in the log line. One more instruction ("name each one after the component in the log line") fixed the capacity case, but on the dependency case the agent wrote db where the answer key expects db_pool. That last one is a good test of the grader, not just the agent. Is db close enough? Maybe. But if you loosen the rule, write it down as a scoring change and rerun every case. Don't edit the answer key until the table goes green. Where does Jev fit into this story? Look at version 01 again. Two exact checks failed, while the paragraph itself was fine. A judge that only reads prose would have passed it. The reverse happens too: a run can pass all four gates while its paragraph adds a claim no tool showed, like “the team rolled the change back at 14:05” (an example I made up). No gate reads prose, so only the judge sees that. You need both checks, and neither one gets to overrule the other. A few things I’d do before putting this in a real pipeline: If you want to go deeper, this harness is the running example in my free book, [Evaluating AI Agents](https://github.com/VersusControl/devops-ai-guidelines/tree/main/07-evaluating-ai-agents). It builds everything here from scratch: recording a case, replaying it, keeping the answer away from the agent, the hard gates, the Jev judge, and then turning scores into a benchmark and a CI gate. [Build an AI Agent Evaluation with JEV](https://pub.towardsai.net/build-an-ai-agent-evaluation-with-jev-392c146ca816) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.