TL;DR:
This tutorial connects TypeSafe’s Jev to Arize AX through a remote evaluator. It evaluates recorded spans after the agent responds, scoring whether the response resolves the user’s request. You’ll return a yes/no label and probability score, test the integration on a span, and configure the evaluation of historical and incoming spans.
We run 32 evals on Alyx, our AI engineering agent. Most of them ask simple, bounded questions. Did the agent answer what the user asked? Did it pick the right tool? Every one of those judgments is an LLM call, and the cost grows with every trace we score.
That’s why we wanted to test TypeSafe’s Jev on our traces. Jev takes your data and bounded question, then returns a typed answer (you can learn more about Jev in our write up). For this walkthrough, that answer is the estimated probability that an agent’s response resolves the user’s request.
This comes with a tradeoff. Because Jev doesn’t generate text, it won’t explain why a response passed or failed. What you get for a yes/no question is an explicit probability. That’s not an explanation, but it’s a useful signal for deciding which results to accept and which to send for further review by an LLM judge.
We’ve already covered how Jev works, using it for real-time guardrails, and benchmarking it against Opus 5 and GPT-5.6 Terra on cost and accuracy. Here, we’ll connect Jev to Arize AX and score traces with a remote evaluator.
To follow along, you’ll need:
- an Arize AX project with spans containing agent inputs and outputs (use our quickstart guide to begin for free )
- A place to run the FastAPI service at a public HTTPS URL.
You can use the Jev remote-evaluator cookbook in our docs and sample agent and evaluator code for the full setup.
Let’s jump in.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Define what a resolved request means #
For this evaluator, we wanted to answer one simple question:
Does the response resolve the user’s request?
Jev supports several question types that suit evals. We’re using Noul, which returns the probability that the answer to a yes/no question is yes. You can swap it for choice to pick from a set of categories, or score to grade against a rubric, then update the response mapping to match.
Jev can also answer several questions against the same state in parallel, but we’re keeping this walkthrough to one question and one eval result. The TypeSafe docs cover the question types in detail.
Here’s the question from the evaluator:
QUESTION_NAME = "resolves_request"
QUESTIONS = {
QUESTION_NAME: {
"type": "noul",
"instructions": "Does the response resolve the user's request?",
"criteria": {
"true": "The response takes the action the user asked for, or clearly provides the requested solution.",
"false": "The response does not resolve the request, only acknowledges it, or defers without a solution.",
},
}
}
The criteria define what counts as yes and what counts as no. Writing a good agent eval still takes work: the question has to be clear, and the model needs enough context to answer it.
Here we pass Jev the user’s request and the agent’s response, so it judges the response on what it says. If you wanted to confirm that a refund actually went through, you’d also need to pass evidence from the tool call or the transaction.
This criterion measures against request resolution. A correct refusal or escalation can still count as unresolved, so use a separate criterion to evaluate whether that behavior was appropriate.
How remote evaluators work #
A remote evaluator is an HTTPS endpoint you host. Arize AX sends each record to your endpoint, your service scores it however it likes, and you send back a result. Your service can call Jev, another model, or your own logic.
For a span, the request looks like this:
{
"metadata": {
"request_id": "…",
"evaluator": "Jev Evaluator",
"record_id": "…"
},
"input": {
"input": "",
"output": ""
}
}
Metadata comes with every call, so you don’t set it. The evaluator’s input schema names the fields inside input, and each one is filled from the span. Here those are the span’s raw input and output values, which the OpenAI instrumentor records as JSON strings. Unpacked and trimmed, the input for our password question looks like this:
{
"model": "gpt-5.4-mini",
"instructions": "You are a customer support assistant...",
"input": "How can I reset my password if I forgot it?"
}
And the output is the full response object:
{
"id": "resp_…",
"model": "gpt-5.4-mini",
"output": [
{
"type": "message",
"content": [
{
"type": "output_text",
"text": "If you've forgotten your password, use the Forgot Password? link..."
}
]
}
]
}
So Jev sees the user’s question, the agent’s system prompt and the full response object, not just the reply text. Your endpoint returns a label, a score, or both:
{
"label": "yes",
"score": 0.82
}
That’s the complete contract. Anything that can accept that request and return that response can be an evaluator.
How we built a remote evaluator with Jev in Arize AX #
Our evaluator has three parts: traces to score, a small translation service in front of Jev, and an evaluator in Arize AX that knows how to call that service.
Capture agent inputs and outputs with OpenInference
You need traces before you can evaluate anything. For this test we built a small customer-support agent and instrumented it with the OpenInference instrumentor for OpenAI, so every call lands in our project as an LLM span. The span’s input holds the full request, including the system prompt, and its output holds the full response.
We gave it a set of requests. Half are general questions it can answer on the spot, like how to reset a password. The other half ask for changes to the customer’s account, such as a refund, and the agent has no tools to make them. Nothing in its prompt tells it to succeed or fail, so the results are whatever the model actually does.
Call Jev from a service exposed as an API
Arize AX and Jev speak different formats, so the evaluator is a thin translation layer: a small FastAPI service with one endpoint that gets called for every span. It does three things.
First, it turns the incoming request into Jev’s state. The input object already holds the span’s request and response, so it passes straight through, excluding fields where the value is None:
state = {key: value for key, value in body["input"].items() if value is not None}
Next, it sends that state and the question to Jev’s System One endpoint:
async with httpx.AsyncClient(timeout=30) as client:
jev = await client.post(
f"{TYPESAFE_BASE_URL}/v1/systemone",
headers={"Authorization": f"Bearer {TYPESAFE_API_KEY}"},
json={"model": JEV_MODEL, "state": state, "questions": QUESTIONS},
)
Jev’s answer comes back keyed by question name:
{
"model": "jev-1.13.0",
"answers": {
"resolves_request": {
"type": "noul",
"noul": 0.82
}
}
}
Finally, it maps the probability back to the response format from the contract:
jev.raise_for_status()
probability = jev.json()["answers"][QUESTION_NAME]["noul"]
return JSONResponse(
{
"label": "yes" if probability >= 0.5 else "no",
"score": probability,
}
)
We use 0.5 as a demonstration threshold. Choose a cutoff using human-labeled examples of your own requests and responses, then measure performance on a separate test set. Saving the raw probability lets you compare thresholds without calling Jev again. Existing stored labels still reflect the threshold used when they were created.
The evaluator gets called from outside your network, so the service needs a public HTTPS URL. In production that’s wherever you already host services. For our test, we ran it on a laptop behind a Cloudflare tunnel.
Configure and test the evaluator in Arize AX
The evaluator itself is a small piece of configuration. We named ours Jev Evaluator, scoped it to spans, and pointed it at the service’s endpoint URL. Then we described the input object with a JSON schema:
{
"type": "object",
"required": ["input"],
"properties": {
"input": {
"type": "object",
"required": ["input", "output"],
"properties": {
"input": { "type": "string" },
"output": { "type": "string" }
}
}
}
}
The schema is the contract between the two sides. It names the fields your endpoint expects, those fields get filled from the span, and whatever arrives becomes Jev’s state. You can also add custom headers to every call, which is where a shared secret goes in a production deployment.
Finally, we pointed the evaluator at the project holding our traces. When you pick the project, Arize AX maps input to the span’s attributes.input.value and output to attributes.output.value automatically, so there was nothing to map by hand. Check that both mappings contain the request and response you intend to evaluate. We filtered it to LLM spans, so it only scores the model calls where the request and reply live.
Before running anything at scale, Test Remote On Spans sends a single span to your endpoint and shows you the HTTP status and response body. It’s the quickest way to prove the whole path works: your service is reachable, your service reaches Jev, and a label and score come back.
The last piece is a task that decides when the evaluator runs. A one-time backfill scores the spans already in the project, and running continuously scores new spans as they arrive. We turned on both. Each of our spans got a label and score, and because those results are stored like any other eval, you can filter on them, chart them over time, and send the uncertain ones for human review.
What does Jev’s probability score mean? #
The score is Jev’s estimated probability that the answer is yes. Near 0 is a confident no, near 1 is a confident yes, and around 0.5 means Jev isn’t sure. It isn’t a measure of how much of the request was resolved. The Noul docs explain more.
We ran the sample requests through the agent and scored the resulting spans with jev-1.13.0 twice. The results barely moved between runs. Questions the agent could handle scored 0.8 and up. Questions where the agent said it couldn’t make the change scored 0.06 to 0.11.
Repeating these examples gives us an initial check on consistency. Measuring accuracy and calibration requires comparing the evaluator’s judgements with human labels on representative requests and responses.
What to test next with Jev #
Cost, speed, and quality together are what make this interesting. Cheap and fast only count if the judgments are useful, and what we’ve seen so far is good enough that we want to keep testing. If we can get reliable eval results at a fraction of the cost and latency, we can evaluate far more of our traffic, and run evals in places where an LLM call was too slow or too expensive.
Multiple questions in one call is the other thing we want to test. We’ve been researching whether a single LLM call can reliably judge several criteria at once, and we haven’t been able to make it work consistently. Jev is built to answer multiple questions against the same state in parallel, so it’s a direct test of a problem we’re already working on.
LLM-as-a-judge and agent-as-a-judge aren’t going away. Plenty of evals need deeper reasoning, a written explanation, or an agent that can go and gather more context before it decides. Those capabilities are worth paying for.
But Jev makes us question how many of our evals actually need them. The useful exercise is to go through your evals and ask which ones need a written explanation or additional context, and which can be handled as bounded decisions using the information already in the trace. For us, that includes evals we’ve held back on because of cost or latency.
Remote evaluators let you run that test on your own traces today. Native Jev as a Judge is coming to Arize AX very soon.
Run the Jev evaluator on your own agent traces #
We’ve written up the full walkthrough in the Arize docs, and the code for both the sample agent and the evaluator service is in the Arize-ai/tutorials repo.
To try Jev on your own agent traces, build a Jev remote evaluator in Arize AX.