Jev-as-a-Judge: How to build a remote evaluator in Arize AX with Jev

Arize published a tutorial showing how to connect TypeSafe's Jev to Arize AX through a remote evaluator that scores recorded agent spans after the agent responds, returning a yes/no label and a probability that the response resolves the user's request. Arize runs 32 evals on its AI engineering agent Alyx, and the walkthrough uses Jev's Noul question type with a FastAPI service hosted at a public HTTPS URL, with a cookbook and sample code in the Arize-ai/tutorials GitHub repository. Because Jev does not generate text, it provides an explicit probability rather than an explanation, which Arize frames as a signal for deciding which results to accept and which to send to an LLM judge for further review.

TL;DR: This tutorial connects TypeSafe’s Jev to Arize AX https://arize.com/products/ax/ through a remote evaluator https://arize.com/docs/ax/evaluate/remote-evaluators . It evaluates recorded spans after the agent responds, scoring whether the response resolves the user’s request. You’ll return a yes/no label and probability score, test the integration on a span, and configure the evaluation of historical and incoming spans. We run 32 evals https://arize.com/glossary/evaluations/ on Alyx https://arize.com/products/alyx/ , our AI engineering agent. Most of them ask simple, bounded questions. Did the agent answer what the user asked? Did it pick the right tool? Every one of those judgments is an LLM call, and the cost grows with every trace https://arize.com/glossary/trace/ we score. That’s why we wanted to test TypeSafe’s Jev https://typesafe.ai/blog/introducing-system-one-models-and-jev on our traces. Jev takes your data and bounded question, then returns a typed answer you can learn more about Jev in our write up https://arize.com/blog/typesafe-jev-llm-judge/ . For this walkthrough, that answer is the estimated probability that an agent’s response resolves the user’s request. This comes with a tradeoff. Because Jev doesn’t generate text, it won’t explain why a response passed or failed. What you get for a yes/no question is an explicit probability. That’s not an explanation, but it’s a useful signal for deciding which results to accept and which to send for further review by an LLM judge https://arize.com/guides/llm-as-a-judge/ . We’ve already covered how Jev works https://arize.com/blog/typesafe-jev-llm-judge/ , using it for real-time guardrails https://arize.com/blog/llm-guardrails-jev/ , and benchmarking it against Opus 5 and GPT-5.6 Terra on cost and accuracy https://arize.com/blog/jev-as-a-judge/ . Here, we’ll connect Jev to Arize AX and score traces with a remote evaluator. To follow along, you’ll need: - an Arize AX project with spans containing agent inputs and outputs use our quickstart guide to begin for free https://arize.com/docs/ax/get-started/get-started-tracing - A place to run the FastAPI service at a public HTTPS URL. You can use the Jev remote-evaluator cookbook https://arize.com/docs/ax/cookbooks/evaluate/jev-remote-evaluator in our docs and sample agent and evaluator code https://github.com/Arize-ai/tutorials/tree/main/python/cookbooks/jev remote evaluator for the full setup. Let’s jump in. Build better agents with Arize Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today. Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability https://arize.com/phoenix?utm source=blog&utm medium=referral&utm campaign=ax-inline-cta&utm content=jev-remote-evaluator-inline-cta-phoenix . Define what a resolved request means For this evaluator, we wanted to answer one simple question: Does the response resolve the user’s request? Jev supports several question types that suit evals. We’re using Noul https://docs.typesafe.ai/primitives/noul , which returns the probability that the answer to a yes/no question is yes. You can swap it for choice to pick from a set of categories, or score to grade against a rubric, then update the response mapping to match. Jev can also answer several questions against the same state in parallel, but we’re keeping this walkthrough to one question and one eval result. The TypeSafe docs https://docs.typesafe.ai/introduction cover the question types in detail. Here’s the question from the evaluator: QUESTION NAME = "resolves request" QUESTIONS = { QUESTION NAME: { "type": "noul", "instructions": "Does the response resolve the user's request?", "criteria": { "true": "The response takes the action the user asked for, or clearly provides the requested solution.", "false": "The response does not resolve the request, only acknowledges it, or defers without a solution.", }, } } The criteria define what counts as yes and what counts as no. Writing a good agent eval still takes work https://arize.com/guides/ai-agent-handbook/agent-evaluation/ : the question has to be clear, and the model needs enough context to answer it. Here we pass Jev the user’s request and the agent’s response, so it judges the response on what it says. If you wanted to confirm that a refund actually went through, you’d also need to pass evidence from the tool call https://arize.com/glossary/tool-calling/ or the transaction. This criterion measures against request resolution. A correct refusal or escalation can still count as unresolved, so use a separate criterion to evaluate whether that behavior was appropriate. How remote evaluators work A remote evaluator is an HTTPS endpoint you host. Arize AX sends each record to your endpoint, your service scores it however it likes, and you send back a result. Your service can call Jev, another model, or your own logic. For a span, the request looks like this: { "metadata": { "request id": "…", "evaluator": "Jev Evaluator", "record id": "…" }, "input": { "input": "", "output": "" } } Metadata comes with every call, so you don’t set it. The evaluator’s input schema names the fields inside input, and each one is filled from the span. Here those are the span’s raw input and output values, which the OpenAI instrumentor records as JSON strings. Unpacked and trimmed, the input for our password question looks like this: { "model": "gpt-5.4-mini", "instructions": "You are a customer support assistant...", "input": "How can I reset my password if I forgot it?" } And the output is the full response object: { "id": "resp …", "model": "gpt-5.4-mini", "output": { "type": "message", "content": { "type": "output text", "text": "If you've forgotten your password, use the Forgot Password? link..." } } } So Jev sees the user’s question, the agent’s system prompt and the full response object, not just the reply text. Your endpoint returns a label, a score, or both: { "label": "yes", "score": 0.82 } That’s the complete contract. Anything that can accept that request and return that response can be an evaluator. How we built a remote evaluator with Jev in Arize AX Our evaluator has three parts: traces to score, a small translation service in front of Jev, and an evaluator in Arize AX that knows how to call that service. Capture agent inputs and outputs with OpenInference You need traces before you can evaluate anything. For this test we built a small customer-support agent and instrumented it with the OpenInference https://arize.com/docs/ax/concepts/otel-openinference/overview instrumentor for OpenAI, so every call lands in our project as an LLM span. The span’s input holds the full request, including the system prompt, and its output holds the full response. We gave it a set of requests. Half are general questions it can answer on the spot, like how to reset a password. The other half ask for changes to the customer’s account, such as a refund, and the agent has no tools to make them. Nothing in its prompt tells it to succeed or fail, so the results are whatever the model actually does. Call Jev from a service exposed as an API Arize AX and Jev speak different formats, so the evaluator is a thin translation layer: a small FastAPI service with one endpoint that gets called for every span. It does three things. First, it turns the incoming request into Jev’s state. The input object already holds the span’s request and response, so it passes straight through, excluding fields where the value is None: state = {key: value for key, value in body "input" .items if value is not None} Next, it sends that state and the question to Jev’s System One endpoint: async with httpx.AsyncClient timeout=30 as client: jev = await client.post f"{TYPESAFE BASE URL}/v1/systemone", headers={"Authorization": f"Bearer {TYPESAFE API KEY}"}, json={"model": JEV MODEL, "state": state, "questions": QUESTIONS}, Jev’s answer comes back keyed by question name: { "model": "jev-1.13.0", "answers": { "resolves request": { "type": "noul", "noul": 0.82 } } } Finally, it maps the probability back to the response format from the contract: jev.raise for status probability = jev.json "answers" QUESTION NAME "noul" return JSONResponse { "label": "yes" if probability = 0.5 else "no", "score": probability, } We use 0.5 as a demonstration threshold. Choose a cutoff https://arize.com/blog/jev-as-a-judge/ using human-labeled https://arize.com/glossary/human-evaluation/ examples of your own requests and responses, then measure performance on a separate test set. Saving the raw probability lets you compare thresholds without calling Jev again. Existing stored labels still reflect the threshold used when they were created. The evaluator gets called from outside your network, so the service needs a public HTTPS URL. In production that’s wherever you already host services. For our test, we ran it on a laptop behind a Cloudflare tunnel. Configure and test the evaluator in Arize AX The evaluator itself is a small piece of configuration. We named ours Jev Evaluator, scoped it to spans, and pointed it at the service’s endpoint URL. Then we described the input object with a JSON schema: { "type": "object", "required": "input" , "properties": { "input": { "type": "object", "required": "input", "output" , "properties": { "input": { "type": "string" }, "output": { "type": "string" } } } } } The schema is the contract between the two sides. It names the fields your endpoint expects, those fields get filled from the span, and whatever arrives becomes Jev’s state. You can also add custom headers to every call, which is where a shared secret goes in a production deployment. Finally, we pointed the evaluator at the project holding our traces. When you pick the project, Arize AX maps input to the span’s attributes.input.value and output to attributes.output.value automatically, so there was nothing to map by hand. Check that both mappings contain the request and response you intend to evaluate. We filtered it to LLM spans, so it only scores the model calls where the request and reply live. Before running anything at scale, Test Remote On Spans sends a single span to your endpoint and shows you the HTTP status and response body. It’s the quickest way to prove the whole path works: your service is reachable, your service reaches Jev, and a label and score come back. The last piece is a task that decides when the evaluator runs. A one-time backfill scores the spans already in the project, and running continuously scores new spans as they arrive. We turned on both. Each of our spans got a label and score, and because those results are stored like any other eval, you can filter on them, chart them over time, and send the uncertain ones for human review. What does Jev’s probability score mean? The score is Jev’s estimated probability that the answer is yes. Near 0 is a confident no, near 1 is a confident yes, and around 0.5 means Jev isn’t sure. It isn’t a measure of how much of the request was resolved. The Noul docs https://docs.typesafe.ai/primitives/noul explain more. We ran the sample requests through the agent and scored the resulting spans with jev-1.13.0 twice. The results barely moved between runs. Questions the agent could handle scored 0.8 and up. Questions where the agent said it couldn’t make the change scored 0.06 to 0.11. Repeating these examples gives us an initial check on consistency. Measuring accuracy and calibration requires comparing the evaluator’s judgements with human labels on representative requests and responses. What to test next with Jev Cost, speed, and quality together are what make this interesting. Cheap and fast only count if the judgments are useful, and what we’ve seen so far is good enough that we want to keep testing. If we can get reliable eval results at a fraction of the cost and latency, we can evaluate far more of our traffic, and run evals in places where an LLM call was too slow or too expensive. Multiple questions in one call is the other thing we want to test. We’ve been researching whether a single LLM call can reliably judge several criteria at once, and we haven’t been able to make it work consistently. Jev is built to answer multiple questions against the same state in parallel, so it’s a direct test of a problem we’re already working on. LLM-as-a-judge https://arize.com/guides/llm-as-a-judge/ and agent-as-a-judge https://arize.com/blog/agent-as-a-judge-evaluate-agents-with-agents/ aren’t going away. Plenty of evals need deeper reasoning, a written explanation, or an agent that can go and gather more context before it decides. Those capabilities are worth paying for. But Jev makes us question how many of our evals actually need them. The useful exercise is to go through your evals and ask which ones need a written explanation or additional context, and which can be handled as bounded decisions using the information already in the trace. For us, that includes evals we’ve held back on because of cost or latency. Remote evaluators let you run that test on your own traces today. Native Jev as a Judge is coming to Arize AX very soon. Run the Jev evaluator on your own agent traces We’ve written up the full walkthrough in the Arize docs https://arize.com/docs/ax/cookbooks/evaluate/jev-remote-evaluator , and the code for both the sample agent and the evaluator service is in the Arize-ai/tutorials https://github.com/Arize-ai/tutorials/tree/main/python/cookbooks/jev remote evaluator repo. To try Jev on your own agent traces, build a Jev remote evaluator in Arize AX https://arize.com/docs/ax/cookbooks/evaluate/jev-remote-evaluator .