Jev vs. Luna for AI Observability

Groundcover is evaluating TypeSafe's newly released Jev model for AI observability, testing whether its parallel-sampler architecture and reinforcement learning for calibrated decisions can classify agent trace outcomes more cheaply than generative LLMs. The team defined Status (success, fail, pending) and Sentiment (satisfaction, frustration, undecided) dimensions and ran the experiment on synthetic conversation and tool-response data from its own agent-service traces, not production content.

For AI engineering teams building agents, classic observability signals are not enough. All traces can be fast and green, logs all info and all infra metrics healthy - and users can still walk out disappointed. The agent might have answered the wrong question or stopped short of the requested work over a missing tool or some critical piece of information. In the AI realm, a perfectly healthy request does not equal a happy user. Extracting these semantic intelligence signals from traces is part of what we are working on in the AI observability effort at groundcover. Together with latency, tokens, and tool calls which are signals that are more on the infra side, we need structured answers to questions like “ Did the agent finish the job?” or “ How did the user feel during each turn of the conversation?”. A generative LLM can help us achieve this, but enrichment at trace volume means paying for every judgment with precious time and cash. TypeSafe’s newly released Jev offers an interesting alternative: a model designed specifically to choose among predefined options and return a probability distribution over them. Compared to traditional generative models, which would generate JSON containing the requested probabilities, this sounded like a great candidate for our agent-success and user-sentiment classification job, which piqued our interest. Why Jev caught our attention TypeSafe calls Jev a System One model . Its documentation describes reinforcement learning for calibrated decisions RLCD , a post-training approach aimed at making decisions and their probabilities match outcomes. To put it simply, this means when the model says it’s 80% sure on 100 different traces, we should expect it to be right on about 80 of them. TypeSafe’s AI primer https://docs.typesafe.ai/introduction/machine-learning-primer explains their approach with more detail. How Jev produces its answers is interesting too. TypeSafe describes their novel architecture as a parallel sampler that produces these probability outputs together - rather than generating answers token by token. Since we already know the fields we want to infer status, sentiment… and allowed labels success/fail, happy/frustrated… this promise of parallel output fits our trace enrichment use-case nicely. We need a model to interpret the conversation and evidence, then return a small decision our code can use. Jev’s Choice primitive https://docs.typesafe.ai/primitives/choice lets us define those labels and their meaning, letting us ask separate questions, like status and sentiment, about the same trace. First, define what a good answer means We chose two separate dimensions: Status describes whether the latest actionable request was satisfied. Sentiment captures the latest attitude the user expressed toward the assistant and not whether i.e. the incident the user is reporting was bad . type Status = | "success" // The intended work was completed | "fail" // The run ended without satisfying the request | "pending" // Necessary user input is still missing type Sentiment = | "satisfaction" // The user expresses approval | "frustration" // The user expresses dissatisfaction | "undecided" // The conversation does not establish either For each of these dimensions, the model gives a possible label a probability. We pick the most likely label and use its probability as the score. Jev also has a separate confidence measure that describes how strongly it favors one answer over the others. Even a confident answer in correctly formatted JSON can be wrong. More on TypeSafe’s confidence measure https://docs.typesafe.ai/confidence . For inspiration, we inspected traces from ‘agent-service’, groundcover’s own agent, in one of our production BYOC environments to understand span structure. The conversations and tool response sample set used for this experiment is synthetic . We did not use production content as benchmark data. For each conversation, we also created the expected labels before inference, which we didn’t share with either classifier. Smoke test: can both models do the job? We started with 10 hand-authored cases covering all nine possible status-sentiment combinations. Successful work AND a frustrated user can coexist, just as a failed task can still be followed by positive feedback. It was important for us to establish that both models can successfully infer these combos. For each case, we noted the expected sentiment and status. We were off to a good start as both models got all ten status and sentiment labels right. Jev’s estimated cost was about 0.23¢ . The selected compact-schema Luna run was about 0.87¢ at Standard API rates. This was a useful check that the task and output contract worked. To be fair, for Luna we used a structured output request, and the model returned only the requested probabilities without using tools and without any added prose. Our application code could then recover labels, scores, and trace IDs from the compact result. 100 traces - Differences start to surface For phase 2 we kept the original hand-crafter ten traces, handed them to an and LLM and asked it to create 90 more scenarios in the same style. We fed Jev and Luna with the same complete conversation and tool transcripts, and ran four concurrent requests per model. Jev correctly classified status in 99 of 100 cases, compared to 97 for Luna. In sentiment we started seeing a split though: 80 of 100 for Jev and 90 for Luna. It’s important to note here that all twenty of Jev and Luna’s sentiment “mistakes” selected “undecided”. This makes sense, as sentiment is by nature a tricky thing to infer. Some conversations contained stronger signals like the user replying “Thanks, this is really great” or “I’m not happy with this”, while others were way more subtle. The cases in question were notably of the second, not clear-cut variety. 10,000 traces - Watching it scale For our largest run, we spawned subagents to create 10,000 traces across 104 scenario families with different requirements, tool evidence, entities, and conversation histories. The result was a corpus of 10,000 traces which contained 95,494 spans. We created the 10,000 traces by varying details within 104 base scenarios. Many traces therefore describe similar situations. These results only show how the models performed on our generated examples - they don’t tell us how accurate they would be on real production traces. Higher accuracy in this run also doesn’t mean the models improved compared to the 100 traces phase - as we tested them on different examples. On this set, Jev reached 99.81% status accuracy and 91.27% sentiment accuracy . Luna reached 99.55% and 99.50% , respectively. This further strengthened our notion that at least in this scenario, Jev was being more conservative about deciding whether a sentiment was either positive or negative and preferred to go with the safer “undecided”. The bonanza: Cost and timing This is where things got interesting: the cost and timing difference was substantial. Jev processed the 10,000-trace workload in 17 minutes 3 seconds , at an estimated $0.782 . Luna took a whopping 82 minutes 12 seconds , with recorded usage priced at $2.717 under Standard rates as opposed to “Fast” rates, which would roughly be double . Where Jev fits and where to be careful So, at this point we have a good sense that enrichming traces is a good match for Jev’s strength. There are known labels, repeated decisions, typed results - and putting it all together is a software workflow that owns what happens next. Our “Status” label performed very well in our runs Plus we learned that asking about status and sentiment as separate “Choice” questions over the same evidence keeps their meanings separate, as advertised. Jev’s limitations, however, matter just as much. What it does really well is it selects from answers we defined. It does not and cannot invent, for example, a new topic name or write an explanation. TypeSafe’s Jev 1.13 limitations https://docs.typesafe.ai/model-jaggedness/jev-1.13 documentation also highlights weaknesses around counting, arithmetic, date comparisons and irrelevant long context - all of which traces can contain. So - exact timing and arithmetic belong in code, while relevant context and tool evidence will require careful handling in our flow. The interesting next step is an LLM fallback TypeSafe has a cool smart-home demo https://docs.typesafe.ai/demos/smart-home typesafe-and-llm-pairing that pairs Jev with a traditional LLM. In the demo we send compound requests to an LLM to split into smaller commands, then these commands are evaluated with TypeSafe. This way TypeSafe handles bounded judgments, while the GenAI model handles, well, work that needs generation. In the context of our AI observability effort, we could see this coming in handy for inferring conversation topics. Topics are a major data point required by agent builders in addition to status and sentiment - so we’d also like to classify what the user was actually trying to do create a dashboard, buy a fridge, write a piece of code… . We can expect topic count for a given agent to have a low-cardinality nature, so most traces should map to an existing taxonomy. However, sometimes none of the existing topics in our DB fits, or a narrow topic would be better represented by a more general category to include the currently examined trace. We can make those possibilities explicit enum options: TopicDecision = “incident investigation” | “configuration help” | “product question” | “propose new topic” | “review topic grouping” Jev could choose a known topic or recommend escalation through one of the last two options. Our code could then pass the ball to an LLM that would inspect the trace, current taxonomy, and representative examples, then propose a new topic or a consolidation. Jev is choosing a declared action here, not generating a novel enum value itself. This mixed model approach could allow us to leverage Jev for what it seems to excel in. Promising enough to keep pushing Between the great status classification, the typed interface and the considerably lower cost and time footprint, Jev is showing a real promise. The gap in sentiment analysis is something to watch out for here though. In further experiments we could try and improve on these results with different prompting, and it’s likely that future model iterations will also bridge those quality differences. It’s also worth mentioning that In a real-world scenario groundcover has a bunch of other signals to correlate these model predictions with to further reinforce success or failure. The next meaningful test should also be a more representative set with properly reviewed real conversations from real-world agents running in production, including our own. Beyond status and sentiment, we’d also want to evaluate the topic-enum fallback, as well as other attributes and KPIs our users care about. How to read these numbers Selected configurations Jev: Our first run requested “jev-latest” and returned jev-1.13.0. Subsequent runs pinned that version. Two independent “Choice” questions shared a single trace’s evidence. The 100 and 10,000 trace runs used persistent HTTP connections and four workers; the large run had a 16-request/second upper cap. Luna: We requested “gpt-5.6-luna” with medium reasoning, ten traces per request, strict compact JSON, no tool calls and no narrative output or prose of any sort. Our code reconstructed labels, scores, and trace IDs. The larger runs used four workers. All of these are the selected settings we measured, not a claim of optimal settings. Workload timing The initial ten-trace Jev pilot made sequential requests and used full span fixtures. In later runs we used four workers and removed repeated telemetry copies while preserving complete transcripts. This is why the 100-case Jev workload took less time than the ten-case pilot - it is not evidence of a model scaling effect. In all batched inference runs we recorded both the internal client elapsed timers, and the “total time” which is UTC finish minus start. Luna’s UTC interval was 4,932.200 seconds, while its combined client’s elapsed timer was 4,819.977 seconds, a 112.223-second difference whose cause we did not establish. Three attempts timed out and succeeded on retry. The workload time compensates for retry delays. Request-level median times were 0.348 seconds for Jev’s single trace requests and 18.141 seconds for Luna’s ten-trace batches. Token accounting For the large run, Jev reported 18,616,884 input tokens. Luna reported 19,905,049 input tokens, including 11,247,872 cached tokens, and 633,724 output tokens, including 180,838 reasoning tokens. That leaves 452,886 non-reasoning output tokens, about 45.3 per trace. Reasoning is already part of output, so were not charged twice in our estimate. Costs use published rates checked on Sep 17, 2026: TypeSafe pricing https://docs.typesafe.ai/models and OpenAI API pricing https://developers.openai.com/api/docs/pricing . They are API-usage estimates, not account invoices or Codex subscription charges.