Small questions, measurable gains with Jev – CraftCX

CraftCX moved four ticket-insight labels from OpenAI's GPT-5 nano to TypeSafe AI's Jev, improving mean macro recall by about 20% (79.3% to 95.6%) on 16 labeled synthetic conversations, according to a September 24, 2026 CraftCX blog post by Jason Dugdale. In a separate nine-case customer-signal grouping eval, the Jev decision path matched GPT-6 Luna's 9/9 expected choices while cutting estimated model cost 93% ($0.00288 to $0.00020) and total elapsed time roughly 16x (92.2 seconds to 5.8 seconds). CraftCX now uses Jev for classification and scoring via TypeSafe AI's Choice, Score and Noul primitives and keeps LLMs for text generation such as summaries and resolution notes.

All posts https://craftcx.com/blog September 24, 2026 by Jason Dugdale Small questions, measurable gains with Jev Explore with AI Consider a support chat where a CSV import fails and the upload guide leaves out a file size limit. Support works with the customer to fix the import, and the customer confirms the import succeeds. CraftCX needs to label that request as resolved and flag the documentation gap. Those labels help us find patterns across many support conversations. We moved those decisions and two other ticket-insight labels from GPT-5 nano, one of OpenAI's most efficient models, to Jev https://typesafe.ai/blog/introducing-system-one-models-and-jev by TypeSafe AI. On a set of 16 labeled synthetic conversations, mean macro recall improved by ~20% increasing from 79.3% to 95.6% . A separate signal-grouping eval found 93% lower estimated model cost and about 16x shorter total elapsed time across nine cases where both paths called a model. Ticket insights: finding the right label Jev takes the conversation and questions with defined answers, then returns typed decisions. It now handles resolution status, urgency, documentation gaps, and sentiment. The text model still writes summaries and resolution notes. We gave the GPT-5 nano and Jev paths the same 16 synthetic conversations. Each had reference labels for all four fields. For each possible label, we measured the share of expected examples that the path found, then averaged those recall scores across labels and fields. The largest gains on this set were sentiment and urgency, both improving 36%. We also revised the questions when moving to Jev, so the comparison measures the full decision path. These synthetic cases test our label rules; they do not estimate production recall. | Macro recall on 16 labeled conversations | GPT-5 nano | Jev | Relative improvement | |---|---|---|---| | Documentation gap | 85.7% | 92.9% | 8.4% | | Resolution status | 93.8% | 100.0% | 6.6% | | Sentiment | 68.9% | 95.8% | 39.0% | | Urgency | 68.8% | 93.8% | 36.3% | | Average macro recall improvement | 79.3% | 95.6% | 20.6% | Customer-signal grouping: making the same choice with less work This second eval used nine different synthetic cases where both paths called a model. CraftCX Support Intelligence https://craftcx.com/support-intelligence helps teams find common patterns or 'signals' in their support conversations - new incidents, frequent bug reports, recurring feature requests, etc. For each assessed ticket, we identify these signals, then determine if they're net new, or part of a larger recurring group. We compared a GPT-6 Luna grouping agent and review step with a Jev decision path. Both made the expected choice in all nine cases. | Grouping eval on 9 cases | GPT-6 Luna agent and review | Jev decision path | |---|---|---| | Expected choices made | 9/9 | 9/9 | | Estimated model cost | $0.00288 | $0.00020 | | Total elapsed time | 92.2 seconds | 5.8 seconds | Estimated model cost fell by 93%, and summed elapsed time across these cases was reduced by ~16x. We excluded one other case from this comparison because it gave Jev no candidates to choose from, so Jev made no model call. Both the model and workflow changed, so this measures the paths we tested rather than model speed alone. The eval scored grouping choices, not generated group text or production performance. System One Models These results give us a clear division of work. The Choice, Score and Noul primitives supported by the System One model allow us to offload most classification and scoring work to Jev at significantly reduced cost and latency, compared to LLMs. We continue to use LLMs for text generation tasks, such as explaining scoring and AI Support Agent performance. I'm excited about what System One models can support and enable, and already working to bring powerful new features to CraftCX powered by these models.