All posts September 24, 2026
by
Jason Dugdale
Explore with AI
Consider a support chat where a CSV import fails and the upload guide leaves out a file size limit. Support works with the customer to fix the import, and the customer confirms the import succeeds. CraftCX needs to label that request as resolved and flag the documentation gap. Those labels help us find patterns across many support conversations.
We moved those decisions and two other ticket-insight labels from GPT-5 nano, one of OpenAI's most efficient models, to Jev by TypeSafe AI. On a set of 16 labeled synthetic conversations, mean macro recall improved by ~20% (increasing from 79.3% to 95.6%). A separate signal-grouping eval found 93% lower estimated model cost and about 16x shorter total elapsed time across nine cases where both paths called a model.
Ticket insights: finding the right label #
Jev takes the conversation and questions with defined answers, then returns typed decisions. It now handles resolution status, urgency, documentation gaps, and sentiment. The text model still writes summaries and resolution notes.
We gave the GPT-5 nano and Jev paths the same 16 synthetic conversations. Each had reference labels for all four fields. For each possible label, we measured the share of expected examples that the path found, then averaged those recall scores across labels and fields.
The largest gains on this set were sentiment and urgency, both improving >36%. We also revised the questions when moving to Jev, so the comparison measures the full decision path. These synthetic cases test our label rules; they do not estimate production recall.
| Macro recall on 16 labeled conversations | GPT-5 nano | Jev | Relative improvement |
|---|---|---|---|
| Documentation gap | 85.7% | 92.9% | 8.4% |
| Resolution status | 93.8% | 100.0% | 6.6% |
| Sentiment | 68.9% | 95.8% | 39.0% |
| Urgency | 68.8% | 93.8% | 36.3% |
| Average macro recall improvement | 79.3% | 95.6% | 20.6% |
Customer-signal grouping: making the same choice with less work #
This second eval used nine different synthetic cases where both paths called a model. CraftCX Support Intelligence helps teams find common patterns or 'signals' in their support conversations - new incidents, frequent bug reports, recurring feature requests, etc. For each assessed ticket, we identify these signals, then determine if they're net new, or part of a larger recurring group. We compared a GPT-6 Luna grouping agent and review step with a Jev decision path. Both made the expected choice in all nine cases.
| Grouping eval on 9 cases | GPT-6 Luna agent and review | Jev decision path |
|---|---|---|
| Expected choices made | 9/9 | 9/9 |
| Estimated model cost | $0.00288 | $0.00020 |
| Total elapsed time | 92.2 seconds | 5.8 seconds |
Estimated model cost fell by 93%, and summed elapsed time across these cases was reduced by ~16x. We excluded one other case from this comparison because it gave Jev no candidates to choose from, so Jev made no model call. Both the model and workflow changed, so this measures the paths we tested rather than model speed alone. The eval scored grouping choices, not generated group text or production performance.
System One Models #
These results give us a clear division of work. The Choice, Score and Noul primitives supported by the System One model allow us to offload most classification and scoring work to Jev at significantly reduced cost and latency, compared to LLMs. We continue to use LLMs for text generation tasks, such as explaining scoring and AI Support Agent performance.
I'm excited about what System One models can support and enable, and already working to bring powerful new features to CraftCX powered by these models.