This started with an internal POC at SFEIR involving Jev, TypeSafe AI’s decision-oriented SaaS model, and enterprise IT ticket classification.
The initial question was straightforward: how well can a decision API classify our tickets without being trained on our company’s historical data?
We compared several approaches on a dataset of real IT support tickets:
The dataset contained around 30,000 tickets. We trained on 20,000 and evaluated on 500 held-out tickets. This was an initial experiment under time constraints, not a comprehensive benchmark.
The results were particularly interesting:
| Model | Department accuracy | Priority accuracy |
|---|---|---|
| Jev | 41.0% | 57.0% |
| Qwen3-4B, zero-shot | 42.4% | 57.2% |
| Laya, zero-shot | 44.8% | 51.8% |
| Laya, LoRA | 57.8% | 65.6% |
| Laya, full fine-tuning | 62.0% | 72.2% |
The main observation wasn’t simply that a small local model could outperform a decision API.
It was the difference between zero-shot classification and learning from actual business data.
For this particular task, fine-tuning Laya improved accuracy by 17.2 percentage points for department classification and 20.4 points for priority classification compared with its zero-shot baseline. This raises an architectural question: when we have enough labeled business data, should we keep trying to describe our business rules through prompts and criteria, or should we train a specialized model to learn the decision boundaries directly?
I’m interested in hearing from people working on enterprise AI:
The full article is in French: Classification de texte en entreprise : arrêtez de demander à votre LLM de trier le courrier.
I’d love to hear your thoughts and experiences.