Local Models vs Decision APIs: The Real Difference May Be Domain Knowledge

An internal SFEIR proof-of-concept comparing TypeSafe AI's decision-oriented SaaS model Jev against local models on 30,000 real IT support tickets found that a fine-tuned Laya model reached 62.0% department accuracy and 72.2% priority accuracy, beating Jev's 41.0% and 57.0%. Fine-tuning Laya improved its own zero-shot baseline by 17.2 percentage points on department classification and 20.4 points on priority classification, with training on 20,000 tickets and evaluation on 500 held-out tickets. SFEIR framed the result as an architectural question of whether enterprises with enough labeled business data should keep encoding business rules in prompts or train specialized models to learn decision boundaries directly.

This started with an internal POC at SFEIR involving Jev, TypeSafe AI’s decision-oriented SaaS model, and enterprise IT ticket classification. The initial question was straightforward: how well can a decision API classify our tickets without being trained on our company’s historical data? We compared several approaches on a dataset of real IT support tickets: The dataset contained around 30,000 tickets. We trained on 20,000 and evaluated on 500 held-out tickets. This was an initial experiment under time constraints, not a comprehensive benchmark. The results were particularly interesting: | Model | Department accuracy | Priority accuracy | |---|---|---| | Jev | 41.0% | 57.0% | | Qwen3-4B, zero-shot | 42.4% | 57.2% | | Laya, zero-shot | 44.8% | 51.8% | | Laya, LoRA | 57.8% | 65.6% | | Laya, full fine-tuning | 62.0% | 72.2% | The main observation wasn’t simply that a small local model could outperform a decision API. It was the difference between zero-shot classification and learning from actual business data. For this particular task, fine-tuning Laya improved accuracy by 17.2 percentage points for department classification and 20.4 points for priority classification compared with its zero-shot baseline. This raises an architectural question: when we have enough labeled business data, should we keep trying to describe our business rules through prompts and criteria, or should we train a specialized model to learn the decision boundaries directly? I’m interested in hearing from people working on enterprise AI: The full article is in French: Classification de texte en entreprise : arrêtez de demander à votre LLM de trier le courrier. https://www.sfeir.dev/ia/classification-de-texte-en-entreprise-arretez-de-demander-a-votre-llm-de-trier-le-courrier/ I’d love to hear your thoughts and experiences.