{"slug": "when-do-llms-replace-fine-tuned-nlu-a-decision-framework-for-intent-detection-in", "title": "When Do LLMs Replace Fine-Tuned NLU? A Decision Framework for Intent Detection in Production Conversational Systems", "summary": "A new arXiv study (2608.20371v1) comparing zero-shot Claude Haiku against fine-tuned NLU models for intent detection finds that fine-tuned RoBERTa beats Claude by 11.8 points on ATIS (95.9 vs. 84.1, p<0.001) and is three orders of magnitude cheaper and faster, while the two are statistically tied on CLINC150 (89.1 vs. 88.5, p=0.24). The LLM excels in out-of-scope detection (OOS recall 85.6 vs. 58.1), robustness to ASR noise (92.5 vs. 80.0 at 0 dB), and dynamic schemas, where it serves multiple apps at ~94% accuracy with zero retraining versus 0% for a classifier trained on one app. The authors provide a decision framework for when to use LLMs versus fine-tuned classifiers in production conversational systems.", "body_md": "arXiv:2608.20371v1 Announce Type: new\nAbstract: A common claim is that zero-shot large language models (LLMs) can replace fine-tuned NLU classifiers for intent detection. We test this claim head-to-head and find that the honest answer is: it depends on the intent space. On full ATIS and CLINC150 we compare a fine-tuned RoBERTa, a TF-IDF+logistic-regression baseline, sentence-embedding kNN, and Claude Haiku zero-shot, reporting bootstrap 95% confidence intervals and paired significance tests. When abundant in-domain labels exist, fine-tuned RoBERTa is as good or better and three orders of magnitude cheaper and faster: on ATIS it beats Claude zero-shot by 11.8 points (95.9 vs. 84.1, p<0.001). On the broad 150-intent CLINC150 schema the two are statistically tied (89.1 vs. 88.5, p=0.24): the LLM matches a fully supervised model with no training data. The LLM's advantages appear in three production-relevant regimes: out-of-scope detection (OOS recall 85.6 vs. 58.1 for RoBERTa); robustness to realistic ASR noise via a controlled text-to-speech to noise to Whisper pipeline (92.5 vs. 80.0 at 0 dB); and dynamic per-deployment schemas, where a classifier trained on one app's intents scores 0% on a new app's intents while the schema-prompted LLM serves both at ~94% with zero retraining. We distill these findings into a decision framework for practitioners.", "url": "https://wpnews.pro/news/when-do-llms-replace-fine-tuned-nlu-a-decision-framework-for-intent-detection-in", "canonical_source": "https://arxiv.org/abs/2608.20371", "published_at": "2026-08-24 04:00:00+00:00", "updated_at": "2026-08-24 04:14:51.054657+00:00", "lang": "en", "topics": ["artificial-intelligence", "natural-language-processing", "large-language-models"], "entities": ["arXiv", "RoBERTa", "Claude Haiku", "ATIS", "CLINC150", "TF-IDF", "Whisper"], "alternates": {"html": "https://wpnews.pro/news/when-do-llms-replace-fine-tuned-nlu-a-decision-framework-for-intent-detection-in", "markdown": "https://wpnews.pro/news/when-do-llms-replace-fine-tuned-nlu-a-decision-framework-for-intent-detection-in.md", "text": "https://wpnews.pro/news/when-do-llms-replace-fine-tuned-nlu-a-decision-framework-for-intent-detection-in.txt", "jsonld": "https://wpnews.pro/news/when-do-llms-replace-fine-tuned-nlu-a-decision-framework-for-intent-detection-in.jsonld"}}