{"slug": "when-synthetic-data-hurts-on-catastrophic-forgetting-in-skill-retrieval-for-llm", "title": "When Synthetic Data Hurts: On Catastrophic Forgetting in Skill Retrieval for LLM Agents", "summary": "A production skill router covering 34,396 skills shows that fine-tuning on synthetic data improves in-distribution skill retrieval for LLM agents but causes catastrophic forgetting on real and out-of-distribution data, according to an arXiv paper (2609.10750v1). Forgetting-mitigation approaches borrowed from continual learning — embedding-anchor regularization, Learning without Forgetting (LwF), Elastic Weight Consolidation (EWC), and L2-initialization — retained OOD retrieval performance and improved synthetic in-distribution retrieval by 13.98% for a 0.6B Qwen retriever and reranker. The authors present the results as a practical benchmark and fine-tuning recipe for scarce, multi-positive supervision.", "body_md": "arXiv:2609.10750v1 Announce Type: cross \nAbstract: LLM agents increasingly rely on external skills retrieved at runtime, making skill selection from large repositories a critical challenge. We present a production skill router over 34,396 skills and a large-scale study of skill retrieval using limited real supervision and synthetic data. We found that the synthetic-data fine-tuning improves in-distribution retrieval but it causes catastrophic forgetting on real and out-of-distribution (OOD) data. We evaluate several forgetting mitigation fine-tuning approaches inspired by continual learning, including embedding-anchor regularization, Learning without Forgetting (LwF), Elastic Weight Consolidation (EWC), and L2-initialization. The results show that these approaches not only retain the performance on OOD skills retrieval but also improve the retrieval on synthetic in-distribution skills by 13.98\\% for 0.6B Qwen retriever and reranker. Our results provide a practical benchmark and a robust fine-tuning recipe for scarce, multi-positive supervision.", "url": "https://wpnews.pro/news/when-synthetic-data-hurts-on-catastrophic-forgetting-in-skill-retrieval-for-llm", "canonical_source": "https://www.machinebrief.com/news/when-synthetic-data-hurts-on-catastrophic-forgetting-in-skil-iiyc", "published_at": "2026-09-11 04:00:00+00:00", "updated_at": "2026-09-11 06:27:45.352497+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-research", "machine-learning", "natural-language-processing"], "entities": ["Qwen", "Learning without Forgetting", "Elastic Weight Consolidation", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/when-synthetic-data-hurts-on-catastrophic-forgetting-in-skill-retrieval-for-llm", "markdown": "https://wpnews.pro/news/when-synthetic-data-hurts-on-catastrophic-forgetting-in-skill-retrieval-for-llm.md", "text": "https://wpnews.pro/news/when-synthetic-data-hurts-on-catastrophic-forgetting-in-skill-retrieval-for-llm.txt", "jsonld": "https://wpnews.pro/news/when-synthetic-data-hurts-on-catastrophic-forgetting-in-skill-retrieval-for-llm.jsonld"}}