{"slug": "fine-tuning-small-language-models-for-cybersecurity-data-ordering-knowledge-and", "title": "Fine-Tuning Small Language Models for Cybersecurity: Data Ordering, Knowledge Distillation, and the Educator Effect", "summary": "Researchers at the Second International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security reported that fine-tuning small language models on synthetic cybersecurity data yields mixed results: Gemma 2 2B improved by 9.3 percentage points and Phi-3.5 3.8B by 4.0 points, while Llama 3.1 8B dropped 24.0 points, a phenomenon they call the 'educator effect.' The study, led by Ozkan Kilic, Raja Soundaramourty, and Ramu Chenchaiah, used QLoRA on V100 GPUs with ~147,600 QA pairs and found randomized data ordering outperformed curriculum ordering, though significance testing was not performed on the 75-question exam.", "body_md": "##### Abstract\n\nData-sovereignty rules forbid cloud-hosted AI in many high-security environments, leaving compact on-premise models as the only path to AI-assisted cybersecurity. We fine-tune three small open-source models, Gemma 2 2B, Phi-3.5 3.8B, Llama 3.1 8B, on ~147,600 synthetic cybersecurity QA pairs using QLoRA on V100 GPUs. Under strict MCQ evaluation Gemma 2 gains +9.3 pp, Phi +4.0 pp, and Llama drops −24.0 pp. We term this the educator effect: models trained on pedagogical data internalize explanatory behavior at the expense of format compliance. Severity appears to scale with capacity, though capacity is confounded with architecture and learning rate. A controlled ablation shows randomized ordering outperforms curriculum, without significance testing on the 75-question exam.- Anthology ID:\n- 2026.nlpaics-1.6\n- Volume:\n[Proceedings of the Second International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security](/volumes/2026.nlpaics-1/)- Month:\n- June\n- Year:\n- 2026\n- Address:\n- Alicante, Spain\n- Editors:\n[Ruslan Mitkov](/people/ruslan-mitkov/),[Rafael Muñoz](/people/rafael-munoz/),[Elena Lloret](/people/elena-lloret/unverified/),[Tharindu Ranasinghe](/people/tharindu-ranasinghe/),[Ernesto L. Estevanell-Valladares](/people/ernesto-luis-estevanell-valladares/),[Salima Lamsiyah](/people/salima-lamsiyah/unverified/),[Andrés Montoyo](/people/andres-montoyo/),[Saad Ezzini](/people/saad-ezzini/)- Venue:\n[NLPAICS](/venues/nlpaics/)- SIG:\n- Publisher:\n- Department of Languages and Information Systems, University of Alicante\n- Note:\n- Pages:\n- 55–63\n- Language:\n- URL:\n[https://aclanthology.org/2026.nlpaics-1.6/](https://aclanthology.org/2026.nlpaics-1.6/)- DOI:\n- Cite (ACL):\n- Ozkan Kilic, Raja Soundaramourty, and Ramu Chenchaiah. 2026.\n[Fine-Tuning Small Language Models for Cybersecurity: Data Ordering, Knowledge Distillation, and the Educator Effect](https://aclanthology.org/2026.nlpaics-1.6/). In*Proceedings of the Second International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security*, pages 55–63, Alicante, Spain. Department of Languages and Information Systems, University of Alicante. - Cite (Informal):\n[Fine-Tuning Small Language Models for Cybersecurity: Data Ordering, Knowledge Distillation, and the Educator Effect](https://aclanthology.org/2026.nlpaics-1.6/)(Kilic et al., NLPAICS 2026)- PDF:\n[https://aclanthology.org/2026.nlpaics-1.6.pdf](https://aclanthology.org/2026.nlpaics-1.6.pdf)", "url": "https://wpnews.pro/news/fine-tuning-small-language-models-for-cybersecurity-data-ordering-knowledge-and", "canonical_source": "https://aclanthology.org/2026.nlpaics-1.6/", "published_at": "2026-07-31 00:00:00+00:00", "updated_at": "2026-08-10 23:17:57.041013+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["Gemma 2 2B", "Phi-3.5 3.8B", "Llama 3.1 8B", "QLoRA", "V100 GPUs", "Ozkan Kilic", "Raja Soundaramourty", "Ramu Chenchaiah"], "alternates": {"html": "https://wpnews.pro/news/fine-tuning-small-language-models-for-cybersecurity-data-ordering-knowledge-and", "markdown": "https://wpnews.pro/news/fine-tuning-small-language-models-for-cybersecurity-data-ordering-knowledge-and.md", "text": "https://wpnews.pro/news/fine-tuning-small-language-models-for-cybersecurity-data-ordering-knowledge-and.txt", "jsonld": "https://wpnews.pro/news/fine-tuning-small-language-models-for-cybersecurity-data-ordering-knowledge-and.jsonld"}}