Fine-Tuning Small Language Models for Cybersecurity: Data Ordering, Knowledge Distillation, and the Educator Effect

Researchers at the Second International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security reported that fine-tuning small language models on synthetic cybersecurity data yields mixed results: Gemma 2 2B improved by 9.3 percentage points and Phi-3.5 3.8B by 4.0 points, while Llama 3.1 8B dropped 24.0 points, a phenomenon they call the 'educator effect.' The study, led by Ozkan Kilic, Raja Soundaramourty, and Ramu Chenchaiah, used QLoRA on V100 GPUs with ~147,600 QA pairs and found randomized data ordering outperformed curriculum ordering, though significance testing was not performed on the 75-question exam.

Abstract Data-sovereignty rules forbid cloud-hosted AI in many high-security environments, leaving compact on-premise models as the only path to AI-assisted cybersecurity. We fine-tune three small open-source models, Gemma 2 2B, Phi-3.5 3.8B, Llama 3.1 8B, on ~147,600 synthetic cybersecurity QA pairs using QLoRA on V100 GPUs. Under strict MCQ evaluation Gemma 2 gains +9.3 pp, Phi +4.0 pp, and Llama drops −24.0 pp. We term this the educator effect: models trained on pedagogical data internalize explanatory behavior at the expense of format compliance. Severity appears to scale with capacity, though capacity is confounded with architecture and learning rate. A controlled ablation shows randomized ordering outperforms curriculum, without significance testing on the 75-question exam.- Anthology ID: - 2026.nlpaics-1.6 - Volume: Proceedings of the Second International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security /volumes/2026.nlpaics-1/ - Month: - June - Year: - 2026 - Address: - Alicante, Spain - Editors: Ruslan Mitkov /people/ruslan-mitkov/ , Rafael Muñoz /people/rafael-munoz/ , Elena Lloret /people/elena-lloret/unverified/ , Tharindu Ranasinghe /people/tharindu-ranasinghe/ , Ernesto L. Estevanell-Valladares /people/ernesto-luis-estevanell-valladares/ , Salima Lamsiyah /people/salima-lamsiyah/unverified/ , Andrés Montoyo /people/andres-montoyo/ , Saad Ezzini /people/saad-ezzini/ - Venue: NLPAICS /venues/nlpaics/ - SIG: - Publisher: - Department of Languages and Information Systems, University of Alicante - Note: - Pages: - 55–63 - Language: - URL: https://aclanthology.org/2026.nlpaics-1.6/ https://aclanthology.org/2026.nlpaics-1.6/ - DOI: - Cite ACL : - Ozkan Kilic, Raja Soundaramourty, and Ramu Chenchaiah. 2026. Fine-Tuning Small Language Models for Cybersecurity: Data Ordering, Knowledge Distillation, and the Educator Effect https://aclanthology.org/2026.nlpaics-1.6/ . In Proceedings of the Second International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security , pages 55–63, Alicante, Spain. Department of Languages and Information Systems, University of Alicante. - Cite Informal : Fine-Tuning Small Language Models for Cybersecurity: Data Ordering, Knowledge Distillation, and the Educator Effect https://aclanthology.org/2026.nlpaics-1.6/ Kilic et al., NLPAICS 2026 - PDF: https://aclanthology.org/2026.nlpaics-1.6.pdf https://aclanthology.org/2026.nlpaics-1.6.pdf