# Fine-Tuning Small Language Models for Cybersecurity: Data Ordering, Knowledge Distillation, and the Educator Effect

> Source: <https://aclanthology.org/2026.nlpaics-1.6/>
> Published: 2026-07-31 00:00:00+00:00

##### Abstract

Data-sovereignty rules forbid cloud-hosted AI in many high-security environments, leaving compact on-premise models as the only path to AI-assisted cybersecurity. We fine-tune three small open-source models, Gemma 2 2B, Phi-3.5 3.8B, Llama 3.1 8B, on ~147,600 synthetic cybersecurity QA pairs using QLoRA on V100 GPUs. Under strict MCQ evaluation Gemma 2 gains +9.3 pp, Phi +4.0 pp, and Llama drops −24.0 pp. We term this the educator effect: models trained on pedagogical data internalize explanatory behavior at the expense of format compliance. Severity appears to scale with capacity, though capacity is confounded with architecture and learning rate. A controlled ablation shows randomized ordering outperforms curriculum, without significance testing on the 75-question exam.- Anthology ID:
- 2026.nlpaics-1.6
- Volume:
[Proceedings of the Second International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security](/volumes/2026.nlpaics-1/)- Month:
- June
- Year:
- 2026
- Address:
- Alicante, Spain
- Editors:
[Ruslan Mitkov](/people/ruslan-mitkov/),[Rafael Muñoz](/people/rafael-munoz/),[Elena Lloret](/people/elena-lloret/unverified/),[Tharindu Ranasinghe](/people/tharindu-ranasinghe/),[Ernesto L. Estevanell-Valladares](/people/ernesto-luis-estevanell-valladares/),[Salima Lamsiyah](/people/salima-lamsiyah/unverified/),[Andrés Montoyo](/people/andres-montoyo/),[Saad Ezzini](/people/saad-ezzini/)- Venue:
[NLPAICS](/venues/nlpaics/)- SIG:
- Publisher:
- Department of Languages and Information Systems, University of Alicante
- Note:
- Pages:
- 55–63
- Language:
- URL:
[https://aclanthology.org/2026.nlpaics-1.6/](https://aclanthology.org/2026.nlpaics-1.6/)- DOI:
- Cite (ACL):
- Ozkan Kilic, Raja Soundaramourty, and Ramu Chenchaiah. 2026.
[Fine-Tuning Small Language Models for Cybersecurity: Data Ordering, Knowledge Distillation, and the Educator Effect](https://aclanthology.org/2026.nlpaics-1.6/). In*Proceedings of the Second International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security*, pages 55–63, Alicante, Spain. Department of Languages and Information Systems, University of Alicante. - Cite (Informal):
[Fine-Tuning Small Language Models for Cybersecurity: Data Ordering, Knowledge Distillation, and the Educator Effect](https://aclanthology.org/2026.nlpaics-1.6/)(Kilic et al., NLPAICS 2026)- PDF:
[https://aclanthology.org/2026.nlpaics-1.6.pdf](https://aclanthology.org/2026.nlpaics-1.6.pdf)