Diet-KIT: Post-Training Quantization for Speech LLMs

Researchers from the Karlsruhe Institute of Technology developed Diet-KIT, a post-training quantization system that compresses the Qwen2-Audio-7B speech LLM from 16 GB to 3.98 GB while maintaining translation quality. The system uses Half-Quadratic Quantization with sensitivity-guided layer selection, achieving COMET scores of 74.4 on en→de and 77.1 on en→zh, compared to 75.6 and 79.5 for the uncompressed model.

Abstract We present Diet-KIT, a system for the IWSLT speech translation compression task under a strict 4 GB on-disk storage constraint, starting from the 16 GB Qwen2-Audio-7B base model. Compression is achieved with a sequential pipeline based on Half-Quadratic Quantization HQQ . Based on systematic ablations, we find that 4-bit quantization preserves translation quality well, whereas 3-bit quantization induces a sharp performance cliff, precluding aggressive compression across the whole model. We further show that the embedding table tolerates 2-bit quantization with negligible loss, while the LM head requires higher precision. To satisfy the storage constraint, we propose a sensitivity-guided layer selection method that identifies MLP sublayers tolerant to 3-bit compression via a per-layer sensitivity analysis, which consistently outperforms manual and random layer selection. Finally, AWQ calibration is applied as a data-driven refinement stage. The final system achieves 3.98 GB on disk with COMET scores of 74.4 on en→de and 77.1 on en→zh, compared to 75.6 and 79.5 for the uncompressed fine-tuned model.- Anthology ID: - 2026.iwslt-1.21 - Volume: Proceedings of the 23rd International Conference on Spoken Language Translation IWSLT 2026 /volumes/2026.iwslt-1/ - Month: - July - Year: - 2026 - Address: - San Diego, USA in-person and online - Editors: Elizabeth Salesky /people/elizabeth-salesky/ , Antonios Anastasopoulos /people/antonios-anastasopoulos/ , Matteo Negri /people/matteo-negri/ , Marcello Federico /people/marcello-federico/ - Venues: IWSLT /venues/iwslt/ | WS /venues/ws/ - SIG: SIGSLT /sigs/sigslt/ - Publisher: - Association for Computational Linguistics - Note: - Pages: - 189–196 - Language: - URL: https://aclanthology.org/2026.iwslt-1.21/ https://aclanthology.org/2026.iwslt-1.21/ - DOI: - Cite ACL : - Danni Liu, Sai Koneru, and Jan Niehues. 2026. Diet-KIT: Post-Training Quantization for Speech LLMs https://aclanthology.org/2026.iwslt-1.21/ . In Proceedings of the 23rd International Conference on Spoken Language Translation IWSLT 2026 , pages 189–196, San Diego, USA in-person and online . Association for Computational Linguistics. - Cite Informal : Diet-KIT: Post-Training Quantization for Speech LLMs https://aclanthology.org/2026.iwslt-1.21/ Liu et al., IWSLT 2026 - PDF: https://aclanthology.org/2026.iwslt-1.21.pdf https://aclanthology.org/2026.iwslt-1.21.pdf