{"slug": "diet-kit-post-training-quantization-for-speech-llms", "title": "Diet-KIT: Post-Training Quantization for Speech LLMs", "summary": "Researchers from the Karlsruhe Institute of Technology developed Diet-KIT, a post-training quantization system that compresses the Qwen2-Audio-7B speech LLM from 16 GB to 3.98 GB while maintaining translation quality. The system uses Half-Quadratic Quantization with sensitivity-guided layer selection, achieving COMET scores of 74.4 on en→de and 77.1 on en→zh, compared to 75.6 and 79.5 for the uncompressed model.", "body_md": "##### Abstract\n\nWe present Diet-KIT, a system for the IWSLT speech translation compression task under a strict 4 GB on-disk storage constraint, starting from the 16 GB Qwen2-Audio-7B base model. Compression is achieved with a sequential pipeline based on Half-Quadratic Quantization (HQQ). Based on systematic ablations, we find that 4-bit quantization preserves translation quality well, whereas 3-bit quantization induces a sharp performance cliff, precluding aggressive compression across the whole model. We further show that the embedding table tolerates 2-bit quantization with negligible loss, while the LM head requires higher precision. To satisfy the storage constraint, we propose a sensitivity-guided layer selection method that identifies MLP sublayers tolerant to 3-bit compression via a per-layer sensitivity analysis, which consistently outperforms manual and random layer selection. Finally, AWQ calibration is applied as a data-driven refinement stage. The final system achieves 3.98 GB on disk with COMET scores of 74.4 on en→de and 77.1 on en→zh, compared to 75.6 and 79.5 for the uncompressed fine-tuned model.- Anthology ID:\n- 2026.iwslt-1.21\n- Volume:\n[Proceedings of the 23rd International Conference on Spoken Language Translation (IWSLT 2026)](/volumes/2026.iwslt-1/)- Month:\n- July\n- Year:\n- 2026\n- Address:\n- San Diego, USA (in-person and online)\n- Editors:\n[Elizabeth Salesky](/people/elizabeth-salesky/),[Antonios Anastasopoulos](/people/antonios-anastasopoulos/),[Matteo Negri](/people/matteo-negri/),[Marcello Federico](/people/marcello-federico/)- Venues:\n[IWSLT](/venues/iwslt/)|[WS](/venues/ws/)- SIG:\n[SIGSLT](/sigs/sigslt/)- Publisher:\n- Association for Computational Linguistics\n- Note:\n- Pages:\n- 189–196\n- Language:\n- URL:\n[https://aclanthology.org/2026.iwslt-1.21/](https://aclanthology.org/2026.iwslt-1.21/)- DOI:\n- Cite (ACL):\n- Danni Liu, Sai Koneru, and Jan Niehues. 2026.\n[Diet-KIT: Post-Training Quantization for Speech LLMs](https://aclanthology.org/2026.iwslt-1.21/). In*Proceedings of the 23rd International Conference on Spoken Language Translation (IWSLT 2026)*, pages 189–196, San Diego, USA (in-person and online). Association for Computational Linguistics. - Cite (Informal):\n[Diet-KIT: Post-Training Quantization for Speech LLMs](https://aclanthology.org/2026.iwslt-1.21/)(Liu et al., IWSLT 2026)- PDF:\n[https://aclanthology.org/2026.iwslt-1.21.pdf](https://aclanthology.org/2026.iwslt-1.21.pdf)", "url": "https://wpnews.pro/news/diet-kit-post-training-quantization-for-speech-llms", "canonical_source": "https://aclanthology.org/2026.iwslt-1.21/", "published_at": "2026-06-30 00:00:00+00:00", "updated_at": "2026-06-30 18:52:32.619779+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "ai-research"], "entities": ["Qwen2-Audio-7B", "Karlsruhe Institute of Technology", "IWSLT", "Half-Quadratic Quantization", "AWQ", "COMET"], "alternates": {"html": "https://wpnews.pro/news/diet-kit-post-training-quantization-for-speech-llms", "markdown": "https://wpnews.pro/news/diet-kit-post-training-quantization-for-speech-llms.md", "text": "https://wpnews.pro/news/diet-kit-post-training-quantization-for-speech-llms.txt", "jsonld": "https://wpnews.pro/news/diet-kit-post-training-quantization-for-speech-llms.jsonld"}}