{"slug": "efficient-on-device-diffusion-llm-inference-with-mobile-npu", "title": "Efficient On-Device Diffusion LLM Inference with Mobile NPU", "summary": "Researchers introduced llada.cpp, the first NPU-aware inference framework for accelerating diffusion large language models on smartphones, achieving 17x-42x latency reduction over CPU baselines while preserving generation quality. The framework addresses challenges like shrinking workloads, KV cache reuse, and memory overhead through multi-block speculative decoding, dual-path progressive revision, and swap-optimized memory runtime.", "body_md": "arXiv:2606.13740v1 Announce Type: new\nAbstract: Diffusion large language models (dLLMs) accelerate generation by denoising multiple tokens in parallel, making them attractive for latency-sensitive mobile inference. However, repeated denoising introduces substantial computation on smartphones. Mobile neural processing units (NPUs) offer high-throughput dense matrix computation, but efficiently exploiting them remains challenging: token commitment shrinks per-block effective workloads, token revision complicates KV cache reuse, and limited NPU-visible address space incurs costly remapping and data transfer overheads.\nIn this paper, we propose llada.cpp, the first NPU-aware inference framework for accelerating dLLMs on smartphones. llada.cpp aligns block-wise dLLM inference with the execution characteristics of mobile NPUs through three techniques. (1) Multi-Block Speculative Decoding fills the shrinking workload in late-stage current-block decoding with speculative future-block tokens. (2) Dual-Path Progressive Revision keeps committed tokens revisable until stable and refreshes unstable tokens through a CPU-side path without stalling dense NPU execution. (3) Swap-Optimized Memory Runtime compacts NPU-visible address layouts and overlaps data staging with NPU computation to reduce remapping and transfer overheads. We implement llada.cpp as an end-to-end framework and evaluate it across diverse hardware platforms and dLLM workloads. llada.cpp reduces LLaDA-8B generation latency by 17x-42x over the CPU baseline with prefix KV cache reuse, while preserving generation quality.", "url": "https://wpnews.pro/news/efficient-on-device-diffusion-llm-inference-with-mobile-npu", "canonical_source": "https://arxiv.org/abs/2606.13740", "published_at": "2026-06-15 04:00:00+00:00", "updated_at": "2026-06-15 04:19:36.522852+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "ai-infrastructure", "ai-research", "ai-tools"], "entities": ["llada.cpp", "LLaDA-8B", "NPU", "CPU", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/efficient-on-device-diffusion-llm-inference-with-mobile-npu", "markdown": "https://wpnews.pro/news/efficient-on-device-diffusion-llm-inference-with-mobile-npu.md", "text": "https://wpnews.pro/news/efficient-on-device-diffusion-llm-inference-with-mobile-npu.txt", "jsonld": "https://wpnews.pro/news/efficient-on-device-diffusion-llm-inference-with-mobile-npu.jsonld"}}