{"slug": "lora-diffusion-parameter-efficient-fine-tuning-via-low-rank-trajectory", "title": "LoRA-Diffusion: Parameter-Efficient Fine-Tuning via Low-Rank Trajectory Decomposition", "summary": "Researchers propose LoRA-Diffusion, a parameter-efficient fine-tuning method that applies low-rank decomposition to the denoising trajectory of diffusion-based language models, achieving the highest mean performance on SST-2 and strong results on QNLI and MRPC while reducing per-task storage compared with full fine-tuning.", "body_md": "arXiv:2608.12328v1 Announce Type: new\nAbstract: Parameter-efficient fine-tuning methods such as LoRA have transformed the adaptation of large autoregressive language models, enabling task-specific customization with substantially fewer trainable parameters. However, these methods have not been successfully extended to diffusion-based language models, which generate text through iterative denoising rather than sequential token prediction. We propose LoRA-Diffusion, a parameter-efficient fine-tuning approach that applies low-rank decomposition to the denoising trajectory instead of model weights. Unlike weight-based LoRA, which modifies individual transformation matrices, our method learns low-rank perturbations to the entire diffusion path from noise to output. We introduce trajectory-level low-rank adapters that modify each denoising step, step-adaptive rank allocation across diffusion phases, and compositional multi-task learning that allows merging task-specific modules at inference without retraining. On SST-2, QNLI, and MRPC, we report token-level denoising validation accuracy over five random seeds. LoRA-Diffusion achieves the highest mean performance on SST-2 and strong performance on QNLI and MRPC. Joint multi-task training further shows that LoRA-Diffusion achieves the highest token-level accuracy among the evaluated methods. The approach reduces per-task storage compared with full fine-tuning and establishes a parameter-efficient fine-tuning framework for diffusion language models.", "url": "https://wpnews.pro/news/lora-diffusion-parameter-efficient-fine-tuning-via-low-rank-trajectory", "canonical_source": "https://arxiv.org/abs/2608.12328", "published_at": "2026-08-14 04:00:00+00:00", "updated_at": "2026-08-14 04:12:23.314714+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "generative-ai", "ai-research"], "entities": ["LoRA-Diffusion", "SST-2", "QNLI", "MRPC"], "alternates": {"html": "https://wpnews.pro/news/lora-diffusion-parameter-efficient-fine-tuning-via-low-rank-trajectory", "markdown": "https://wpnews.pro/news/lora-diffusion-parameter-efficient-fine-tuning-via-low-rank-trajectory.md", "text": "https://wpnews.pro/news/lora-diffusion-parameter-efficient-fine-tuning-via-low-rank-trajectory.txt", "jsonld": "https://wpnews.pro/news/lora-diffusion-parameter-efficient-fine-tuning-via-low-rank-trajectory.jsonld"}}