{"slug": "chaindora-tensor-train-factorized-weight-decomposed-low-rank-adaptation-for-llm", "title": "ChainDoRA: Tensor-Train Factorized Weight-Decomposed Low-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning", "summary": "ChainDoRA, a weight-decomposed parameter-efficient fine-tuning framework that builds directional low-rank factors from a connected Tensor-Train chain, reached a seven-task average accuracy of 72.30% on commonsense reasoning benchmarks with LLaMA-7B, versus 69.88% for LoRA and 69.39% for DoRA, according to the arXiv paper 2609.25058v1. ChainDoRA used only 5.35M trainable parameters compared with 56.10M for LoRA and 56.98M for DoRA, a 90.62% reduction relative to DoRA, in a controlled 15,119-example response-only adaptation setting. Ablations over TT rank and adapter placement showed controllable parameter-accuracy trade-offs, indicating connected TT parameterization can cut the parameter cost of magnitude-direction adaptation while preserving downstream reasoning performance.", "body_md": "arXiv:2609.25058v1 Announce Type: new \nAbstract: Parameter-efficient fine-tuning (PEFT) adapts large language models (LLMs) to downstream tasks while updating only a small fraction of their pretrained parameters. Low-Rank Adaptation (LoRA) uses two trainable low-rank matrices, while Weight-Decomposed Low-Rank Adaptation (DoRA) further separates weight magnitude and direction but retains the dense LoRA-style factorization in its directional branch. We propose ChainDoRA, a weight-decomposed adaptation framework that constructs the directional low-rank factors from a connected Tensor-Train (TT) chain, where the adapter rank forms the boundary rank between input- and output-side TT contractions and an independent TT rank controls representation capacity and parameter cost. Under a controlled 15,119-example response-only adaptation setting with LLaMA-7B, ChainDoRA is evaluated against matched LoRA and DoRA baselines on seven commonsense reasoning benchmarks. ChainDoRA with TT rank 16 achieves a seven-task average accuracy of 72.30%, compared with 69.88% for LoRA and 69.39% for DoRA, while requiring only 5.35M trainable parameters versus 56.10M for LoRA and 56.98M for DoRA, corresponding to a 90.62% reduction relative to DoRA. Ablations over TT rank and adapter placement show controllable parameter-accuracy trade-offs, indicating that connected TT parameterization can substantially reduce the parameter cost of magnitude-direction adaptation while preserving, and in this setting improving, downstream reasoning performance.", "url": "https://wpnews.pro/news/chaindora-tensor-train-factorized-weight-decomposed-low-rank-adaptation-for-llm", "canonical_source": "https://arxiv.org/abs/2609.25058", "published_at": "2026-09-23 04:00:00+00:00", "updated_at": "2026-09-23 04:25:46.485385+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "machine-learning", "natural-language-processing"], "entities": ["ChainDoRA", "LoRA", "DoRA", "Tensor-Train", "LLaMA-7B", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/chaindora-tensor-train-factorized-weight-decomposed-low-rank-adaptation-for-llm", "markdown": "https://wpnews.pro/news/chaindora-tensor-train-factorized-weight-decomposed-low-rank-adaptation-for-llm.md", "text": "https://wpnews.pro/news/chaindora-tensor-train-factorized-weight-decomposed-low-rank-adaptation-for-llm.txt", "jsonld": "https://wpnews.pro/news/chaindora-tensor-train-factorized-weight-decomposed-low-rank-adaptation-for-llm.jsonld"}}