{"slug": "loca-forward-only-llm-tuning-after-one-shot-calibration-with-local-credit", "title": "LoCA: Forward-Only LLM Tuning after One-Shot Calibration with Local Credit Assignment", "summary": "Researchers introduced Local Credit Assignment (LoCA), a two-stage method for parameter-efficient fine-tuning of large language models that requires only one backward pass for calibration, after which tuning proceeds forward-only. In evaluations on five discriminative benchmarks with Qwen2.5 models from 0.5B to 14B, LoCA achieved lower evaluation cross-entropy than LoRA in 16 of 25 task-scale comparisons, reduced full-run GPU peak memory by 26–29%, CPU steady-state memory by 36–52%, and per-pass time by 43–48% after calibration. The method enables tuning when repeated backpropagation is impractical.", "body_md": "arXiv:2608.03020v1 Announce Type: new\nAbstract: Parameter-efficient post-training reduces the number of trainable parameters, but still requires repeated end-to-end backpropagation through the frozen backbone. Every adaptation step therefore needs backward-capable hardware and must store or recompute activations. We ask whether this repeated backward chain can be replaced by a one-time calibration. We introduce Local Credit Assignment (LoCA), a two-stage method for small-shift adaptation. One probe backward pass fits a low-rank map at each transformer block from the final prediction error to a local hidden-state correction. LoCA then reuses these maps to form blockwise regression targets from forward activations and fits low-rank adapters with closed-form ridge solves. No further backbone backward pass is required. We evaluate LoCA on five discriminative benchmarks with Qwen2.5 models from 0.5B to 14B. In 16 of 25 reported task--scale comparisons, LoCA yields lower evaluation cross-entropy than the corresponding LoRA run. Its measured full-run GPU peak, including calibration, is 26--29\\% lower than LoRA's. After calibration, its CPU steady-state memory is 36--52\\% lower and its per-pass time is 43--48\\% lower. A shared scale-normalized candidate set is reused across all tested Qwen2.5 sizes and on SmolLM2-1.7B. LoCA thus amortizes global credit assignment into one calibration and enables later forward-only tuning when repeated backpropagation is impractical. The code associated with this paper is available \\href{https://github.com/Xia12121/LoCA}{here}.", "url": "https://wpnews.pro/news/loca-forward-only-llm-tuning-after-one-shot-calibration-with-local-credit", "canonical_source": "https://arxiv.org/abs/2608.03020", "published_at": "2026-08-05 04:00:00+00:00", "updated_at": "2026-08-05 04:10:17.718422+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "ai-infrastructure"], "entities": ["LoCA", "Qwen2.5", "LoRA", "SmolLM2-1.7B", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/loca-forward-only-llm-tuning-after-one-shot-calibration-with-local-credit", "markdown": "https://wpnews.pro/news/loca-forward-only-llm-tuning-after-one-shot-calibration-with-local-credit.md", "text": "https://wpnews.pro/news/loca-forward-only-llm-tuning-after-one-shot-calibration-with-local-credit.txt", "jsonld": "https://wpnews.pro/news/loca-forward-only-llm-tuning-after-one-shot-calibration-with-local-credit.jsonld"}}