SSM Adapters via Hankel Reduced-order Modeling: Injection Site Determines Task Suitability in Long-Context Fine-Tuning

Researchers introduced Hankel Reduced order Model (HRM) adapters, a state space model-based fine-tuning method that outperforms LoRA on long-context tasks like QuALITY and QMSum by leveraging FFT-based parallel scan for computational efficiency.

arXiv:2606.26290v1 Announce Type: new Abstract: While parameter-efficient fine-tuning PEFT typically targets attention projectors, its efficacy for tasks requiring sequential state accumulation remains under-explored. We examine if PEFT for such tasks can benefit from state space model SSMs adapters, and if MLP blocks are better injection sites. We introduce Hankel Reduced order Model HRM adapter, an SSM-based residual module initialized via Balanced Truncation of empirical Hankel Grammians. By leveraging the time-invariance of the system matrix $\bar{A}$, HRM enables an exact FFT-based parallel scan, achieving computational parity with LoRA across all context lengths. In iso-parametric evaluations on Mistral-7B 8.4M trainable parameters , HRM outperforms LoRA variants on LongBench tasks, including QuALITY +34.8\% relative accuracy and QMSum +71.6\% relative ROUGE-1 . HRM further demonstrates consistent superiority across 18 configurations of synthetic state-tracking DFA, Parity and character-level language modeling enwik8 . Gate analysis reveals that HRM adapters effectively learn to modulate recurrence, providing a robust architectural alternative to low-rank adaptation for long-context sequence modeling.