PermDoRA -- Understanding Adapter Interference in Language Models: Limits of Parameter-Space Geometry

A new study from researchers testing adapter composition in large language models found that geometry-aware merging strategies, which enforce orthogonality or directional independence in parameter updates, provide no consistent advantage over standard averaging in multi-domain settings. The analysis, conducted on LLaMA-3.1-8B and Mistral-7B using the DoRA-RBAC framework across multiple QA benchmarks, revealed that angular alignment and orthogonality of adapter updates are weak predictors of composition performance. These findings challenge the prevailing hypothesis that adapter interference arises primarily from overlap in linear parameter updates, suggesting instead that interactions in shared nonlinear representations are the dominant factor.

arXiv:2606.11262v1 Announce Type: new Abstract: Access control in large language models LLMs requires modular mechanisms to enable domain-specific behavior without retraining or cross-domain interference. A common hypothesis is that interference during adapter composition arises from overlap in linear parameter updates, suggesting that enforcing orthogonality or directional independence should improve multi-domain performance. We test this hypothesis using DoRA-RBAC, a hierarchical adapter composition framework based on weight-decomposed low-rank adaptation. We compare conventional Euclidean merging with a geometry-aware Riemannian-inspired merging strategy that approximates the Frechet mean via normalized directional averaging across multiple QA benchmarks GPQA, PubMedQA, SimpleQA, WMDP on LLaMA-3.1-8B and Mistral-7B. Our results show that while single-domain performance matches LoRA, geometry-aware merging provides no consistent advantage over standard averaging in multi-domain settings.Diagnostic analysis further reveals that angular alignment and orthogonality of adapter updates are weak predictors of composition performance. These findings suggest that adapter interference is not governed primarily by parameter-space geometry, but is instead consistent with interactions in shared nonlinear representations.