{"slug": "sefora-sketch-aggregated-federated-low-rank-adaptation-with-heterogeneous-client", "title": "SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks", "summary": "Researchers propose SeFoRA, a sketch-aggregated federated low-rank adaptation algorithm that enables direct aggregation of client updates with heterogeneous LoRA ranks, alleviating bilinear mismatch and reducing communication to a small subspace. The rank-homogeneous variant SeFoRA-Ho converges to a first-order stationary point at rate O(1/T), and experiments fine-tuning RoBERTa-Large on GLUE datasets show it outperforms state-of-the-art methods.", "body_md": "arXiv:2608.10144v1 Announce Type: new\nAbstract: We consider federated parameter efficient fine-tuning of large neural networks with low-rank adaptation (LoRA,~Hu et al.\\ 2022). Combining LoRA with federated PEFT introduces challenges absent from either setting alone: clients may use different LoRA ranks, making their factor matrices dimension-incompatible, and factor-wise averaging suffers from a bilinear mismatch. We propose SeFoRA, a sketch-aggregated federated LoRA algorithm in which each client transmits a linear sketch of its local updates, enabling direct aggregation at the federator. As a result, SeFoRA alleviates the bilinear mismatch, and allows for aggregation in a small subspace of the full model. We introduce a rank-homogeneous version called SeFoRA-Ho which allows for direct adapter aggregation in this setting. We prove convergence to a neighborhood of the first-order stationary point at rate $\\cO(1/T)$ for the rank-homogeneous setting. Numerical experiments on fine-tuning RoBERTa-Large on GLUE datasets show how our algorithms outperform the state-of-the-art.", "url": "https://wpnews.pro/news/sefora-sketch-aggregated-federated-low-rank-adaptation-with-heterogeneous-client", "canonical_source": "https://arxiv.org/abs/2608.10144", "published_at": "2026-08-12 04:00:00+00:00", "updated_at": "2026-08-12 04:13:41.418560+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "large-language-models"], "entities": ["SeFoRA", "SeFoRA-Ho", "RoBERTa-Large", "GLUE", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/sefora-sketch-aggregated-federated-low-rank-adaptation-with-heterogeneous-client", "markdown": "https://wpnews.pro/news/sefora-sketch-aggregated-federated-low-rank-adaptation-with-heterogeneous-client.md", "text": "https://wpnews.pro/news/sefora-sketch-aggregated-federated-low-rank-adaptation-with-heterogeneous-client.txt", "jsonld": "https://wpnews.pro/news/sefora-sketch-aggregated-federated-low-rank-adaptation-with-heterogeneous-client.jsonld"}}