HyperRepair: Recovering Forgotten General Capabilities of Fine-Tuned Language Models

A framework called HyperRepair recovers 93.4% of base language models' average general benchmark score while retaining 99.5% of fine-tuned models' average domain score, according to an arXiv paper (2610.41147v1). HyperRepair learns a hypernetwork from repairs trained on source models and uses activation fingerprints to generate a prompt-conditioned Repairing Side Net that applies corrections selectively, enabling recovery on models from unseen adaptation domains without target-domain examples or further optimization. The authors evaluated HyperRepair across two model families, 12 LoRA adaptation datasets and 15 general benchmarks, addressing the trade-off between general capability and domain specialization at inference-only cost.

arXiv:2610.41147v1 Announce Type: new Abstract: Fine-tuning improves a language model's performance in a target domain but can weaken its general capabilities. Existing methods often face a trade-off between general capabilities and domain specialization, making it difficult to preserve both. We introduce HyperRepair, a framework for recovering general capabilities while retaining domain specialization at only inference cost. To make recovery generalizable across models sharing a pretrained backbone, HyperRepair learns a hypernetwork from repairs trained on source models. The hypernetwork uses activation fingerprints that summarize changes from the pretrained backbone to generate a tailored Repairing Side Net RSN , whose prompt-conditioned gate applies corrections selectively. This enables recovery on models from unseen adaptation domains without target-domain examples or additional optimization. We evaluate HyperRpair across two model families, 12 LoRA adaptation datasets, and 15 general benchmarks. On the default split, HyperRepair achieves 93.4% of the base models' average general score while retaining 99.5% of the fine-tuned models' average domain score.