The target_modules
default that most tutorials and configs use, query and value projections, comes from the original LoRA paper. I have been measuring how well it holds when the objective is retaining a previous capability rather than accuracy on the new task, and I wanted to leave the numbers here in case they are useful to anyone thinking about defaults.
The measurement compares, before any training, the rank-r gradient subspaces of two tasks module by module, and selects where they overlap least. On Llama-3-8B fine-tuning code to prose, over ten paired seeds with the same number of adapted modules in both arms, HumanEval retention goes from 54.1% with the q,v default to 75.6% with the measured placement, 9 of 10 seeds. The selected set contains no v_proj
at all, which is half of the default.
Three things I would rather state myself:
The type ordering reproduces across Llama-3-8B, Mistral-7B, Qwen2.5-7B and TinyLlama-1.1B, with o_proj
first and v_proj
last or absent. The depth pattern does not: two of those four concentrate in deep layers and two do not, so a fixed rule cannot know which case it is in.
A cheap rule over module type and layer depth ranks well (AUROC 0.83 to 0.92) but selects a different set, about a third of the modules differ. Whether that changes retention is a run in progress, pre-registered, and I will post it either way.
One honest caveat: the implementation is AGPL, so it is not something you could pull into an Apache-2.0 project. I am not proposing an integration, just leaving the numbers.
Anyone can reproduce the measurement on their own model in a free Colab, which prints a target_modules
list ready to paste into LoraConfig
:
Code and per-seed data: [https://github.com/BiomeMakers/TIM-OmegaS](https://github.com/BiomeMakers/TIM%EE%80%80-OmegaS)
Paper: [https://doi.org/10.5281/zenodo.21999659](https://doi.org/10.5281/z%EE%80%80enodo.21999659)