Is the q_proj/v_proj default worth revisiting? Measurements across four models

A new measurement study across four models finds that the default LoRA target_modules of q_proj and v_proj underperforms a data-driven placement for capability retention: on Llama-3-8B fine-tuning code to prose, HumanEval retention rises from 54.1% with the default to 75.6% with the measured placement across ten paired seeds, with the selected set containing no v_proj at all. The author, who shared code and per-seed data on GitHub and a paper on Zenodo, notes the type ordering reproduces across Llama-3-8B, Mistral-7B, Qwen2.5-7B, and TinyLlama-1.1B, but the depth pattern does not, and the implementation is AGPL-licensed.

The target modules default that most tutorials and configs use, query and value projections, comes from the original LoRA paper. I have been measuring how well it holds when the objective is retaining a previous capability rather than accuracy on the new task, and I wanted to leave the numbers here in case they are useful to anyone thinking about defaults. The measurement compares, before any training, the rank-r gradient subspaces of two tasks module by module, and selects where they overlap least. On Llama-3-8B fine-tuning code to prose, over ten paired seeds with the same number of adapted modules in both arms, HumanEval retention goes from 54.1% with the q,v default to 75.6% with the measured placement, 9 of 10 seeds. The selected set contains no v proj at all, which is half of the default. Three things I would rather state myself: The type ordering reproduces across Llama-3-8B, Mistral-7B, Qwen2.5-7B and TinyLlama-1.1B, with o proj first and v proj last or absent. The depth pattern does not: two of those four concentrate in deep layers and two do not, so a fixed rule cannot know which case it is in. A cheap rule over module type and layer depth ranks well AUROC 0.83 to 0.92 but selects a different set, about a third of the modules differ. Whether that changes retention is a run in progress, pre-registered, and I will post it either way. One honest caveat: the implementation is AGPL, so it is not something you could pull into an Apache-2.0 project. I am not proposing an integration, just leaving the numbers. Anyone can reproduce the measurement on their own model in a free Colab, which prints a target modules list ready to paste into LoraConfig : Code and per-seed data: https://github.com/BiomeMakers/TIM-OmegaS https://github.com/BiomeMakers/TIM%EE%80%80-OmegaS Paper: https://doi.org/10.5281/zenodo.21999659 https://doi.org/10.5281/z%EE%80%80enodo.21999659