{"slug": "is-the-q-proj-v-proj-default-worth-revisiting-measurements-across-four-models", "title": "Is the q_proj/v_proj default worth revisiting? Measurements across four models", "summary": "A new measurement study across four models finds that the default LoRA target_modules of q_proj and v_proj underperforms a data-driven placement for capability retention: on Llama-3-8B fine-tuning code to prose, HumanEval retention rises from 54.1% with the default to 75.6% with the measured placement across ten paired seeds, with the selected set containing no v_proj at all. The author, who shared code and per-seed data on GitHub and a paper on Zenodo, notes the type ordering reproduces across Llama-3-8B, Mistral-7B, Qwen2.5-7B, and TinyLlama-1.1B, but the depth pattern does not, and the implementation is AGPL-licensed.", "body_md": "The `target_modules`\n\ndefault that most tutorials and configs use, query and value projections, comes from the original LoRA paper. I have been measuring how well it holds when the objective is retaining a previous capability rather than accuracy on the new task, and I wanted to leave the numbers here in case they are useful to anyone thinking about defaults.\n\nThe measurement compares, before any training, the rank-r gradient subspaces of two tasks module by module, and selects where they overlap least. On Llama-3-8B fine-tuning code to prose, over ten paired seeds with the same number of adapted modules in both arms, HumanEval retention goes from 54.1% with the q,v default to 75.6% with the measured placement, 9 of 10 seeds. The selected set contains no `v_proj`\n\nat all, which is half of the default.\n\nThree things I would rather state myself:\n\nThe type ordering reproduces across Llama-3-8B, Mistral-7B, Qwen2.5-7B and TinyLlama-1.1B, with `o_proj`\n\nfirst and `v_proj`\n\nlast or absent. The depth pattern does not: two of those four concentrate in deep layers and two do not, so a fixed rule cannot know which case it is in.\n\nA cheap rule over module type and layer depth ranks well (AUROC 0.83 to 0.92) but selects a different set, about a third of the modules differ. Whether that changes retention is a run in progress, pre-registered, and I will post it either way.\n\nOne honest caveat: the implementation is AGPL, so it is not something you could pull into an Apache-2.0 project. I am not proposing an integration, just leaving the numbers.\n\nAnyone can reproduce the measurement on their own model in a free Colab, which prints a `target_modules`\n\nlist ready to paste into `LoraConfig`\n\n:\n\nCode and per-seed data: [https://github.com/BiomeMakers/TIM-OmegaS](https://github.com/BiomeMakers/TIM%EE%80%80-OmegaS)\n\nPaper: [https://doi.org/10.5281/zenodo.21999659](https://doi.org/10.5281/z%EE%80%80enodo.21999659)", "url": "https://wpnews.pro/news/is-the-q-proj-v-proj-default-worth-revisiting-measurements-across-four-models", "canonical_source": "https://discuss.huggingface.co/t/is-the-q-proj-v-proj-default-worth-revisiting-measurements-across-four-models/178990#post_1", "published_at": "2026-08-20 10:18:18+00:00", "updated_at": "2026-08-20 10:44:58.506138+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research"], "entities": ["Llama-3-8B", "Mistral-7B", "Qwen2.5-7B", "TinyLlama-1.1B", "HumanEval", "LoraConfig", "GitHub", "Zenodo"], "alternates": {"html": "https://wpnews.pro/news/is-the-q-proj-v-proj-default-worth-revisiting-measurements-across-four-models", "markdown": "https://wpnews.pro/news/is-the-q-proj-v-proj-default-worth-revisiting-measurements-across-four-models.md", "text": "https://wpnews.pro/news/is-the-q-proj-v-proj-default-worth-revisiting-measurements-across-four-models.txt", "jsonld": "https://wpnews.pro/news/is-the-q-proj-v-proj-default-worth-revisiting-measurements-across-four-models.jsonld"}}