{"slug": "rag-retrieves-fine-tuning-forgets-hypernetworks-inject-and-now-we-have-the-laws", "title": "RAG Retrieves. Fine-Tuning Forgets. HyperNetworks Inject - and Now We Have the Scaling Laws.", "summary": "Researchers from Nace AI and Purdue University have introduced HyperNetwork-based knowledge injection as a third path for integrating proprietary data into LLMs, backed by the first systematic scaling laws. The approach trains a secondary network to generate LoRA adapters from facts, leaving the base model frozen, and achieves predictable scaling with out-of-distribution generalization. The team built MegaWikiQA, a dataset of 10M+ multi-hop QA pairs, and demonstrated that hypernetworks outperform RAG and fine-tuning in OOD scenarios.", "body_md": "There are two mainstream answers to \"how do I get my LLM to reliably use proprietary data?\" RAG retrieves relevant chunks at inference time. Fine-tuning bakes new knowledge into the weights. Both are widely deployed. Both have well-documented failure modes.\n\nA new paper from Nace AI and Purdue University proposes a third path backed by the first systematic scaling laws: **HyperNetwork-based knowledge injection**. Instead of modifying the LLM, you train a second network that generates LoRA adapters from a batch of facts. The base model never changes.\n\nLoRA reduces cost but shares the same structural problems. You still forget.\n\nA **HyperNetwork** is a secondary model conditioned on input facts. It generates LoRA adapters (ΔW) inserted into the target model at inference time. The base model is frozen. Always.\n\nArchitecture: Fact batch enters a Transformer encoder (no causal mask, Post-LayerNorm, RoPE) -> Mean pooling -> Linear -> ΔW (LoRA rank 4, α=8) -> adapters into upper half of target model layers. Only the HyperNetwork trains.\n\nThe team built **MegaWikiQA** -- 10M+ multi-hop QA pairs from Wikidata5M (4.6M entities, 822 relations, 39 domains, 1-4 hops, explicit OOD splits).\n\nHypernetworks from 167M to 2.8B parameters. Four findings:\n\nMost enterprise teams face this: proprietary knowledge that can't go into pretraining, too dynamic for static fine-tuning, too specific for general RAG.\n\nThe hypernetwork approach: no base model modification, predictable scaling, OOD generalization to new entities and regulations, independently upgradable layers.\n\nRAG retrieves. Fine-tuning forgets. HyperNetworks inject -- learning to generate weight adaptations from facts, not memorizing facts themselves. The scaling laws make this a design principle: steeper OOD scaling exponents, gap widening at scale, smooth power laws you can engineer around.\n\nPaper: [https://arxiv.org/abs/2607.19604](https://arxiv.org/abs/2607.19604) | Code and data: [https://huggingface.co/collections/nace-ai/hypernetwork-datasets](https://huggingface.co/collections/nace-ai/hypernetwork-datasets)\n\n*Originally published at vsebude.it*", "url": "https://wpnews.pro/news/rag-retrieves-fine-tuning-forgets-hypernetworks-inject-and-now-we-have-the-laws", "canonical_source": "https://dev.to/vsbd_vlad/rag-retrieves-fine-tuning-forgets-hypernetworks-inject-and-now-we-have-the-scaling-laws-3hmm", "published_at": "2026-07-28 10:47:30+00:00", "updated_at": "2026-07-28 11:03:25.099163+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["Nace AI", "Purdue University", "MegaWikiQA", "Wikidata5M"], "alternates": {"html": "https://wpnews.pro/news/rag-retrieves-fine-tuning-forgets-hypernetworks-inject-and-now-we-have-the-laws", "markdown": "https://wpnews.pro/news/rag-retrieves-fine-tuning-forgets-hypernetworks-inject-and-now-we-have-the-laws.md", "text": "https://wpnews.pro/news/rag-retrieves-fine-tuning-forgets-hypernetworks-inject-and-now-we-have-the-laws.txt", "jsonld": "https://wpnews.pro/news/rag-retrieves-fine-tuning-forgets-hypernetworks-inject-and-now-we-have-the-laws.jsonld"}}