an attempt at hypernetworks on an RX 6600 A developer built a 36-million-parameter hypernetwork-based language model on a single AMD RX 6600 GPU, in which a small factory network generates per-prompt attention and MLP weights for each of eight depth iterations instead of relying on a large static model. The design combines a single base layer with generated delta layers, head-wise weight generation, a trainable depth gate, a byte-level morph-patcher for typos and rare words, and a compact vocabulary with prefix fallback, while external knowledge is offloaded to Wikipedia lookups at runtime. Test runs show the model emitting structured tool calls for arithmetic and web search, though the writeup offers no benchmark numbers. I realized that my hardware couldn't run anything decent, so I decided to play it safe—buy proper components? Of course not—build my own architecture Full of confidence, I sat down at my desk and started hacking away. In short, the idea was this: train a small model that takes our prompt as input and prints a first layer tailored to it , runs the input through it, discards the layer, and prints the second one. In my crazy head, this was supposed to make the neural network more "suited" to the query after all, it's generated specifically for it , and most importantly—drastically reduce the need for tens of gigabytes of fast memory. To compensate for the difficulty of transferring general knowledge from the "printer model" to the layers, I chose an exquisite approach—I threw knowledge out entirely and plugged Wikipedia into the runtime. The strangest thing is that the idea actually seems to have worked. In practice, several more decisions were made—such as taking one base layer and generating a delta layer, then summing the hidden states , generating weights head-wise each attention and MLP head gets its own matrices , adding a trainable depth gate sigmoid depth scale l , so that each iteration enters the residual with its own weight , zeroing out W o and W out at initialization so the generated layer starts as a no-op, plus a byte morph-patcher for rare words and typos, and a compact vocabulary with prefix fallback. And more. Ultimately, the idea took the following shape: CanonicalHeadwiseFactory . A small network that generates weights for the next layer based on the current hidden states and iteration number: AttentivePooler — attention pooling over hidden states → context vector; proj in ; depth emb l — iteration index embedding; GRUCell , which carries state across iterations; W q, W k, W v, W o d head × d head for attention and W in d head × gen mlp , W out gen mlp × d head for MLP. CanonicalGeneratedBlock . The same structure as a Qwen2 layer pre-norm → attention → residual → pre-norm → MLP → residual , but the attention and MLP weights come from the outside, via the factory. depth iterations h = embed tokens + 0.1 morph bytes g = g init for l in 0..depth-1: g, W attn, W mlp = factory h, g, l base out = base layer h gen out = gen block h, W attn, W mlp h = h prev + σ depth scale l · base out + gen out − h prev logits = lm head norm h < anchor len , not the response. This eliminates future leakage during training, makes training mode equivalent to generation, and allows KV-cache reuse within a segment.