I realized that my hardware couldn't run anything decent, so I decided to play it safe—buy proper components? Of course not—build my own architecture!
Full of confidence, I sat down at my desk and started hacking away. In short, the idea was this: train a small model that takes our prompt as input and prints a first layer tailored to it, runs the input through it, discards the layer, and prints the second one. In my crazy head, this was supposed to make the neural network more "suited" to the query (after all, it's generated specifically for it), and most importantly—drastically reduce the need for tens of gigabytes of fast memory.
To compensate for the difficulty of transferring general knowledge from the "printer model" to the layers, I chose an exquisite approach—I threw knowledge out entirely and plugged Wikipedia into the runtime. The strangest thing is that the idea actually seems to have worked.
In practice, several more decisions were made—such as taking one base layer and generating a delta layer, then summing the hidden states, generating weights head-wise (each attention and MLP head gets its own matrices), adding a trainable depth gate (sigmoid(depth_scale[l]), so that each iteration enters the residual with its own weight), zeroing out W_o and W_out at initialization so the generated layer starts as a no-op, plus a byte morph-patcher for rare words and typos, and a compact vocabulary with prefix fallback. And more.
Ultimately, the idea took the following shape:
CanonicalHeadwiseFactory). A small network that generates weights for the next layer based on the current hidden states and iteration number:
AttentivePooler — attention pooling over hidden states → context vector;proj_in; depth_emb[l] — iteration index embedding;GRUCell, which carries state across iterations; W_q, W_k, W_v, W_o (d_head × d_head) for attention and W_in (d_head × gen_mlp), W_out (gen_mlp × d_head) for MLP. CanonicalGeneratedBlock). The same structure as a Qwen2 layer (pre-norm → attention → residual → pre-norm → MLP → residual), but the attention and MLP weights come from the outside, via the factory. depth iterations
h = embed(tokens) + 0.1 * morph(bytes)
g = g_init
for l in 0..depth-1:
g, W_attn, W_mlp = factory(h, g, l)
base_out = base_layer(h)
gen_out = gen_block(h, W_attn, W_mlp)
h = h_prev + σ(depth_scale[l]) · (base_out + gen_out − h_prev)
logits = lm_head(norm(h))
< anchor_len), not the response. This eliminates future leakage during training, makes training mode equivalent to generation, and allows KV-cache reuse within a segment.<tool_call>, arguments, expression, etc.) is forcibly included to prevent JSON from breaking at inference time.Embedding(256, 32) → MLP → d_model. Added to the regular embedding with a 0.1 weight. Helps with typos, rare words, and OOVs.
The qwen36m preset in build_neuroset.py (available on GitHub, link below):
depth=8, base_layers=1, d_model=896, n_heads=14 × d_head=64; coverage, 0.999 coverage; max_seq_len=512, batch_size=4 × grad_accum=4; lr=1.5e-5 for the factory, lr_base=3e-6 for the base layer;<tool_call>…</tool_call>.
After many sleepless nights of failure and tormenting my RX 6600, I finally got a working 36-million parameter network, which is what we'll discuss next. Let me brag a little with some test runs:
usr> how much is 47 * 13?
agt> <tool_call>
{"name":"calculate","arguments":{"expression":"47 * 13"}}
</tool_call>
<tool_response>{"content":"611"}</tool_response>
Total: 611.
usr> what is the capital of France?
agt> <tool_call>
{"name":"web_search","arguments":{"query":"capital of France"}}
</tool_call>
<tool_response>{"content":"France is a sovereign country in Western Europe. Its capital is Paris…"}</tool_response>
Per Wiki: France is a sovereign country in Western Europe. Capital — Paris…
usr> hello!
agt> Hello. I'm listening.
The model decides on its own when to call a tool, and synthesizes the answer itself based on the retrieved fact.
usr> What is 47 * 13? agt> Course Unit: Football Festival in the Himalayas
Hello there! I hope you're enjoying your meal and having fun. I'm sure you're going to do great things together! I'm an AI and I can't wait to share with you that. Would you like me to do that?
usr> What is the capital of France? agt> In the realm of the bustling city of Paris, where the city's rich history and diverse culture intertwine the serene and serene landscape of the city, I stand before the grandeur of the bustling metropolis of Lisbon. This bustling metropolis, where the city's rich history and diverse culture intertwined, was born, is celebrated for its stunning landscapes and vibrant culture. As a result, the city's rich history and diverse culture are celebrated for its stun
usr> Hello! agt> In the realm of technology and technology, the concept of "technology" is crucial for understanding the intersection of technology and technology. This concept is crucial in the field of technology and engineering, as it provides a framework for understanding how technology can be used to enhance or modify processes and systems. By examining the interactions between technology and engineering, we can gain insights into how these systems are developed and developed. This understanding can lead to more effective and effective technologies that are both practical and innovative. Moreover, it highlights the importance of interdisciplinary collaboration in address
usr> Сколько будет 47 * 13? agt> Чтобы найти вопрос 90 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 = 15 15 1
usr> Какая столица Франции? agt> Это задача про китайского мира, который вы можете использовать для создания и уважения. Вот несколько шагов, которые могут помочь:1. Установка: Затем выполните следующие шаги:1. Программа на создание типа: Законы искусственного интеллекта, котор
usr> Привет! agt> Как я могу помочь тебе сегодня?Я здесь, чтобы помочь тебе сегодня?Я помогу вам сегодня?Я здесь, чтобы помочь тебе сегодня.Я здесь, чтобы помочь вам сегодня.Пользователь: Я могу помочь вам сегодня. Я могу помочь вам сегодня.Пользователь: Я
As you can see, our architecture performs much better. And in terms of speed:
| Model | Params | Prompt | ms | tok/s | byte/s | Bytes | All 10 runs in ms |
|---|---|---|---|---|---|---|---|
| Veyra-30M-Instruct | 34,611,712 | 22 | 355.2 | 140.75 | 579.87 | 2060 | 3552.5 |
| Tensor-2-40M-Instruct | 40,036,832 | 3 | 787.7 | 63.47 | 224.69 | 1770 | 7877.4 |
| Neuroset assistant8 | 35,849,352 | 11 | 579.2 | 86.33 | 169.20 | 980 | 5791.9 |
We pay for intelligence and low parameter count with speed; however, this is primarily driven by aggressive tokenizer chunking, and it is also compensated for by huge weight compression, allowing one to buy less memory with higher bandwidth.
Each model responds to 6 equivalent tasks **in its native language**, and the results are compared directly. Correctness is evaluated (for arithmetic/capital/person — presence of the fact; for the rest — a coherent response in its language without repetitions).
| Task | ours_36m | TinyStories-Instr | Stentor-30M | Veyra-30M | Tensor-2-40M |
|---|---|---|---|---|---|
| Hello! | ✅ | ❌ | ❌ | ❌ | ⚠️ 50/50 |
| How much is 47 × 13? | ✅ | ❌ | ❌ | ❌ | ❌ |
| What is the capital of France? | ✅ | ❌ | ❌ | ❌ | ❌ |
| Who is Pushkin? | ✅ | ❌ | ❌ | ❌ | ❌ |
| What is gravity? | ✅ | ❌ | ❌ | ❌ | ❌ |
| What is the largest planet? | ❌ | ✅ | ❌ | ❌ | ❌ |
| **TOTAL** | **5/6** | **1/6** | **0/6** | **0/6** | **0.5/6** |
Our 36M model handles **almost all** tasks—including exact arithmetic via tool use and facts from Wikipedia. Competitors of the same size failed on these specific prompts: they fell into repetition loops or produced irrelevant text.
Deterministic evaluation (91 checks; tool responses are fixed to isolate model intelligence from search noise):
| Metric | `assistant8` (36M) | `assistant_big6` (129M) |
|---|---|---|
| Query formulation (query) | 27/39 | 29/39 |
| Fact marker in final answer (answer) | 22/39 | 35/39* |
| Arithmetic (calculate) | 7/8 | 8/8 |
| Calibration (do not call tools for chitchat) | 5/5 | 5/5 |
| Synthesis faithfulness (faith) | 0.42 | 0.75 |
`runs/assistant_big6` (128.99M: base layer from Qwen2.5-1.5B, `d_model=1536`, vocab 32,768) is an example showing that even with a tiny amount of training data, the architecture can already produce coherent speech. It was trained for ~45 hours on a single RX 6600 on the same data mix (~60M tokens; open-source models of this size are typically trained on trillions).
usr> tell me about Mars agt> Per Wiki: Mars is the fourth planet from the Sun in the Solar System… usr> what is gravity? agt> From Wiki: Gravity is a universal fundamental interaction… usr> hello! agt> Doing well, thanks! What's the question?
This is not a head-to-head comparison with competitors, but rather a demonstration that the architecture **scales and remains conversational** at ridiculously low compute costs.
python build_neuroset.py --preset qwen36m --data-dir dataset1 --out runs/base
--hours 5 --max-seq-len 768 --lr 2e-4 --lr-base 4e-5 --warmup 400 --yes
python build_neuroset.py --preset qwen36m --data-dir dataset2 --out runs/base2
--init-ckpt runs/base/checkpoints/best.pt
--vocab-pack runs/base/compact_vocab.pt
--hours 1.5 --max-seq-len 768 --lr 6e-5 --lr-base 1.5e-5 --warmup 120 --yes
python build_neuroset.py --preset qwen36m --data-dir dataset3 --out runs/assistant8
--init-ckpt runs/base2/checkpoints/best.pt
--vocab-pack runs/base2/compact_vocab.pt
--max-tokens 40000000 --hours 1 --max-seq-len 768
--lr 4e-5 --lr-base 1e-5 --warmup 200 --yes
The model turned out surprisingly capable—honestly, I didn't expect this, as it was built (vibe-coded in parallel with other projects) and slapped together on the fly.
This is my first article on Habr, so I'd appreciate any feedback on the presentation and methodology. I understand that the benchmarks here serve as a sanity check rather than a systematic evaluation: the sample size is small, and competitor models were tested without prompt tuning. More comprehensive metrics can be found in the project's README—I didn't include them in the article to keep it concise.
For those interested in exploring the code, digging deeper, running it locally, or maybe even training it—welcome to GitHub: [https://github.com/RUD9/Grimoire](https://github.com/RUD9/Grimoire).