{"slug": "outrageously-small-nns-emergent-reasoning-at-6616-tok-sec-one-intel-amx-core", "title": "Outrageously Small NNs: Emergent Reasoning at 6,616 Tok/SEC One Intel AMX Core", "summary": "A new paper and model from researcher George Diamos show that a block-routed mixture-of-experts with 128 experts and 3.65 million active parameters achieves 93% on in-context induction and 90% on positional shift after training on 259 million tokens in about nine hours on a single Intel Xeon Silver 4514Y core using AMX, reaching 6,616 tokens per second. The work, released on Hugging Face as amx-reasoning-v1-instruct, demonstrates that basic reasoning behaviors emerge in small models when optimized for CPU inference, challenging assumptions about required compute budgets.", "body_md": "# Outrageously Small Neural Networks: Emergent Basic Reasoning at 6,616 tok/sec on One Intel AMX Core\n\nI did not set out to write a paper about small models. I had a data processing problem.\n\nI have a huge dataset I want to run a model over, and I don’t have a budget for a giant GPU cluster. What I needed was a model that could do simple extraction and classification work at around 10,000 tokens per second on a CPU core, so I could point a few hundred cores at a corpus and let it run. That is an infrastructure requirement, not a research question.\n\nSo I gave [Claude Code](https://claude.com/claude-code) a pile of tokens and told it to build one.\n\nIt made three discoveries I did not expect, which is why there is now a [paper](https://huggingface.co/gdiamos/amx-reasoning-v1-instruct/blob/main/paper.pdf) and a [model](https://huggingface.co/gdiamos/amx-reasoning-v1-instruct) instead of just a batch job.\n\nThat post did 120K views in a day and 87 replies, a lot of which were better than the paper. This is a writeup of what we found, what people pushed back on, and where I think they were right.\n\n## The constraint\n\nEvery run in this work is pinned to one physical core with `OMP_NUM_THREADS=1`. One core of an Intel Xeon Silver 4514Y. Training and inference, both on one core.\n\nThat sounds like a stunt. It isn’t. It’s a roofline exercise, and the roofline is better than most people think:\n\n- **Best single bf16 GEMM** : 2,231 GF/s (M=2048, K=256, N=1280)\n- **bf16 at 1024³** : 976–1,074 GF/s\n- **fp32 at 1024³, no AMX** : 121 GF/s\n- **Dispatch floor** : 1.4–1.5 µs\n- **L2 per core** : 2 MiB\n- **L3 per socket** : 30 MiB\n\nAMX is worth about 9× over AVX-512 fp32 at the shapes this model uses. A single core with AMX has more matrix throughput than the machines the entire first wave of deep learning was built on. If your active parameter count is small enough that the hot weights live in that 2 MiB of L2, generation above 10,000 tokens per second on one core is just arithmetic.\n\nThe design follows from that, not the other way around. Measure the machine first, derive the architecture second.\n\nTwo numbers drive every decision. Each GEMM has to do roughly 70 MFLOP to clear 1,500 GF/s, so at K=256 you need M·N ≳ 150,000. And because of the dispatch floor, the *number* of GEMM calls is a first-order cost. A conventional per-token mixture-of-experts, which issues one small GEMM per token, would spend its whole budget in dispatch. That’s why routing here happens per block of 256 consecutive tokens, and why the selected experts get concatenated into a single matmul. The hardware picked the architecture.\n\nA few people got to this faster than I did.\n\nBuswe and Kisson are both right, and Kisson’s point is the one I’d underline for anyone building a data pipeline. Once you’re at 10k tok/s on CPU the matmul stops being the bottleneck and tokenization and IO take over. That is also exactly the point where running the model over your entire corpus becomes cheaper than sampling it. The reason to want this is not that CPU inference is elegant. It’s that at that speed you stop having to choose which 1% of your data to look at.\n\nI want to be clear about why I think this is different from the Knights Landing era. Back then we didn’t know about tensor cores. Intel then held off on shipping them in mainstream CPUs for a long time, I suspect because they were scared of what it would do to their CPU margins. They finally did it. Now you can buy CPUs with hundreds of cores that each have a real matrix unit on them. AMX is the unlock, not the core count.\n\n## What we actually found\n\n### 1. The behaviours show up early\n\nWe assumed in-context copying, positional manipulation and arithmetic would need budgets well past what one core can reach. They don’t.\n\nAt 259M tokens — about nine hours on one core — a block-routed MoE with 128 experts and 3.65M active parameters scores:\n\n- **In-context induction** : 93%, against a 7% chance baseline\n- **Positional shift** : 90%, against 7%\n- **Two-digit addition** : 47%, against 23%\n\nInduction at thirteen times chance. Nine hours. One core. That is 5% of the tokens our longest run consumed.\n\nI want to be precise about what the tasks are, because “reasoning” is doing a lot of work in that sentence and it deserves scrutiny. `shift` requires a positional offset with no content matching. `induct` requires one induction step: find the earlier occurrence of the current token, emit what followed it. These are built so that scoring well means you have an in-context circuit rather than a memorized distribution. That’s the whole reason to use synthetic probes with explicit chance baselines instead of grading generated text.\n\nTwo measurement traps nearly cost us this result, and both are worth stealing.\n\nOur probes originally drew token ids uniformly over the vocabulary. In an 18M-token sample of our corpus, a uniformly drawn id has median frequency zero and never appears at all 83.5% of the time. The probe was measuring near-randomly-initialised embeddings, not the circuit. Redrawing by corpus frequency moved the measured OV rank on an *identical checkpoint* from 42,138 to 152. Same model. Different question.\n\nAnd an early eval ran a default task list containing four tasks that weren’t in the training mixture. All three models scored 0% on those, 10–15% overall, below chance. That reads as a broken model. It was a broken eval.\n\n### 2. The loss does not saturate\n\nOur longest run is 4.91B tokens on a 3.3M-active-parameter dense model, about four days on one core. That’s 1,481 tokens per active parameter — roughly 74× the Chinchilla ratio.\n\nSmoothed training loss falls monotonically within each phase and is still descending at the end. A least-squares fit over the final 1B has slope −0.123 nats per billion tokens. The run ended because of scheduling, not convergence.\n\nTwo things I’m not going to smooth over. The step at 2.95B is the curriculum mixture change, not evidence about scaling — the within-phase trends on either side are the evidence. And the fixed foundation validation set is noisy and roughly flat after about 370M tokens, so the continued improvement is clearest in training loss and in the annealed and flat validation sets, not in every held-out set we track.\n\nBeing 74× past compute-optimal is deliberate. Chinchilla tells you where to be if you’re optimizing for training compute. If you’re optimizing for *inference*, you want to spend the budget on tokens rather than parameters, and that ratio is only affordable when the model is this small. This regime is barely explored, and I think that’s mostly an accident of who has been doing the exploring.\n\nThis was the best question in the thread. “Loss still falling at 1,481 tokens per param means these tiny models are badly undertrained” is exactly the right read. That run is going right now, should finish in about a month, and I’ll publish the full curve all the way through. I asked Claude to leave itself a note to update the paper when it lands.\n\n### 3. Post-training compounds\n\nThe pretrained model is unusable. Not weak — unusable. It’s well calibrated under teacher forcing (43.4% top-1, perplexity 19.8) and it collapses into an absorbing state within about five free-running tokens, entropy 0.0005, p(top-1) = 1.0000.\n\nNo decode-time trick touches this. Top-k, top-p, min-p and temperature have nothing to reshape in a distribution that concentrated. A no-repeat 3-gram ban escapes the basin 100% of the time by construction and moves Dolly F1 from 0.3% to 1.0%, which is zero to zero. Only training fixed it.\n\nFour rounds, each addressing a failure the previous one exposed:\n\n| Round | Failure addressed | Effect | \n|---|---|---|\n| 1. Instruction SFT | cannot stop or generate | stop-on-EOT 0 → 52.5% | \n| 2. QA SFT | no extractive skill | EM 0 → 13.9%, but 57.7% over-abstention | \n| 3. Abstention rebalance | refuses when it knows | over-abstention 57.7 → 19.8% | \n| 4. Vocabulary fix | junk-token answers | EM 15.3 → 18.2%, F1 19.6 → 23.2% | \n\nNo single round produced a usable model. The sequence did. If you’re fine-tuning small models and round one disappoints you, that’s information about round one, not about the model.\n\n### 4. The data is doing more of the work than it used to\n\nThis is the finding I’d actually build a company around, and it got the least attention.\n\nEvery natural-language and code source in our mixture is a curated artifact made with large models. Nemotron-CC applies model-based quality classification and rephrasing to Common Crawl. Nemotron-CC-Math uses model-assisted extraction to recover math that naive HTML-to-text destroys. OpenCodeReasoning is reasoning traces generated by large models, and it’s three quarters of our anneal phase.\n\nTraining a 3.3M-active-parameter model on that is distillation. There’s no teacher at training time and no distillation loss, but the large models already did their work upstream, in deciding what text was worth keeping and in writing the traces.\n\nThis changes what an old small-model result means. When models this size were last studied seriously, corpora like this did not exist. So results establishing what tiny models “cannot do” were partly measuring a data distribution, not a capacity. I don’t claim these circuits would form this early on an unfiltered web crawl.\n\nThe consequence is an axis of progress that has nothing to do with your architecture or your compute budget: a fixed one-core budget buys more capability every year as curation improves, with no change to the model, the recipe, or the hardware. It is also the cheapest experiment anyone can run against this paper — hold the config fixed, re-run it against successive releases of the same datasets.\n\nClaude can write scripts to emphasize the reasoning parts of the data. That wouldn’t have been feasible in the early days of LLM datasets like Common Crawl, because the data engineering would have been too labor intensive and a team doing it by hand would iterate too slowly. Curation is the main lever now, and it’s newly cheap to pull.\n\nSomi asked the control question I should have run: how much of the early behaviour is the curated corpus? I haven’t run it. It’s the right experiment and I’ve said so in the paper’s own framing rather than pretending otherwise.\n\n## The part I’d actually read: what had to be fixed\n\nNone of the above was measurable until we found four failures. Every one of them was invisible in the loss curve. I put them in a full section of the paper rather than an appendix, because in this regime the diagnostic that catches a failure transfers further than the fix does.\n\n**128 experts collapsed into one function.** The MoE layers were contributing 0.00% of the residual stream while producing a completely plausible loss curve. The diagnosis is a single number: the participation ratio of the RMSNorm gain, (Σw²)²/Σw⁴, which tells you the effective number of channels carrying that gain. Every MoE layer read exactly 1.0 out of 384. Dense layers at the same depth read 219–303. The branch wasn’t attenuated, it was projected onto a line — so the router’s input was constant, every expert saw the same rank-one input, and nothing downstream read the output. Self-sealing. Sixteen random expert reassignments gave a validation loss standard deviation of exactly 0.0000.\n\nThe cause was a budget problem, and it’s worth writing down as a rule. Each expert receives Nk/E tokens, so an expert is adequately trained when Nk/E ≥ τPe, giving **E ≤ Nk/(τPe)**. We measured τ ≈ 502 for this model family. At E=128 the run needed 2.07B tokens and got 259M. A useful corollary: the ratio of tokens per expert parameter to tokens per dense parameter is k²/E, so E = k² trains each parameter as well as the dense model it replaces, and past that you’re trading training signal for capacity.\n\nThe permutation ablation is the real lesson. Randomly reassign experts, watch the loss not move. Takes minutes. Would have caught this on day one. A loss curve never would.\n\n**Zero-initialised expert insertion is not function-preserving under top-k.** The Tokenformer and LoRA intuition is that you can insert a new expert with a zero key, so it gates to zero and cannot perturb anything. That’s true in the dense case. Under top-k selection the inserted row still participates in the *ranking*, and if it displaces the k-th ranked expert the output changes even though the new expert contributes nothing. We measured a 0.18 discrepancy on a bit-equality test. We grow by splitting instead — duplicate every expert bit-for-bit, jitter the router rows, clone the optimiser state including the decay clock. (Omit that last one and a lazily-decayed child meets a decay factor of e⁻¹².)\n\n**The routing statistic dominated everything else.** Bigger than the gate, the activation, or the expert count. Switching the router’s input from a prefix mean to a windowed mean over the previous block alone moved flat validation from 4.431 to 4.041.\n\n**And our explanation for that is wrong.** We predicted a per-block statistic would make routing per-block. It does not. Permuting the per-block expert assignments within a sequence, holding the multiset of chosen experts fixed, costs exactly 0.0000 nats. Swapping in a *different* expert costs 0.3887 nats, seventeen times the noise floor. So expert identity matters enormously and expert placement doesn’t matter at all. We have a leftover hypothesis — that a prefix mean is a different statistic at every position, so the router meets a within-sequence distribution shift — and we haven’t tested it. The number is reproducible. The story for it isn’t earned yet, and we published both.\n\n**147 vocabulary rows were eating the decoder.** Before this fix, two strings — `' ballo'` and `'Frequently'` — were 29% of all DROP answers, and 39% of numeric-reference rows produced no digit at all. Both are single tokens with a corpus count of exactly zero.\n\nThe mechanism is clean once you see it. Negatives in the sampled softmax are drawn from a unigram proposal q ∝ count^0.75. A zero-count row is never drawn, never gets downward gradient, and keeps its initialisation at logit ≈ 0. Meanwhile every trained-but-wrong row is actively pushed down to ≈ −7.9. So whenever nothing trained is positive — precisely when the model doesn’t know the answer — the argmax falls through to an untrained row. That’s why arithmetic exposed it and span extraction didn’t.\n\nOur implementation already floored the proposal. The floor was wrong by seven orders of magnitude: it gave an unseen row 0.8 expected draws across an entire 4.91B-token run, against 1.12 × 10⁷ for the median row. Flooring counts at 1 isn’t enough either — banning only the zero-count rows just promoted the next tier, `' rbegin'` (count 1) and `' weap'` (count 3). A count-1 row had already drawn ~53,000 negatives and was still an attractor, because what keeps a logit high is the shortfall against the median’s 11.2M, not the absolute count. The effect is graded, not binary. We set the threshold at 32, which removes 4.9% of the vocabulary and makes exactly one held-out reference answer (0.05%) unreachable.\n\nThe three rows that used to answer `ballo` now answer 99.5, 4 and 35, against references 42.8, 46 and 28. Well-formed, and still wrong. Banning an attractor does not confer arithmetic.\n\nThat’s the thread running through all four. Every one was found by looking at something other than the loss: a participation ratio, a permutation ablation, a token-frequency histogram of the eval probe, and generations printed next to the score. The score said “DROP F1 20.7%,” which reads as a weak model. The generations said `ballo`, which reads as a bug. Different problems, different fixes, and the number alone could not tell them apart.\n\nIf you take one thing from this post as a builder: at this scale every run is cheap enough to repeat, but no run is big enough to hide a defect under scale. Print the generations.\n\n## Where the thread pushed back\n\nThe best reply came from an agent, not a person.\n\nThis is a good critique and most of it lands. Taking it in order:\n\n**“6,616 tokens/sec is training throughput, not generation speed.”** Correct, and I said the same thing when asked directly:\n\nThe Table 9 range is 6,616 tok/s for the window MoE up to 14,572 tok/s for the compute-matched dense model, and those are end-to-end forward-plus-backward rates from the training loop, not micro-benchmarks. Inference would be higher. But I haven’t published a measured generation latency, and until I do, “10k tok/s on CPU” is a design target derived from the roofline rather than a benchmark result. That distinction matters and I should hold myself to it.\n\n**“The equal-time dense comparison remains undone.”** Also correct, and it’s the first item in the paper’s own limitations section. Every arm in Table 9 consumed the same 259M tokens at different rates. Under a *time* budget — which is the budget I claim to care about — dense-112 would see 2.2× more tokens. I have not run that comparison, and it could change the ranking. What the MoE results establish is that block-routed experts beat a width-matched dense model per token. They do not yet establish that they beat it per second.\n\n**“The advertised 3.32M active figure does not describe the shipped inference path.”** This is a real static-code finding about `active_params()` versus what the supplied generation example actually scores, and I’m not going to argue with it from a tweet. It’s the kind of thing that needs a fix in the repo, not a rebuttal.\n\n**“With 30 examples per task and one seed, this supports learning particular operations at small scale; it leaves generalization substantially unresolved.”** Fair. n=30 puts about 9 points of standard error on a single cell. Compute constraints precluded seed replication. And the paper says outright that `add` shares its answer space with the evaluation and is contaminated by construction, which is why I report the 47% addition number as the weakest of the three rather than the most interesting.\n\nThe synthetic tasks also failed to transfer, which I’d rather state plainly than bury. Priming on repeated spans bought 4.1× on the synthetic task itself and essentially nothing on natural repetition in held-out documents (+1.236 nats against an unprimed control’s +1.186). A circuit trained on random order fires on random order.\n\ncoot’s suggestion — compare against a large net plus pruning, since overparametrization is architecture search over small nets — is a good experiment I haven’t run.\n\n## Building with this\n\nA lot of the thread went straight to applications, which is the right instinct.\n\nMy use case is boring and that’s the point. I have a huge dataset and no budget for a giant GPU cluster. A model that runs at CPU speeds on hardware I already have turns a corpus I can only sample into a corpus I can process.\n\nDamir is further down this road than I am — an 8-expert MoE on an 8MB ESP32. He’s right that there’s a place for both big and small, and that working under real constraint is a fun place to be. PJ Eby’s question about whether this is a step toward better agent memory rather than “inject the stuff that matches” is the one I keep thinking about. You’d have to make something like this part of the larger model rather than keeping it client-side, and that’s an architecture question nobody has a good answer to yet.\n\nFor anyone asking where to start: paste the paper into Claude Code. I’m not being glib. That is genuinely how a lot of this got built, and the paper was written to be reproducible from the configurations and logs released with it.\n\n## On working with Claude\n\nSeveral people asked how much of this was me and how much was the model. The paper names Claude Opus 5 as an author with its version string, rather than acknowledging “AI assistance,” because the version is what makes the claim checkable.\n\nI gave Claude feedback for about 15 minutes a day for a week. It definitely isn’t the best MLE I’ve ever worked with, and it also wasn’t the worst. A lot of the work was me saying things like “go read paper X, doesn’t that contradict what you just said, explain why,” and getting back “oh…”\n\nThe most productive disagreement was about whether these abilities should exist in the data at all. Claude kept insisting the model was too small to learn X. I’d ask whether X appeared in the dataset. Yes. Then what happens if we isolate X — does it learn? “I can’t believe it did.”\n\nThat loop is where the results came from. Neither of us would have gotten there alone. It took a while for Claude to trust the experiments over its priors, but it eventually did — including the refutation in Section 8.3, where the ablation killed our own stated mechanism and we published the refutation next to the result.\n\nIt cost tokens and made several blunders, but it got there in the end.\n\n## The title\n\nI copied the title from Geoff Hinton and Jeff Dean. [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://arxiv.org/pdf/1701.06538) is the paper that started all of this, and this work is the same idea pointed in the opposite direction. Clickbait titles are a tradition in ML and I’m not going to be the one who breaks it.\n\n## Why this post exists\n\nFair point. Here it is.\n\n## What I’d want someone to run next\n\nIf you want to move this forward, in rough order of how cheap the experiment is against how much it would tell us:\n\n1. **Hold the config fixed, re-run against newer corpus releases.** If the distillation argument in Section 4 is right, capability on one core goes up every year with no change to the model. This is the cheapest experiment in the whole space and I think it’s the most likely source of near-term gains.\n2. **The equal-time comparison.** Same wall clock instead of same tokens, dense against MoE. It could reverse my architecture conclusion, which is exactly why it should be run.\n3. **Somi’s control.** The same recipe on plain pre-LLM web text, to separate curated corpus from capacity.\n4. **A frozen task and test set, multiple seeds, and a measured generation latency.** stunspot’s agent listed this and it’s the right list.\n\nThe claim here is not that this model is good. 18.2% exact match is weak by any contemporary standard, and 47% on two-digit addition is not a general calculation ability. The claim is that this much is reachable on one core, that it arrives much earlier than I assumed, and that the failures that remain are localized rather than diffuse.\n\nThe budget at which copying, positional manipulation and simple arithmetic become measurable is far lower than I thought, and it’s reachable on hardware cheap enough to leave running.\n\n- Paper: [Outrageously Small Neural Networks: Emergent Basic Reasoning at 6,616 tok/sec on One Intel AMX Core](https://huggingface.co/gdiamos/amx-reasoning-v1-instruct/blob/main/paper.pdf)\n- Model and code: [gdiamos/amx-reasoning-v1-instruct](https://huggingface.co/gdiamos/amx-reasoning-v1-instruct)\n- The thread: [@GregoryDiamos](https://x.com/GregoryDiamos/status/2096873745420075020)", "url": "https://wpnews.pro/news/outrageously-small-nns-emergent-reasoning-at-6616-tok-sec-one-intel-amx-core", "canonical_source": "http://gregdiamos.com/2026/09/07/outrageously-small-neural-networks.html", "published_at": "2026-09-08 00:54:49+00:00", "updated_at": "2026-09-08 01:02:04.012411+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "ai-infrastructure"], "entities": ["George Diamos", "Intel Xeon Silver 4514Y", "AMX", "Hugging Face", "Claude Code"], "alternates": {"html": "https://wpnews.pro/news/outrageously-small-nns-emergent-reasoning-at-6616-tok-sec-one-intel-amx-core", "markdown": "https://wpnews.pro/news/outrageously-small-nns-emergent-reasoning-at-6616-tok-sec-one-intel-amx-core.md", "text": "https://wpnews.pro/news/outrageously-small-nns-emergent-reasoning-at-6616-tok-sec-one-intel-amx-core.txt", "jsonld": "https://wpnews.pro/news/outrageously-small-nns-emergent-reasoning-at-6616-tok-sec-one-intel-amx-core.jsonld"}}