In early September, someone posted a link to a static site with no branding and a plain-text title: “I trained a small transformer in 1.5hrs and it beats many LLMs.” It hit the front page of Hacker News and stayed there. By the time the thread cooled off it had pulled in 167 comments – the kind of engagement number that usually means either a flame war or a genuine “wait, what?” moment. This was the second one.
The author showed up in the comments under the handle evilmathkid, and the first thing they did was correct the record before anyone else could misread it: “This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs.” That single line is the whole story, compressed. Everyone assumed this was another distillation trick, another small model riding on the coattails of a big one. It wasn’t. It was a from-scratch autoregressive transformer, small enough to train on one GPU in the time it takes to watch two movies, and it landed a result that – until this – only LLMs or their derivatives had touched.
That’s worth sitting with for a second. We’ve written before about the industry’s slow pivot away from parameter-count chest-thumping and toward inference efficiency – smaller “student” models distilled from bigger “teachers,” doing more with less. This is a different animal entirely. Nothing here was distilled from anything. It was built and trained cold, and it still cleared a bar that had previously belonged to models with orders of magnitude more parameters.
It’s also worth noticing why the comment section reacted the way it did. Most “small model punches above its weight” stories in the last two years have followed a predictable shape: take a large model’s outputs, use them as training signal for a smaller model, and call the smaller model “efficient.” That’s a legitimate technique, and it’s produced genuinely useful systems, but it’s also a shape people have gotten used to discounting a little – because the small model is, in a real sense, still leaning on the big one’s shoulders. Somewhere in its training data, however indirectly, is the reasoning of a much larger network. Take that scaffolding away entirely and the assumption is that performance collapses. That’s the assumption this project broke, which is exactly why a comment thread full of people who build models for a living spent 167 replies arguing about it.
What actually got built #
According to the writeup covered by Mango Developer, the model was trained from scratch in 1.5 hours on a single RTX 5090, and it hit 44% on ARC-1 – a score that matches specialized architectures built specifically for this benchmark, referred to in the piece as TRM and HRM. No pretraining corpus scraped from the internet. No fine-tune of an existing checkpoint. Just an architecture, a training loop, and a consumer GPU that most of you reading this could buy today.
Put the training cost in perspective. An RTX 5090 is a card you can order from a retail site, not a rack of A100s or H100s you need a cloud contract and a procurement team to access. Ninety minutes of wall-clock time on that single card is roughly the electricity and depreciation cost of leaving a gaming PC running through a movie. Compare that to the training budgets behind the general-purpose LLMs this result is being measured against – budgets that run into the tens or hundreds of millions of dollars, distributed across thousands of accelerators, over weeks or months. The gap isn’t “this is a bit cheaper.” It’s closer to five or six orders of magnitude in compute, for a task where the small model is competitive rather than merely “not embarrassing.”
ARC-1 is not a benchmark you cheese with vocabulary tricks or clever prompting. It’s part of the ARC-AGI family, built explicitly to resist memorization – every puzzle is novel, abstract, and requires the model to infer a transformation rule from a handful of examples and apply it to something it’s never seen. A typical ARC-1 task hands you two or three small colored grids as “before” and “after” pairs – say, a 5x5 grid where every isolated blue square gets surrounded by a ring of yellow, and every red square stays untouched – and then gives you a fourth “before” grid and asks you to produce the “after.” There’s no vocabulary to memorize and no pattern you could have seen in a training corpus, because the puzzle was generated to be unlike anything that came before it. You either infer the rule from the examples in front of you, on the spot, or you don’t. It’s the benchmark researchers reach for specifically because language models, even huge ones, tend to faceplant on it – a model can have ingested the entire internet and still have no leverage on a grid transformation it’s never encountered, because raw scale doesn’t substitute for the specific skill of rapid rule induction from a tiny number of examples. A YouTube breakdown of the result frames it the same way: a small transformer trained in 1.5 hours hit 44% on ARC-1, outperforming general-purpose LLMs that cost orders of magnitude more to run.
If you’re used to the Hugging Face Model Hub and the Open LLM Leaderboard as your mental map of “what’s good,” this doesn’t show up on either. It’s not competing in that arena. It’s not an LLM entry at all – it’s a specialist, and specialists don’t always play by the leaderboard’s rules. A leaderboard built around benchmarks like MMLU or HellaSwag is measuring breadth: how well a model handles thousands of different task types with a single set of weights. ARC-1 measures something narrower and, in some ways, harder – the ability to generalize to a genuinely new rule from almost no examples. Those are different axes, and a model that’s mediocre on the first can still be excellent on the second, which is exactly the position this small transformer occupies.
The design choices that made it work #
The technical detail that jumped out to us – and the part that separates this from a lucky architecture guess – is the ablation study. Per the Mango Developer breakdown, the approach uses test-time training on task-specific puzzles, combined with 3D RoPE embeddings and per-task learned embeddings. Both of those pieces turned out to be load-bearing. Strip either one out and accuracy drops to roughly 24% – almost half the final score, gone. That’s not a marginal tweak. That’s the difference between “works” and “doesn’t.”
An ablation study, for anyone who hasn’t run one, is exactly what it sounds like: you build the full system, measure it, then remove one piece at a time and remeasure, to find out which pieces were actually doing work versus which were just along for the ride. It’s easy to end up with a pile of design choices that all felt necessary at the time and never find out that half of them were dead weight. The fact that this team ran that process and published the numbers – rather than just publishing the 44% headline number and letting people assume the whole architecture was equally important – is part of why the result is credible rather than just viral.
Test-time training is the part worth explaining if you haven’t run into it. Instead of training once and freezing the weights before inference, the model keeps learning at inference time, adapting to the specific puzzle in front of it using the few examples it’s given. It’s closer to how you’d solve an IQ-test puzzle yourself – you don’t apply a fixed rule you memorized in school, you look at the three examples, infer the pattern, and apply it to the fourth. Baking that into the training loop, rather than bolting it on after, is what let a small model compete with approaches that lean on raw scale.
Concretely, this means the model isn’t just doing a single forward pass over the puzzle at inference time the way a standard LLM would when you paste a prompt into it. It’s taking the handful of example input/output grid pairs that come bundled with each ARC-1 task, running a lightweight training update using those examples as its own miniature dataset, and only then attempting the held-out test grid. Each puzzle effectively gets its own brief, private training run before the model commits to an answer. That’s expensive per-puzzle compared to a single forward pass, but it’s cheap in absolute terms because the base model is small and the puzzle-specific adaptation is short. It’s also a fundamentally different bet than the one general LLMs make: instead of trying to have already seen enough patterns during pretraining to recognize this one by analogy, the model is explicitly re-deriving the rule from scratch, every single time, using only the examples the puzzle itself provides.
3D RoPE – rotary position embeddings extended into a third dimension – and per-task learned embeddings are both about giving the model a better sense of where it is inside a grid-shaped puzzle and which puzzle it’s currently solving. Standard RoPE, the kind used in most modern language models, encodes position along a single sequence axis – token 1, token 2, token 3, and so on down a line of text. That works fine for sentences, because sentences are one-dimensional: word order is really the only spatial relationship that matters. ARC-1 tasks are spatial grids, not linear text, so a position encoding scheme built for sentences doesn’t naturally fit – a cell’s meaning depends on its row, its column, and often its relationship to cells diagonally or across the grid, not just its position in some flattened left-to-right token stream. Extending RoPE into a third dimension lets the model encode row, column, and an additional axis – plausibly something like which example within the task, or a channel for color/value – directly into the position signal, instead of forcing the model to reconstruct 2D or 3D spatial structure indirectly from a 1D encoding the way a plain LLM would have to. Per-task learned embeddings, layered on top, give the model an explicit signal for “this is puzzle A, not puzzle B” when it’s being trained or adapted across many different tasks, so it doesn’t have to re-infer from scratch, purely from the grid content, which family of rules it should even be considering. Adapting the geometry to the actual shape of the problem, rather than forcing text-shaped assumptions onto a grid-shaped task, is the kind of decision that looks obvious in hindsight and invisible until someone actually tries it and measures the gap – and the ablation numbers are the proof that it wasn’t cosmetic. Losing it costs roughly twenty points of accuracy, which on a benchmark this hard is the difference between a model that’s genuinely competitive and one that’s barely functional.
The loss function twist nobody expected #
The detail that should make anyone who’s fine-tuned a model sit up is the loss function finding. Switching from a standard loss to a supervised loss computed only on the output tokens improved accuracy from 40% to 44% – despite the validation loss getting worse in the process. Per the same Mango Developer coverage, this exposes a real failure mode: using validation loss as a stand-in for sample efficiency can straight-up lie to you.
To unpack why this is a real distinction and not just a technicality: a standard autoregressive loss, applied naively, computes error across every token the model produces during training – including tokens that are effectively just scaffolding, formatting, or restating the input before the model gets to the part that actually constitutes the answer. A loss that’s computed only on the output tokens ignores all of that scaffolding and grades the model exclusively on the grid cells it’s actually predicting as the solution. Both losses can go down over the course of training, but they’re rewarding different things – one is partly rewarding “predict the input format correctly,” the other is entirely rewarding “predict the right answer.” It’s not hard to see, once it’s spelled out, why a model optimized purely on the second objective would end up better at the thing you actually care about, even if its aggregate loss number – which still includes all that scaffolding – looks worse by comparison.
That’s a bigger deal than it sounds. Most of the machine learning tooling ecosystem treats val loss as the north star – it’s the number your training dashboard plots, the number early-stopping callbacks watch, the number you eyeball at 2am to decide whether the run is working. Here, the number that looked worse produced the model that actually performed better on the thing you care about. If you’ve ever killed a promising training run early because the loss curve ticked the wrong direction, this is the cautionary tale. The metric you’re watching and the outcome you want are not always the same axis.
It’s worth stating plainly what the practical takeaway is, because it generalizes well beyond this one project: validation loss is a proxy, not the target. It’s a proxy that happens to correlate with the target often enough that it’s become the default thing everyone watches, but “often enough” is not “always,” and this project is a clean, measured example of a case where the correlation breaks down. Anyone iterating on a training loop – not just on ARC-style puzzles, but on any task where the loss is computed over a mix of “answer” tokens and “everything else” tokens – should take this as a prompt to check whether their loss function is actually weighting the tokens they care about, or just averaging over everything and calling it a metric.
The writeup also mentions an optimizer called NorMuon in the mix. According to the NorMuon paper, it’s a neuron-wise normalized variant of the Muon optimizer that pairs matrix-level orthogonalization with adaptive per-neuron scaling, addressing shortcomings in both Adam and standard Muon – the kind of optimizer choice that squeezes more sample efficiency out of a short, compute-constrained training run rather than one built for massive distributed pretraining. The distinction matters in a training run this short: with only 1.5 hours on the clock, there’s no budget for an optimizer that needs thousands of steps to find its footing. Adam is the default choice for most training loops because it’s robust and forgiving, but it’s not necessarily the fastest optimizer to converge per-step on every architecture. Muon’s matrix-level orthogonalization keeps weight updates from collapsing into redundant directions – a common failure mode where different neurons end up learning near-duplicate representations, wasting capacity. NorMuon’s addition of per-neuron adaptive scaling on top of that orthogonalization is aimed at fixing a known weakness of plain Muon, where treating all neurons with the same update scale can shortchange the ones that need larger or smaller steps. In a regime where every training step is precious because there are so few of them, an optimizer that converges in fewer steps isn’t a nice-to-have, it’s close to a prerequisite for the whole approach being feasible at all.
Why “it’s not an LLM” is the actual headline #
It would be easy to read this story as “small model beats big model” and file it next to every other distillation story from the past year. That misses what evilmathkid was pointing at in that HN comment. The interesting claim isn’t that a small model can match a big one on a task – that’s been demonstrated plenty, in distillation papers, in quantization papers, in retrieval-augmented setups where a small model borrows a bigger one’s knowledge through a search index instead of through its own weights. The interesting claim is that you don’t need an LLM’s machinery at all to hit a benchmark that, until now, only LLM-shaped systems had cracked. No transformer trained on internet-scale text, no tokenizer built around vocabulary and grammar, no chain-of-thought prompting borrowed from a much larger sibling model. Just an architecture shaped like the problem, trained directly on the problem, for less time than it takes to fly cross-country.
That distinction matters for anyone deciding what to build next. If your instinct, every time you hit a hard reasoning problem, is to reach for a bigger model or a longer prompt chain, this result is a data point against that reflex. Task-specific architecture, trained cold, with the right position encoding and the right loss function, beat the “throw an LLM at it” approach on its own turf. Not every problem needs a general-purpose language model wrapped around it. Some problems need a purpose-built structure and a training loop that actually understands the shape of the puzzle.
It’s worth being precise about what this does and doesn’t imply, because it would be easy to overreach in the other direction too. This isn’t evidence that LLMs are obsolete, or that general-purpose language ability doesn’t matter – an LLM’s whole value proposition is breadth, the ability to handle an open-ended range of tasks it was never specifically built for, and no purpose-built 44%-on-ARC-1 architecture is going to write an email, summarize a contract, or hold a conversation. What this result narrows down is the specific claim that general-purpose scale is the only route to strong performance on hard reasoning benchmarks. It demonstrates a second route: identify the actual shape of the problem – grid-structured, rule-inducible from a handful of examples – and build the smallest system that fits that shape exactly, rather than the largest system that fits everything approximately.
We’ve made a version of this argument before, when we looked at how fine-tuning existing models often costs more than it saves compared to building something purpose-fit from the start. This project is the sharpest version of that argument we’ve seen yet: not “fine-tuning is worse than building from scratch” in some abstract efficiency sense, but a working system, with published ablations, that took ninety minutes on one consumer GPU to reach a score that specialized ARC architectures and general-purpose LLMs alike had to work much harder to reach. The lesson isn’t “abandon LLMs.” It’s “check whether the problem in front of you actually needs one before you reach for it” – and this is as clean a demonstration of that check paying off as you’re likely to find in a Hacker News thread.