37% on ARC-AGI 2 without an LLM An autoregressive transformer trained only on ARC puzzles, with no LLM or text data, scored 37.08 on the Kaggle public leaderboard for ARC-AGI-2's semi-private tasks, according to the third iteration of the mdlARC project by Vakde. The 100M-parameter looped model, submission 57908007, was submitted before the deadline but scored after the leaderboard froze, so the official board lists the previous looped model at 35.69 (rank 141); the author reports it peaked at 6th place and beats NVARC's 27.64% from a fine-tuned 4B Qwen. On ARC-1 Public Eval, a 19M-parameter looped mdlARC reached 70.25% trained from scratch on 1307 public puzzles, with looping alone adding 15–19 points and the 70% run costing 3.7 H200 hours (about $13). This is the third version of mdlARC, a series where I try to find out how far an autoregressive transformer can get on ARC when only trained on ARC puzzles -- no text, no LLMs Vakde, 2025 ref-vakde25 , Vakde, 2026 ref-vakde26 . - Likely one of the highest scores without an LLM. 3x higher than next best among all 746 ARC Prize 2026 writeups37.08 is the Kaggle public score of my final submission 57908007 ; Kaggle’s public leaderboard is scored on ARC-AGI-2’s semi-private tasks. The submission was made before the deadline but scored after the leaderboard froze, so the official leaderboard shows my previous looped model at 35.69, rank 141 screenshot fig-shot . - Likely the smallest high-scoring submission. 100M parameters. Top submissions all seem to be Qwen/Deepseek LLMs with 4B, 27B or more - Peaked at 6th place on this leaderboard, before the late surge of Qwen entries. - It beats last year’s winning LLM. NVARC fine-tuned a 4B Qwen model on generated puzzles, and scored 27.64% Sorokin and Puget, 2025 ref-nvarc . - Room for improvement. Final model is severely undertrained and unoptimised. I added looping in the last week of the competition and took the score from 33% → 35% → 37% What's new and different? - Looping gives extraordinary gains - Different from HRM/TRM -- see below vs-trm - Different from HRM/TRM -- - Demo pairs in context - Hub and Spoke Attention - Truly tiny models - We don't hide trained parameters like HRM/TRM' 430M+ parameters - Eg: An actual 1M looped model beats my old 76M one Why do it without an LLM? Because I care about sample efficiency, and LLMs are trained on so much data, including ARC puzzles , that it can't be measured. It is now OBVIOUS that with enough training data, ARC-1/2 becomes trivial to solve. OTOH it is still super interesting to see how well a model can perform when the data is restricted to ARC puzzles, especially very few of them. To test ideas for sample efficiency, I am launching a new Benchmark, ARC Data Squeeze arc-data-squeeze . More details below Offline results on ARC-AGI Public Eval I used ARC-1 Public Eval to test ideas, training from scratch only on the 1307 public puzzles Looped mdlARC scores 70.25% on ARC-1 -- the current SOTA on this restricted set Just 19M parameters. Also scores 15.14% on ARC-2 Beats my prev results with far fewer parameters: 44% in March 76M and 27.5% last December 28M - Looping is a huge win. A 8.4M model goes from 47.44 to 66.25% when its middle four layers run six times. At the same depth, the looped model beats stacks of unique layers with up to 4× more parameters, by 2.3 to 7.6 points. - Truly sub-1M models on ARC. A 1M model scores 49.00%, above my 76M model from March 44 This measures all the progress since March, not just the loop. The loop alone is measured in Looping alone adds 15–19 points looping-alone . - Still cheap to train: The 70% score requires 3.7 H200 hours -- about $13 Full results are in appendix B b-squeeze . The biggest surprise: all of this with almost no hand-written knowledge. No object detectors, no program search, no generated data. ARC is a remarkably well-made dataset. New to ARC? Click here for a quick intro to how it works. The Abstraction and Reasoning Corpus Chollet, 2019 ref-chollet19 is a test of abstract reasoning. Each task shows a few demonstration pairs: an input grid and the output grid it should become. All the pairs follow one hidden rule, and you must produce the exact output for a new test input in at most two attempts. Every task uses a different rule, so solving them means learning concepts like objects, counting and symmetry, and combining them into new ones. It’s fascinating to watch a transformer learn that from so little data. Try one: ARC-AGI-1 has 400 public training and 400 public evaluation tasks. ARC-AGI-2 Chollet et al., 2025 ref-arcagi2 is harder: 1,000 training and 120 public evaluation tasks, and the competition is scored on hidden ones, with 12 hours on four L4 GPUs and no internet. Most entries start from a pretrained LLM Akyürek et al., 2024 ref-akyurek24 ; Franzen et al., 2024 ref-architects24 ; Sorokin and Puget, 2025 ref-nvarc ; mdlARC starts from random weights and learns only from ARC grids. Its name comes from the minimum description length principle MDL; Rissanen, 1978 ref-rissanen78 ; Grünwald, 2007 ref-grunwald07 : the model that explains the data in the fewest bits, counting its own size, should generalise best. How it works mdlARC is an ordinary autoregressive transformer, the same kind of model as GPT. It reads the demo pairs and the test input, then writes the test output one cell at a time. The loop The looped model Figure 1 fig-arch , at the top has 8 transformer blocks. Blocks 1–2 run once, blocks 3–6 run six times in a row, and blocks 7–8 run once. That gives 28 block passes, or 28 layers of depth, from only 8 blocks of weights. Two details help a little. During training, the first five passes also try to exit early through blocks 7–8 and are scored deep supervision; Lee et al., 2014 ref-dsn . And every block gets a direct copy of the input token embeddings, so the loop never loses track of what it is looking at. A plain loop without either still does most of the work appendix B.4 b4-what-the-loop-needs . Hub-and-spoke attention Long tasks are a problem. Ten 30×30 demo pairs make a sequence of about 20,000 tokens, and full attention costs grow with the square of that. On Kaggle’s L4 GPUs, fine-tuning on such tasks would not fit. The fix comes from how a person solves a task: you compare each demo with the test input, but you rarely compare demo 2 with demo 5. In hub-and-spoke attention, each demo pair attends only to itself and to the test input, the hub. The test tokens attend to themselves and, through a second attention with its own softmax, to all the demos. A learned gate in each head mixes the two. The demos share information only through the hub, and the cost grows linearly with the number of demos. An ARC task as a sequence Each grid is written row by row, with one token per cell. The vocabulary has just 14 tokens: the 10 colours, plus start, new row, a separator between input and output, and end. A whole task becomes one sequence: the demo pairs, then the test input. The model writes the test output token by token and decides for itself when to stop, so the output size is never fixed in advance. Positions use rotary embeddings Su et al., 2021 ref-rope in three dimensions: row, column and which grid a token belongs to. Because the input and output grids share row and column coordinates, a cell in the output can find the matching cell in the input directly. The training loss covers only the output tokens. The full recipe behind the 37% submission its data, pretraining, fine-tuning inside Kaggle’s 12 hours, and answering is in appendix A a-submission . What’s new Weight sharing across depth is an old idea. Universal Transformers loop a whole transformer Dehghani et al., 2018 ref-ut , ALBERT shares weights across all layers Lan et al., 2019 ref-albert , and recent work loops language models to spend more compute at test time Geiping et al., 2025 ref-geiping25 or to reason in latent space Saunshi et al., 2025 ref-saunshi25 . On ARC, HRM Wang et al., 2025 ref-hrm and TRM Jolicoeur-Martineau, 2025 ref-trm recursively refine an answer, and LoopViT Shu et al., 2026 ref-loopvit and Loop-OWM Gao et al., 2026 ref-loopowm loop vision models. What’s different here: - Where the loop sits. One loop in the middle of an ordinary autoregressive decoder, running once per token. HRM and TRM keep a draft of the whole answer grid and rewrite it; my model never goes back to change a cell it has written. - No per-task parameters. Every rule comes from the demos in context, and the parameter count is the whole model. - A controlled test of the loop. Loop against no loop at a fixed size, and loop against extra unique layers at equal depth. - Hub-and-spoke attention, which makes attention cost linear in the number of demos. - Scale without language. The same recipe runs from 1M to 100M parameters, from 1,307 public tasks to a full Kaggle submission. How is this different from TRM? - No hidden parameters. No per-task embeddings, so the count is the whole model. TRM’s 7M network also learns about 430M per-puzzle embedding values. HRM and TRM give every augmented training puzzle its own learned 512-dimensional embedding. For TRM that is about 876,000 puzzle IDs, so 430–450M values, counted outside the 7M TRM issue 18 https://github.com/SamsungSAILMontreal/TinyRecursiveModels/issues/18 . HRM’s checkpoint holds a table of about 1.05M × 512 HRM issue 67 https://github.com/sapientinc/HRM/issues/67 . This also TRM reports about 3 days on 4 H100s. - The looping is different. TRM iterates over the whole answer and spits it out once. mdlARC is still autoregressive so its just weight tying. Results in detail 37% on ARC-AGI-2 without an LLM For months, my unlooped models, up to 1B parameters, plateaued at 30–32% on Kaggle the best was 33.89 . Then I added the loop. Both looped models beat every unlooped one on their first submission, with up to 10× fewer parameters: Across all 746 writeups submitted to ARC Prize 2026, the next-best score from a system with no LLM is 12.08, also from a looped transformer trained from scratch. These Kaggle entries differ in more than the loop: data, fine-tuning and decoding changed too. So the clean test below holds everything fixed except the loop. Looping alone adds 15–19 points This test uses the ARC Data Squeeze setup: every model trains from scratch on the same 1,307 public tasks and is scored on the 400 ARC-AGI-1 evaluation tasks. The evaluation tasks’ demo pairs are in training, but never their answers, as in HRM and TRM.This is transductive learning: the model may learn from the unlabelled parts of the test tasks. TRM, HRM and earlier mdlARC versions use the same protocol. No run is ever tuned on the evaluation answers. Take an 8.4M-parameter model with 8 blocks. Without a loop it scores 47.44. Loop its middle four blocks 2, 3, 4 and 6 times and the score climbs to 55.88, 60.13, 61.88 and 66.25 / 77.25. The parameter count never changes. But is it just depth? To check, I also trained unlooped models with more unique blocks, 12, 16, 24 and 32, at the same or greater depth. They have up to 4× more parameters. They lose at every depth, by 2.3 to 7.6 points: The second seed of the ×6 model scores 62.00 / 76.12, so single runs vary by about 4 points. The loop’s gain is far larger than that. The 32-block model is the fairest test: it is deeper, has 4× the parameters and uses more compute, and the looped model still beats it with either seed. How it compares Results on ARC are hard to compare: methods differ in training data, test-time training and how much of the evaluation set they saw while tuning. This table puts the closest work side by side. Scores are percentages on the public evaluation sets, as reported by each paper. | Method | Parameters | Extra data | Test-time training | ARC-AGI-1 | ARC-AGI-2 | |---|---|---|---|---|---| | HRM Wang et al., 2025 ref-hrm | 27M + per-puzzle embeddings | — | — | 40.3 | 5.0 | | TRM Jolicoeur-Martineau, 2025 ref-trm | 7M + ~430M per-puzzle embeddings | — | — | 44.6 | 7.8 | | CompressARC Liao and Gu, 2025 ref-compressarc | 76K | — | each task alone, from scratch | 20 | — | | VARC Hu et al., 2025 ref-varc | 18M | RE-ARC | yes | 54.5 | 8.3 | | LoopViT Shu et al., 2026 ref-loopvit | 18M | RE-ARC | yes | 65.8 | 14.2 | | Loop-OWM Gao et al., 2026 ref-loopowm | 10.6M + ~2M per-task tables | RE-ARC, ARC-GEN | yes | 68.5 | 22.5 | | rekursiv.ai’s improved TRM Kondratyuk and Dillon, 2026 ref-rekursiv | TRM-sized | — | — | 71.4 | 17.5 | | transductARC Schweighofer, 2026 ref-transductarc | ~75M | RE-ARC | — | 76 | ~17 | | Looped mdlARC, 19M | 18.9M | — | — | 70.25 / 79.88 | 15.14 / 20.56 | | Looped mdlARC, 19M + RE-ARC | 18.9M | RE-ARC | — | 81.38 / 88.00 | 20.14 / 32.08 | As reported by TRM; HRM’s paper gives only the ARC-AGI-1 score. ARC Prize re-ran HRM on its semi-private sets and measured 32 and 2 ARC Prize, 2025 ref-arcprizehrm . transductARC’s numbers are means over 8 seeds. Looped mdlARC’s ARC-AGI-2 runs reuse the ARC-AGI-1 recipe unchanged. From official data alone, the 19M looped model reaches 70.25 / 79.88 on ARC-AGI-1. rekursiv.ai’s agent-improved TRM reports a similar 71.4 on ARC-AGI-1 and a higher 17.5 on ARC-AGI-2.rekursiv.ai’s automated research system tuned TRM over 235 runs scored on the public evaluation set, and reports 75.5 on ARC-AGI-1 with a five-model ensemble. Its code is not public. With RE-ARC added, my model reaches 81.38 / 88.00, above transductARC’s 76 with 4× fewer parameters, and above the looped vision models LoopViT and Loop-OWM, which both also fine-tune on each test task. Loop-OWM leads on ARC-AGI-2.The 70.25 / 79.88 record adds ranking by the model’s own likelihood. The Kaggle writeup quotes 71.88 / 81.50 for the same model with scale augmentation and rule-based answer filters borrowed from transductARC; ARC Data Squeeze’s rules don’t allow those, so they count only for its Open table. The RE-ARC runs use them too, so RE-ARC’s own effect is not isolated. Why does looping work? Nobody fully knows why deep learning generalises, so treat this section as my best explanation, with the evidence for it and the results that would prove it wrong. In short: looping buys depth without buying parameters, and on a dataset this small, parameters are what you pay for. The compression view Think of a trained model as a compressed description of its training data. MDL scores a model by two costs added together: the bits needed to describe the model, and the bits needed to describe the data given the model Rissanen, 1978 ref-rissanen78 ; Grünwald, 2007 ref-grunwald07 . A model that memorises does well on the second cost and badly on the first. A model that finds the real rule does well on both, and that is the one that generalises. Neural networks fit this picture too: a network whose weights take fewer bits to describe generalises better Hinton and van Camp, 1993 ref-hinton93 . Depth helps a transformer compute: many ARC rules are step-by-step procedures, like extending a line until it hits something or filling a region. But every unique layer adds parameters, which is description length. A loop gets the depth for free. Twenty-eight layers of computation cost the same as eight layers of weights. ARC’s training data is tiny, about 2M tokens, so the model’s own size is a large part of the total description. That is exactly where saving parameters should matter most. Parameter count is only a rough stand-in for description length, and I don’t measure code length directly. But the view makes predictions, and they can be checked. What the evidence says | If looping works because… | …then we should see | What I measured | |---|---|---| | It saves description length, not just adds compute | Loops beat unique layers at equal depth and compute | They do, by 2.3–7.6 points, with up to 4× fewer parameters and up to 17% fewer FLOPs Figure 9 fig-depth | | Spare capacity lets a model memorise instead of compute | Adding memorisation capacity should shrink the loop’s gain | Per-task embeddings halve it: +16.8 points without them, +8.5 with them | | The loop learns one reusable step | The step should still work when run a different number of times | Trained with 6 loops, the model scores 57.6, 63.5, 66.0, 66.25, 66.0 and 65.25 with 3 to 8 Figure 10 fig-loop | | The steps are the same step, not six different ones | Giving each pass its own identity should not help | Iteration embeddings, per-pass norms and gates were all neutral appendix C.6 c6-loop-variants-that-did-not-help | | Saving parameters matters because data is scarce | With far more data, raw size should start to pay | On 1,307 tasks, doubling the width did not help. On the 160,000-puzzle Kaggle data, the wider model learned faster and led on held-out tasks before fine-tuning weaker evidence: they tied after it | Per-task embeddings are the clearest test. HRM and TRM give each training puzzle its own learned vector. I added the same kind of embedding to my model and measured the loop’s gain with and without it. The gain halved. Interestingly, accuracy with teacher forcing barely changed 42.18% vs 43.25% of outputs exactly right . What got worse is consistency: the model’s answers on rotated and recoloured copies of the same task stopped agreeing, so voting failed more often. A model that can look up a rule for each puzzle has less reason to learn one rule that works on every view. The loop count result says what was learned. An unshared stack of layers can’t even run a seventh copy of its middle. The looped model can, and it keeps its score. That means the four looped blocks learned something like one step of an algorithm, applied over and over, and the answer settles rather than diverges. It’s not perfect. Some tasks get worse past the trained depth the third task in Figure 10 fig-loop “overthinks” , and more loops stop helping after six: ×8 scores 66.5 / 75.25 at this size, and ×11 doesn’t help a 1M model either. It isn’t refining the answer HRM and TRM also loop, but over the whole answer: they keep a draft of the full output grid and rewrite it again and again. ARC Prize’s own analysis of HRM found that its hierarchical design barely mattered, that this outer refinement loop drove much of the gain, and that most of the score came from training on the evaluation tasks themselves rather than from transfer between tasks ARC Prize, 2025 ref-arcprizehrm . My model never goes back to change a cell it has written. Its loop refines the hidden state of the one token it is about to write. Yet the loop still adds 15–19 points. So refining the answer is not needed to get a large gain from looping on ARC: depth with shared weights is enough. And because my model has no puzzle IDs, it can’t memorise a task; it has to read the rule from the demos every time. What this doesn’t explain - Why six. Loop counts past six stopped helping at every size I tried, and I don’t know what sets that number. - How big the effect really is. I ran one seed per variant, and the two seeds of the main model differ by 4.25 points. The unlooped baselines were not tuned separately. - Why the right answer is so often generated but not picked. The oracle is 9–14 points above top-2 in every run. A better way to choose between candidates is the cheapest win left. Beyond ARC, and what’s next Nothing in the loop is specific to ARC. It is a change of a few lines to how an ordinary decoder runs its layers, with no new kind of layer, no special output format and no per-task state. That is why I expect it to carry over to other problems where data is scarce and the rule has to be computed rather than looked up. The evidence so far: - It held across every setting I tried: ARC-AGI-1 and ARC-AGI-2; 1M to 100M parameters; training from scratch on 1,307 tasks, and pretraining on 160,000 generated puzzles followed by test-time fine-tuning. - Recipes moved between benchmarks without retuning. The ARC-AGI-2 Data Squeeze runs reuse the recipe developed on ARC-AGI-1, unchanged. - Other groups see related effects in other domains. Looped and recurrent-depth transformers help language models reason Geiping et al., 2025 ref-geiping25 ; Saunshi et al., 2025 ref-saunshi25 and learn algorithms in context Yang et al., 2023 ref-yang23 . - Hub-and-spoke attention is general too. Any few-shot problem made of independent examples, plus one query, can use it to make attention linear in the number of examples. I haven’t tested the loop outside ARC yet. Sudoku is set up and is the obvious next test. And this is a first version: I started looping in the last week of the competition, so the implementation is rough, and the 100M Kaggle model was undertrained. The best way to loop is still an open question, which is what ARC Data Squeeze is for. ARC Data Squeeze: a new sample-efficiency benchmark I’m turning this setup into a benchmark: ARC Data Squeeze https://github.com/mvakde/arc-data-squeeze . Train from scratch on the 1,307 public tasks ~2M tokens and score as high as you can, with any architecture and any compute. The current SOTA is my looped mdlARC model: just 19M parameters, and 70% on ARC-AGI-1. Can a 1M model reach 70%? Rules, costs and full results are in appendix B b-squeeze and the repo https://github.com/mvakde/arc-data-squeeze . Can we beat GPT-6 Astra by discovering new sample efficiency tricks? Takeaways for ARC Prize - You don’t need an LLM to be competitive on ARC-AGI-2. A 100M model trained only on grids reaches 37%. - Loop your model. It’s a few lines of code, it adds no parameters, and it was the biggest single gain I’ve found. - Keep per-task parameters out. They halved the loop’s gain. - Selection is the bottleneck. The right answer is often among the candidates but not in the top two. - Fitting 12 hours on 4 L4s is an engineering problem. Hub-and-spoke attention, speculative decoding, a placeholder answer file and failure codes in the output size made it work. - Everything is open. Code v3.1 release https://github.com/mvakde/mdlARC/releases/tag/v3.1-nov2026 , MIT-0 , the 99.8M weights, the Kaggle notebook, the 4,013 generators and the puzzles they made. Links - Code: github.com/mvakde/mdlARC https://github.com/mvakde/mdlARC tag v3.1-nov2026 - ARC Data Squeeze: github.com/mvakde/arc-data-squeeze https://github.com/mvakde/arc-data-squeeze - Weights of the 37.08 model: kaggle.com/models/mithilavakde/mdlarc-loopv3-private https://www.kaggle.com/models/mithilavakde/mdlarc-loopv3-private/pyTorch/d1024-e440 - Kaggle notebook: kaggle.com/code/mithilavakde/440e-1024d https://www.kaggle.com/code/mithilavakde/440e-1024d - ARC Prize 2026 writeup Kaggle : Looped mdlARC https://www.kaggle.com/competitions/arc-prize-2026-paper-track/writeups/looped-mdlarc-autoregressive-transformer-trained - Earlier posts: New Pareto frontier on ARC-AGI https://mvakde.github.io/blog/new-pareto-frontier-arc-agi/ Dec 2025 , 44% on ARC-AGI-1 in 67 cents https://mvakde.github.io/blog/44-on-arc-1/ Mar 2026 Appendix Scores are top-2 / oracle on the public evaluation sets; a single number means no oracle was logged. One seed per row unless stated. A. How the 37% submission was built A.1 Overview Data. The Kaggle model was pretrained on 161,798 puzzles in 3,826 families. They start from the 1,120 public ARC-AGI-2 tasks. Frontier LLMs, working in a coding agent, then wrote two kinds of generator programs, each checked automatically and none reviewed by a person: - Variations. The LLM summarises a task's rule and writes a program that makes new puzzles with the same rule, about 42 per task. This is an upgraded version of RE-ARC Hodel, 2024 ref-rearc . - Compositions. Following BARC Li et al., 2024 ref-barc and NVARC Sorokin and Puget, 2025 ref-nvarc , the LLM combines the rules of two ARC-AGI-2 tasks into a new task and writes its generator. The model learns only from the grids these programs output. It never sees the programs or any text. I couldn't afford to test whether the variations helped. Pretraining. One run of 440 epochs, 14.1 hours on 8 B300 GPUs about $600 . Each epoch shows one puzzle from every family. Every solved pair becomes a target, with a random subset of the other pairs as context. Every sequence is randomly rotated or reflected, and its colours are permuted. Variations, demo subsets and augmentations are cycled in a shuffled, balanced order, which beat independent random sampling in earlier mdlARC runs. The optimiser is NorMuon Li et al., 2025 ref-normuon for the weight matrices and AdamW for the rest. Test-time fine-tuning. On Kaggle, the whole model is fine-tuned for 180 epochs on the demo pairs of the hidden tasks Sun et al., 2020 ref-sun20 ; Akyürek et al., 2024 ref-akyurek24 , split across the four L4s. A time guard shortens fine-tuning if it would run past hour 10. Answering. Each task is answered 40 times, under 40 different rotations, reflections and colour permutations, each time leaving one demo out of the context. The answers are mapped back to the original orientation, identical grids are counted, and the two most common become the two attempts. This voting scheme is standard on ARC Franzen et al., 2024 ref-architects24 . Three extra output heads, trained to predict 2–4 tokens ahead Gloeckle et al., 2024 ref-gloeckle24 , draft tokens that the model then checks. This speculative decoding Stern et al., 2018 ref-stern18 ; Leviathan et al., 2023 ref-leviathan23 only changes speed, never answers. Surviving the rerun. Kaggle reruns your notebook on hidden tasks and returns no logs. So the notebook first writes a valid placeholder answer file, every stage catches its own errors, and failures are encoded in the size of the output file. That is how I learned one data-parallel submission scored 0.00 because fine-tuning hung on all GPUs. A.2 Pretraining data | Part | Families | Instances | |---|---|---| | Public ARC-AGI-2 tasks 1,000 training + 120 evaluation, with answers , each with its LLM-written variations | 1,120 | 48,160 | | LLM-written compositions of two ARC-AGI-2 tasks 1,988 eval–eval, 300 train–train, 418 mixed parents | 2,706 | 113,638 | | Total | 3,826 | 161,798 | No ConceptARC, RE-ARC or NVARC data, and no text or code. Generators were written mainly by GPT Sol-class models in Codex CLI; each passed automated checks and reproduces its puzzles exactly when re-run. No puzzle was reviewed by a human. Whether the ~42 variations per task helped is untested. A.3 Pretraining Data per epoch. - An epoch shows one instance from every puzzle family. A family is an original task, or a composition, with about 42 generated variations. - Every solved pair in that instance becomes a training target. Its context is a random subset of the other pairs: at least half, at most 4. - Variations, demo subsets and augmentations are cycled in shuffled order rather than drawn independently. In earlier mdlARC experiments this balanced order beat independent random sampling. - Each family has about 43 versions, so 440 epochs is roughly 10 passes over the generated data. - Each sequence is augmented with one of 80 transformations: one of 8 rotations or reflections, plus a permutation of the colours that appear in the context. Optimizer. | | NorMuon Li et al., 2025 ref-normuon | AdamW | |---|---|---| | Applied to | Linear weight matrices inside the blocks | Embeddings, output heads including MTP , biases, norms, scalar gates | | Peak learning rate | 0.00693 | 0.00125 | | Other | Momentum 0.95 warmed up from 0.85 , β2 0.95 | | - Weight decay 0.1 on the feed-forward weights and the other matrices, 0.01 on the attention weights and token embeddings; one-dimensional parameters get none. - Gradient clipping 1.0. Seed 42. Schedule. Warmup–stable–decay: linear warmup over the first 0.5% of epochs, then a constant learning rate, then linear decay over the last 20% of epochs to 10⁻⁴ × peak. Two phases. | Phase | Epochs | Start | Peak LR NorMuon / AdamW | Decay | Time | |---|---|---|---|---|---| | 1 | 0 → 270 | Random init | 0.00693 / 0.00125 | Epochs 216 → 270 | 8.7 h | | 2 | 270 → 440 | Phase-1 weights, optimizer state kept | 0.005544 / 0.001 0.8× , re-warmed over 2 epochs | Epochs 352 → 440 | 5.4 h | Phase 1 decayed its learning rate to near zero before phase 2 resumed, so phase 2 re-warms an already-converged model rather than continuing a single schedule. It reused the data build from phase 1. Batch, hardware and cost. - Batch 160 8 GPUs × 20 sequences , no gradient accumulation. - 8×B300 with FlashAttention 4, about 1.9 minutes per epoch. - 14.1 hours in total, about 113 B300-hours ~$600 . - A checkpoint was kept every 50 epochs, plus epochs 352 and 440. The final checkpoint is 910 MB. A.4 Test-time fine-tuning - Starts from the epoch-440 checkpoint. All weights are trained, with a fresh optimizer and the same schedule shape: decay starts at epoch 144 of 180. - Every demo pair of every hidden task is a target, with up to 3 of the task's other demos as context. The cap is 3 because the worst-case sequence 7,444 tokens needs 20.68 GiB per L4 under data-parallel training; after a CUDA out-of-memory error the cap drops to 2. - 180 epochs, 80 augmentations. NorMuon 0.0008 / AdamW 0.00014, with weight decay and dropout as in pretraining. - One sequence per step, gradient accumulation 10 on each of the 4 GPUs: effective batch 40. One model is fine-tuned on all hidden tasks, with data-parallel training across the 4 GPUs. - Time guard. Fine-tuning must finish by hour 10 of the notebook. If measured epoch times predict an overrun, the epoch count is cut to fit 90% of the remaining time. A run already under way may overrun by up to 0.3 h; decoding absorbs it. There are up to 4 attempts. - How the recipe was chosen. It was tuned with a development checkpoint trained without the val40 tasks, scoring on val40 and val120 appendix A.7 . The number of optimizer updates mattered more than the number of epochs, and repeats of the same setup varied by several points 14 fine-tunes of one checkpoint: mean 23.4, standard deviation 3.7 top-2 . Full fine-tuning beat LoRA and freezing on val40: 32.50 vs 20.83–22.50 LoRA and 12.50 frozen , top-2. 180 epochs was a judgement call: on heavy100 the d768 e330 model , 90 epochs scored 72.67 / 78.67 and 180 scored 71.33 / 76.33, within that noise. A.5 Decoding - Work split. The tasks are split into four groups balanced by estimated fine-tuning and decoding cost largest task first, into the lightest group . Each GPU decodes its own group, with its own augmentation seed 42, 1, 2, 3 . - Two passes. A quick pass with 8 views, as one shard, so every task has an answer early. Then a full pass with 40 views, in 4 shards per GPU, if its estimate quick-pass time × 40/8 × 1.3 fits before hour 11.3. The full pass's votes replace the quick pass's; a crashed shard loses only its own tasks. - Each view uses one augmentation and a leave-one-out context: n − 1 of the task's n demos, up to 6. Decoding is greedy, with batch 16 and a prefill budget of 32,768 tokens; the multi-token heads draft tokens and the model verifies them. - Voting. Undo each view's augmentation, count identical grids, discard any that copy the test input, and submit the two most common as the two attempts. A.6 Fitting in 12 hours on 4×L4 - A valid placeholder submission.json each test input copied back as its answer is written first, and every stage catches its own errors, so a failure leaves a valid file instead of an error. - FlashAttention is installed from an offline wheel. - Reruns on the hidden set return no logs, so failures are encoded in the size of the output file. - On the public file, the whole run was projected to take about 4.3 hours. - One of the three submissions that fine-tuned with data-parallel training scored 0.00. The size code showed the fine-tune had failed on all GPUs, most likely a hang in GPU-to-GPU communication on the rerun machines. The other two scored, including the 37.08. A.7 Held-out sets for the Kaggle model - val40: 40 ARC-AGI-2 evaluation tasks withheld, with their descendant compositions and exact matches, from a development package 2,546 families, 108,008 instances . Used to tune fine-tuning. - val120: val40 plus 80 held-out compositions. - new80: 80 new compositions of ARC-AGI-2 evaluation tasks, made after the training package was frozen. Their parent tasks are in training, so new80 tests new combinations, not new tasks. The submitted checkpoint scores 22.50 / 30.00 fine-tuning with up to 4 context demos instead of 3 . B. ARC Data Squeeze: rules and full results B.1 Rules in brief ARC Data Squeeze follows the NanoGPT slowrun NanoGPT slowrun, 2026 ref-slowrun , which fixes the data and lets compute vary, but it scores what the model can solve rather than its loss. - Fixed data. Train only on what the benchmark's prepare data.py builds: the 1,307 public tasks. - Answers stay hidden. You may train on the evaluation tasks' demos and use their test inputs, but never their test outputs. - From scratch. No LLMs or pretrained models anywhere in the pipeline. Using them as coding assistants is fine. - Both benchmarks. Train one model on the ARC-AGI-1 data and one on the ARC-AGI-2 data, with the same method. - Limited inductive biases. Allowed: the grid layout positions, which grid is which, hub-and-spoke attention , whole-task rotations, reflections, colour swaps and demo order, and voting or ranking with the model's own likelihood. Not allowed: scaling, translation or cropping, hand-computed objects or counts, rule-based filters on candidate answers, DSLs and program search. - Records need public code at a commit and weights for one seed. Each time an entry passes 90% top-2 on ARC-AGI-1, data is removed: first ConceptARC, then everything but each benchmark's official tasks. Full rules: github.com/mvakde/arc-data-squeeze https://github.com/mvakde/arc-data-squeeze . B.2 The data Human-made public tasks only, deduplicated by task ID 773 IDs are shared between ARC-AGI-1 and ARC-AGI-2; near-duplicates under different IDs were not checked . No generated data and no language models. | | ARC-AGI-1 runs | ARC-AGI-2 runs | |---|---|---| | Training set of the evaluated benchmark | 400 ARC-AGI-1 training tasks 1,718 pairs | 1,000 ARC-AGI-2 training tasks 4,308 pairs | | Other ARC set, not already included | 347 ARC-AGI-2 tasks 1,457 pairs | 27 ARC-AGI-1 tasks 123 pairs | | ConceptARC Moskvichev et al., 2023 ref-conceptarc | 160 tasks 907 pairs | 160 tasks 907 pairs | | Evaluated tasks, demos only | 400 tasks 1,363 pairs; 419 test outputs withheld | 120 tasks 359 pairs; 167 withheld | | Pairs per epoch | 5,445 | 5,697 | The Data Squeeze models train from scratch on this set Moskvichev et al., 2023 ref-conceptarc for ConceptARC for 330 epochs, with no generated data and no test-time fine-tuning, and answer each task with 80 views. The record run also re-ranks the top 20 voted candidates by the model's own likelihood. ARC-AGI-1 public evaluation 400 tasks unless noted. Every model is trained from scratch on the set above for 330 epochs and answers each task with 80 views. The demos of the evaluated tasks are in training, but their test outputs never are, as in mdlARC, HRM and TRM. B.3 Looping vs depth 8.4M looped models, d256 | Depth block passes | Looped, 8.4M | Unlooped, same depth | Unlooped size | |---|---|---|---| | 8 | no loop | 47.44 two seeds: 47.25, 47.63 | 1× | | 12 | ×2: 55.88 | 53.56 | 1.5× | | 16 | ×3: 60.13 | 55.75 | 2× | | 20 vs 24 | ×4: 61.88 / 72.88 | 56.25 | 3× | | 28 vs 32 | ×6 + deep supervision: 66.25 / 77.25 seed 42 , 62.00 / 76.12 seed 43 | 58.63 | 4× | At a fixed size, looping adds 15–19 points. At every depth it beats unique layers with up to 4× fewer parameters, and in the last two rows with fewer passes too 83% and 88% of the FLOPs . No variant, looped or unlooped, was tuned separately. B.4 What the loop needs d256 ×6 | Variant | Top-2 / oracle | |---|---| | Full recipe: deep supervision 0.3 + input injection, seed 42 | 66.25 / 77.25 | | Full recipe, seed 43 | 62.00 / 76.12 | | No deep supervision | 61.75 / 75.13 | | No prelude injection | 64.13 / 76.13 | | Plain loop: no deep supervision, no prelude injection | 62.00 / 75.38 | | No prelude injection and no token-embedding skip | 61.50 / 73.13 | | Deep-supervision weight 1.0 | 64.62 / 75.88 | | Dropout 0.2 | 68.50 / 76.13 | - The two seeds of the full recipe differ by 4.25 top-2 points, so every other gap in this table is within noise. Looping itself is not: the unlooped model scores 47.44. - Loop count saturates at ×6: ×8 scores 66.5 / 75.25 at d256; at d384, ×8 scores 67.88 / 78.50 vs 68.38 / 79.62 for ×6; at 1M, ×11 scores 48.88 / 60.75 vs 49.00 / 60.62. - The loop learns a reusable step. Trained at 6 passes and run with 3–8 passes, d256 scores 57.6 / 63.5 / 66.0 / 66.25 / 66.0 / 65.25 top-2 . - Per-task embeddings halve the loop gain: +16.8 top-2 without them 47.44 → 64.25 at ×4 + deep supervision , +8.5 with them 48.13 → 56.63 . Teacher-forced exact accuracy barely changes 42.18 vs 43.25 , so the loss is in the consistency of votes across views. B.5 Small models | Model | Parameters | ARC-AGI-1 | |---|---|---| | My March 2026 mdlARC, unlooped Vakde, 2026 ref-vakde26 | 76.6M | 44 | | Unlooped, today's recipe | 75.6M | 53.75 | | TRM reported; Jolicoeur-Martineau, 2025 ref-trm | 7M network + ~430M per-puzzle embeddings | 44.6 | | Looped ×6, d80, 4 heads, FFN 256 | 0.71M | 33.38 / 48.12 | | Looped ×6, d96, FFN 256 | 0.90M | 41.25 / 55.50 | | Looped ×6, d96, FFN 296 | 0.99M | 41.50 / 55.88 | | Looped ×6, d96, FFN 320 | 1.05M | 49.00 / 60.62 | | Looped ×6, d144 | 2.0M | 55.63 / 68.00 | | Looped ×6, d128 | 2.1M | 51.25 / 65.25 | | Looped ×6, d256 | 8.4M | 66.25 / 77.25 | | Looped ×6, d384 | 18.9M | 68.38 / 79.62 | | Looped ×6, d384, + likelihood ranking ARC Data Squeeze record | 18.9M | 70.25 / 79.88 | | Looped ×6, d384, + scale augmentation, rule-based filters and likelihood ranking B.8; Open table | 18.9M | 71.88 / 81.50 | Below about 1M parameters the scores fall quickly. B.6 ARC-AGI-2 public evaluation 120 tasks, same protocol | Model | Parameters | ARC-AGI-2 | |---|---|---| | Looped ×6, d384, + likelihood ranking ARC Data Squeeze record | 18.9M | 15.14 / 20.56 | | Looped ×6, d384 | 18.9M | 13.89 / 20.56 | | Looped ×8, d256 | 8.5M | 12.64 / 18.89 | | TRM reported; Jolicoeur-Martineau, 2025 ref-trm | 7M + embeddings | 7.8 | | HRM as reported by TRM | 27M + embeddings | 5.0 | B.7 Beyond the rules: adding RE-ARC With RE-ARC's generated examples of the ARC-AGI-1 training tasks Hodel, 2024 ref-rearc , the 18.9M model reaches 81.38 / 88.00 on ARC-AGI-1 plain votes 78.75 and 20.14 / 32.08 on ARC-AGI-2. transductARC, the closest from-scratch work, reports 76 on ARC-AGI-1 with RE-ARC at about 75M parameters 68 without , and about 17 on ARC-AGI-2 Schweighofer, 2026 ref-transductarc . The RE-ARC runs also turned on scale augmentation, rule-based filters and likelihood ranking, so RE-ARC's own effect is not isolated. B.8 Tricks used only in some runs Three tricks from transductARC that the Kaggle submission does not use: - Scale augmentation: each sequence is upscaled by an integer block factor before the rotation or reflection; decoding mixes scaled views into its 80. - Rule-based filters: checks that reject implausible candidate grids. - Likelihood ranking: the top 20 candidates are re-scored by the model's likelihood. On the d384 ×6 model, likelihood ranking alone takes ARC-AGI-1 from 68.38 / 79.62 to 70.25 / 79.88 the ARC Data Squeeze record . Scale views take it to 71.00 / 81.50, and all three to 71.88 / 81.50. ARC Data Squeeze allows only likelihood ranking, so the other two count for its Open table. B.9 Selection is the bottleneck Across runs the oracle is 9–14 points above top-2: the right answer is often generated but not chosen. B.10 Compute per run Training plus the full evaluation: | Model | ARC-AGI-1 | Compute per run | |---|---|---| | mdlARC v1, Dec 2025 Vakde, 2025 ref-vakde25 | 27.5 | 2.9 A100-hours, $1.80 | | mdlARC v2, Mar 2026 Vakde, 2026 ref-vakde26 | 44 | 2 hours on an RTX 5090, $0.67 | | Looped mdlARC, 19M the record | 70.25 / 79.88 | 3.7 H200-hours, ~$13 | | Looped mdlARC, 1.05M | 49.00 / 60.62 | 3.8 A100-hours, including a 57-minute evaluation | | TRM Jolicoeur-Martineau, 2025 ref-trm | 44.6 reported | about 3 days on 4 H100s reported | The 100M Kaggle model is the exception: its offline pretraining took 14.1 hours on 8 B300 GPUs about $600 . C. Model details C.1 Sizes | Model | Width | Heads | FFN | Blocks / passes | Parameters | |---|---|---|---|---|---| | Kaggle submission | 1024 | 16 | 2688 | 8 / 28 | 99.8M | | Earlier looped Kaggle model | 768 | 12 | 3072 | 8 / 28 | 75.6M | | Data Squeeze models | 80–384 | 4–6 | 256–1536 | 8 / 28 | 0.71M–18.9M | Blocks 1–2 run once, blocks 3–6 run six times, blocks 7–8 run once: 28 block passes from 8 blocks of weights. Parameter counts are trainable parameters. C.2 Tokens and positions - Grids are written row by row with 14 tokens: the 10 colours plus start , newline , separator and end . - The sequence opens with one start token. Each pair then reads input rows, separator , output rows, end . The demo pairs come first, then the test input. The model writes the test output token by token and finishes with end , so the output size is never fixed in advance. - Positions use 3D rotary embeddings Su et al., 2021 ref-rope over row , column and role demo input, demo output, test input, test output . No position records which demo comes first, so the demos form an unordered set. C.3 The transformer block - Pre-norm blocks with RMSNorm, multi-head attention with 3D RoPE, and a SwiGLU feed-forward layer Shazeer, 2020 ref-swiglu . - Each block also gets a learned skip connection from the token embedding, scaled by a scalar that starts at 0. The prelude's normalised output is re-added at the start of loop passes 2–6, scaled by one learned scalar that starts at 0. - Deep supervision Lee et al., 2014 ref-dsn : during training, the states after loop passes 1–5 also run through blocks 7–8 and the output head; the mean of their losses is added with weight 0.3. Only the final pass is used at test time. - The output head is separate from the input embedding untied . There are no task embeddings. - Dropout 0.1 on the residual branches and the embeddings. Attention dropout is 0.1 in the Data Squeeze models and 0 in the Kaggle model, whose FlashAttention 4 kernel has no attention dropout. C.4 Hub-and-spoke attention - Demo tokens attend only within their own pair and to the test input the hub . No cross-demo attention matrix is ever built, so demos share information only through the hub. - Test-pair tokens use two attentions with separate softmaxes: one over the test pair itself the test input bidirectionally, the output causally and one over all the demos. A per-head gate, a sigmoid of a linear map of the hidden state, mixes them. It is initialised to an even mix. - Attention cost grows linearly with the number of demos instead of quadratically. - There is no ablation against full attention. C.5 Multi-token prediction - Three extra linear heads on the final hidden state predict the tokens 2, 3 and 4 positions ahead, with loss weight 0.3, on output tokens only Gloeckle et al., 2024 ref-gloeckle24 . - During training they add a learning signal. During decoding they draft tokens for speculative decoding Stern et al., 2018 ref-stern18 ; Leviathan et al., 2023 ref-leviathan23 , which is exact: it changes speed, never answers. C.6 Loop variants that did not help Dense mixing across passes, iteration embeddings, gates, per-iteration norms and parallel streams were all neutral against the plain loop in controlled tests; pointer-style variants scored 63.0–65.6 vs 66.25; H-Net chunking with a ×6 loop scored 58.5 / 72.63 vs 66.25 / 77.25. A d768 model combining several variants scored higher than its plain counterpart, but that run changed other settings too, so their effect is not isolated. C.7 Changing the loop count after training The loop learns one step it can repeat. Take a trained model and change only how many times the loop runs: Nothing else changes between these frames: same weights, same input. With too few loops the model gets the task wrong. With the six it was trained on, it gets it right. References - Akyürek, E., Damani, M., Zweiger, A., Qiu, L., Guo, H., Pari, J., Kim, Y. and Andreas, J. 2024 . The Surprising Effectiveness of Test-Time Training for Abstract Reasoning https://arxiv.org/abs/2411.07279 . arXiv:2411.07279. - ARC Prize team, analysis led by K. Schürholt 2025 . The Hidden Drivers of HRM’s Performance on ARC-AGI https://arcprize.org/blog/hrm-analysis . ARC Prize blog, Aug 15, 2025. - Chollet, F., Knoop, M., Kamradt, G., Landers, B. and Pinkard, H. 2025 . ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems https://arxiv.org/abs/2505.11831 . arXiv:2505.11831. - Chollet, F. 2019 . On the Measure of Intelligence https://arxiv.org/abs/1911.01547 . arXiv:1911.01547. - Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J. and Kaiser, Ł. 2018 . Universal Transformers https://arxiv.org/abs/1807.03819 . ICLR 2019. - Franzen, D., Disselhoff, J. and Hartmann, D. 2024 . The LLM ARChitect: Solving ARC-AGI Is A Matter of Perspective https://da-fr.github.io/arc-prize-2024/the architects.pdf . ARC Prize 2024 technical report. See also Product of Experts with LLMs https://arxiv.org/abs/2505.07859 2025 , arXiv:2505.07859. - Gao, G., Schölkopf, B. and Geiger, A. 2026 . Slots, Transitions, Loops: Learning Composable World Models for ARC https://arxiv.org/abs/2606.12316 Loop-OWM . arXiv:2606.12316. - Geiping, J. et al. 2025 . Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach https://arxiv.org/abs/2502.05171 . arXiv:2502.05171. - Gloeckle, F. et al. 2024 . Better & Faster Large Language Models via Multi-token Prediction https://arxiv.org/abs/2404.19737 . arXiv:2404.19737. - Grünwald, P. 2007 . The Minimum Description Length Principle https://homepages.cwi.nl/~pdg/book/book.html . MIT Press. - Hinton, G. E. and van Camp, D. 1993 . Keeping the Neural Networks Simple by Minimizing the Description Length of the Weights https://doi.org/10.1145/168304.168306 . COLT ’93. - Hodel, M. 2024 . Addressing the Abstraction and Reasoning Corpus via Procedural Example Generation https://arxiv.org/abs/2404.07353 RE-ARC . arXiv:2404.07353. - Hu, K. et al. 2025 . ARC Is a Vision Problem https://arxiv.org/abs/2511.14761 VARC . arXiv:2511.14761. - Jolicoeur-Martineau, A. 2025 . Less is More: Recursive Reasoning with Tiny Networks https://arxiv.org/abs/2510.04871 TRM . arXiv:2510.04871. - Kondratyuk, D. and Dillon, J. V. 2026 . Pushing the Limits of Autonomous ML Science on ARC-AGI https://rekursiv.ai/blog/pushing-limits-arc-agi/ . rekursiv.ai blog, Jul 16, 2026. - Lan, Z. et al. 2019 . ALBERT: A Lite BERT for Self-supervised Learning of Language Representations https://arxiv.org/abs/1909.11942 . arXiv:1909.11942. - Lee, C.-Y., Xie, S., Gallagher, P., Zhang, Z. and Tu, Z. 2014 . Deeply-Supervised Nets https://arxiv.org/abs/1409.5185 . arXiv:1409.5185. - Leviathan, Y., Kalman, M. and Matias, Y. 2023 . Fast Inference from Transformers via Speculative Decoding https://arxiv.org/abs/2211.17192 . ICML 2023. - Li, W.-D. et al. 2024 . Combining Induction and Transduction for Abstract Reasoning https://arxiv.org/abs/2411.02272 BARC . arXiv:2411.02272. - Li, Z. et al. 2025 . NorMuon: Making Muon More Efficient and Scalable https://arxiv.org/abs/2510.05491 . arXiv:2510.05491. Builds on Muon: Jordan, K. 2024 , Muon: An optimizer for hidden layers in neural networks https://kellerjordan.github.io/posts/muon/ . - Liao, I. and Gu, A. 2025 . ARC-AGI Without Pretraining https://iliao2345.github.io/blog posts/arc agi without pretraining/arc agi without pretraining.html CompressARC . arXiv:2512.06104. - Moskvichev, A., Odouard, V. V. and Mitchell, M. 2023 . The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain https://arxiv.org/abs/2305.07141 . arXiv:2305.07141. - NanoGPT Slowrun https://github.com/qlabs-eng/slowrun 2026 . Fixed 100M training tokens, any compute; lowest validation loss wins. - Rissanen, J. 1978 . Modeling by Shortest Data Description