37% on ARC-AGI-2 Kaggle Without LLMs

Mithil Vakde reported scoring 37% on the ARC-AGI-2 Kaggle competition with a 100M-parameter looped transformer trained from scratch, without using an LLM, peaking at 6th place and beating last year's winning LLM entry. Vakde said adding looping in the final week lifted the score from 33% to 35% to 37%, and that at equal depth looping beat non-looped stacks with up to 4x the parameters by 2.3-7.6%. Vakde also reported 44% on ARC-AGI-1 for 67 cents, trained in 2 hours on a 5090, matching TRM and beating HRM, and argued the non-LLM approach matters because LLM sample efficiency on ARC cannot be measured.

Mithil Vakde on X: "I got 37% on ARC-AGI-2 Kaggle without LLMs 100M looped transformer trained from scratch - Likely highest score without an LLM - Peaked at 6th place - Beats last year's winning LLM - Looping, added in the last week: 33% → 35% → 37% thanks to @arcprize @fchollet @mikeknoop and @GregKamradt for the competition Link: https://t.co/iDBUrHhmdI writeup for the paper track " I got 37% on ARC-AGI-2 Kaggle without LLMs 100M looped transformer trained from scratch - Likely highest score without an LLM - Peaked at 6th place - Beats last year's winning LLM - Looping, added in the last week: 33% → 35% → 37% thanks to @arcprize@fchollet@mikeknoop and @GregKamradt for the competition Link: mvakde.github.io/blog/looped-md… writeup for the paper track I got 37% on ARC-AGI-2 Kaggle without LLMs 100M looped transformer trained from scratch - Likely highest score without an LLM - Peaked at 6th place - Beats last year's winning LLM - Looping, added in the last week: 33% → 35% → 37% thanks to @arcprize@fchollet@mikeknoop and @GregKamradt for the competition Link: mvakde.github.io/blog/looped-md… writeup for the paper track What's new and different: Looping the middle layers All demo pairs in context Hub-and-spoke sparse attention to fit in 4xL4s Custom synthetic data inspired by NVARC by @JFPuget and Ivan Sorkin Looping is a huge win: At equal depth, it beat my non-looped stacks with up to 4× the params, by 2.3–7.6% even at equal depth Even sub 1M models perform very well Room for growth -- I ran out of time, so final model is severely undertrained. Performance can go much higher How does the looping differ from HRM/TRM: 1 They claim 7M/25M params, but actually train 430M+ params. mdlARC beats their scores at TRUE 1M params 2 mdlARC is autoregressive 1 pixel at a time , so justification for looping is MDL. HRM/TRM use loops to rewrite drafts of the whole answer diffusion like 3 No hierarchical looping needed. Didnt help at all 4 MUCH cheaper and faster to train Why do it without an LLM? Because I care about sample efficiency. LLMs see so much data, ARC puzzles included, that you can't measure it. We now know that with enough data, ARC-1/2 become trivial So the interesting question is how far a model gets on a few ARC puzzles alone 44% on ARC-AGI-1 in 67 cents Trained from scratch in 2hrs on a 5090 Matches TRM, beats HRM and is way faster & cheaper No recursion, just a transformer Also, 7% on ARC-2 🧵