Mithil Vakde on X: "I got 37% on ARC-AGI-2 Kaggle without LLMs! 100M looped transformer trained from scratch
- Likely highest score without an LLM
- Peaked at 6th place
- Beats last year's winning LLM
- Looping, added in the last week: 33% → 35% → 37% thanks to @arcprize @fchollet @mikeknoop and @GregKamradt for the competition!
Link: https://t.co/iDBUrHhmdI
(writeup for the paper track)"
I got 37% on ARC-AGI-2 Kaggle without LLMs! 100M looped transformer trained from scratch
- Likely highest score without an LLM
- Peaked at 6th place
- Beats last year's winning LLM
- Looping, added in the last week: 33% → 35% → 37% thanks to @arcprize@fchollet@mikeknoop and @GregKamradt for the competition!
Link: mvakde.github.io/blog/looped-md…
(writeup for the paper track)
I got 37% on ARC-AGI-2 Kaggle without LLMs! 100M looped transformer trained from scratch
- Likely highest score without an LLM
- Peaked at 6th place
- Beats last year's winning LLM
- Looping, added in the last week: 33% → 35% → 37% thanks to @arcprize@fchollet@mikeknoop and @GregKamradt for the competition!
Link: mvakde.github.io/blog/looped-md…
(writeup for the paper track)
What's new and different: Looping the middle layers All demo pairs in context Hub-and-spoke sparse attention (to fit in 4xL4s) Custom synthetic data (inspired by NVARC by @JFPuget and Ivan Sorkin)
Looping is a huge win: At equal depth, it beat my non-looped stacks with up to 4× the params, by 2.3–7.6% (even at equal depth) Even sub 1M models perform very well! Room for growth -- I ran out of time, so final model is severely undertrained. Performance can go much higher
How does the looping differ from HRM/TRM:
- They claim 7M/25M params, but actually train 430M+ params. mdlARC beats their scores at TRUE 1M params
- mdlARC is autoregressive (1 pixel at a time), so justification for looping is MDL. HRM/TRM use loops to rewrite drafts of the whole answer (diffusion like)
- No hierarchical looping needed. Didnt help at all
- MUCH cheaper and faster to train
Why do it without an LLM? Because I care about sample efficiency. LLMs see so much data, ARC puzzles included, that you can't measure it. We now know that with enough data, ARC-1/2 become trivial So the interesting question is how far a model gets on a few ARC puzzles alone
44% on ARC-AGI-1 in 67 cents! Trained from scratch in 2hrs on a 5090 Matches TRM, beats HRM and is way faster & cheaper No recursion, just a transformer Also, 7% on ARC-2 🧵