{"slug": "grokking-a-777-parameter-transformer-learned-real-math", "title": "Grokking: A 777-Parameter Transformer Learned Real Math", "summary": "A 777-parameter transformer trained on 10-digit addition reached 99.69% test accuracy, demonstrating \"grokking\" — the phenomenon where a network generalizes to the underlying algorithm long after memorizing training data — according to The Menon Lab's account of the yhavinga/gpt-acc-jax model. The architecture uses 1 layer, hidden dimension 7, 1 attention head, FFN dimension 14, vocabulary size 14 and context length 35, with tied embeddings, no FFN bias, learned positional embeddings and a learning rate of 0.02; the lab reports a sharp \"parameter cliff\" near 800 parameters below which no amount of training produces generalization. The result extends the 2022 OpenAI paper \"Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets\" by Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin and Vedant Misra, which demonstrated grokking on modular arithmetic such as a + b mod 97, to arbitrary-precision decimal arithmetic.", "body_md": "# Grokking: How a 777-Parameter Transformer Learned Real Math\n\nIn January 2025, something remarkable happened.\n\nA transformer with only **777 parameters** learned to perform 10-digit addition with 99.69% accuracy.\n\nLet that sink in.\n\nTo memorize all possible 10-digit addition problems, you’d need roughly 10^20 parameters. This model had 777. It couldn’t possibly memorize. Which means it had to **actually learn the algorithm**.\n\nThis is grokking.\n\n## What Is Grokking?\n\nGrokking is a phenomenon where neural networks suddenly generalize long after they’ve memorized the training data.\n\nHere’s what it looks like:\n\n1. \n**Phase 1 (Memorization):** The model achieves 100% training accuracy by memorizing examples. Test accuracy stays near random (0%).\n2. \n**Phase 2 (Plateau):** Nothing visible happens. Training loss is low. Test accuracy is still garbage. Most people stop training here.\n3. \n**Phase 3 (Grokking):** Suddenly — often thousands of epochs later — test accuracy jumps from ~0% to ~100%. The model has “figured out” the underlying algorithm.\n\nThe original 2022 paper — *Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets* by Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra at OpenAI — demonstrated this phenomenon on modular arithmetic (like `a + b mod 97`). If you have not read it, the [YouTube walkthrough](https://youtu.be/5fbW-5TxBNw) is the best 20-minute investment you can make in understanding why this matters. The 777-parameter result pushed it further: grokking works on **arbitrary-precision decimal arithmetic** — the kind of math we actually use.\n\n## The 777-Parameter Architecture\n\nThe winning model from [yhavinga/gpt-acc-jax](https://github.com/yhavinga/gpt-acc-jax):\n\n```\nLayers: 1\nHidden dim: 7\nAttention heads: 1\nFFN dim: 14 (2× expansion)\nVocab size: 14\nContext length: 35\n```\n\nKey optimizations that got it this small:\n\n- **Tied embeddings:** Input and output embeddings share weights\n- **No FFN bias:** Removes a few dozen parameters\n- **Learned positions:** Sinusoidal positions failed; learned ones are essential\n- **High learning rate (0.02):** Tiny models need higher LR to grok\n\nWhat **didn’t** work:\n\n- RMSNorm (broke generalization)\n- No delimiters (needed +/= tokens)\n- RoPE embeddings (crashed)\n- Going below d=7 (sharp “parameter cliff”)\n\n## The Parameter Cliff\n\nThis is fascinating: there’s a hard threshold around 800 parameters where models abruptly go from 0% to 100% accuracy.\n\nBelow the cliff: no amount of training helps. The model simply lacks capacity. Above the cliff: grokking happens reliably.\n\nThe transition is **sharp** — not gradual. It suggests there’s a minimum representational capacity needed to encode the addition algorithm.\n\n## Why Does Grokking Happen?\n\nThe leading theory involves **weight decay** and **representation compression**.\n\n1. During memorization, the model learns a “lookup table” with large, unstructured weights\n2. Weight decay (L2 regularization) continuously pushes weights toward zero\n3. Eventually, the pressure to shrink weights forces the model to find a more compact representation\n4. The compact representation turns out to be… the actual algorithm\n\nIn Neel Nanda’s mechanistic interpretability work, researchers literally reverse-engineered grokked models and found they’d learned **Fourier features** for modular arithmetic — the same mathematical structure humans would use.\n\nThe network didn’t just learn to generalize. It discovered the correct math.\n\n## What We’re Experimenting With\n\nAt The Menon Lab, we’re running grokking experiments on modular arithmetic:\n\n```\n# Training config (matched to 777-param paper)\nOPERATION = 'add'  # Modular addition\nP = 97  # Prime modulus\nEMBED_DIM = 7  # Match 777-param model\nNUM_EPOCHS = 100000  # 100k epochs for grokking!\nLEARNING_RATE = 2e-2  # Higher LR for tiny models\nWEIGHT_DECAY = 1.0  # Crucial for grokking\n```\n\nOur goal: replicate grokking, visualize the weight evolution, and extend it to other arithmetic operations (subtraction, multiplication, division).\n\nThe code lives at [menonpg/tiny-transformers](https://github.com/menonpg/tiny-transformers).\n\n## Why This Matters\n\nGrokking has profound implications:\n\n**1. Neural networks can learn algorithms, not just patterns.**\nThe 777-parameter model didn’t memorize. It couldn’t. It discovered a general procedure for addition that works on numbers it had never seen.\n\n**2. Overtraining might be underrated.**\nWe typically stop training when validation loss plateaus. Grokking shows that useful learning can happen long after apparent convergence.\n\n**3. Weight decay isn’t just regularization.**\nIt’s a force that pushes networks toward simpler, more general solutions — even when memorization works fine on the training set.\n\n**4. Interpretability gets concrete.**\nWhen a model groks, we can often reverse-engineer what it learned. This opens a path to understanding neural networks mechanistically.\n\n## The Open Questions\n\n- Can we predict *when* grokking will happen?\n- Does grokking scale to larger, more complex algorithms?\n- Can we induce grokking deliberately (not just wait for it)?\n- What does the representation look like mid-grok?\n\nThese are active research areas. And with transformers this small, anyone with a laptop can run experiments.\n\n## Try It Yourself\n\nClone our notebook and run 100k epochs:\n\n```\ngit clone https://github.com/menonpg/tiny-transformers\ncd tiny-transformers/notebooks\n# Open grokking_arithmetic.ipynb in Colab or Jupyter\n```\n\nWatch the training curves. The sudden jump from memorization to generalization is genuinely thrilling to witness.\n\nGrokking reminds us that neural networks are stranger — and more capable — than we give them credit for. A 777-parameter model discovered addition. What else might be waiting to be grokked?\n\n**References:**\n\n- [Power, Burda, Edwards, Babuschkin, Misra (2022) — Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://arxiv.org/abs/2201.02177) ·[YouTube walkthrough](https://youtu.be/5fbW-5TxBNw)\n- [yhavinga/gpt-acc-jax](https://github.com/yhavinga/gpt-acc-jax) - 777-parameter transformer\n- [Nanda & Lieberum - Progress Measures for Grokking](https://arxiv.org/abs/2301.05217)\n- [stockeh/mlx-grokking](https://github.com/stockeh/mlx-grokking) - Grokking in <150 epochs with MLX", "url": "https://wpnews.pro/news/grokking-a-777-parameter-transformer-learned-real-math", "canonical_source": "https://themenonlab.blog/blog/grokking-777-parameter-transformer", "published_at": "2026-10-26 01:27:16+00:00", "updated_at": "2026-10-26 01:43:37.795767+00:00", "lang": "en", "topics": ["machine-learning", "ai-research", "neural-networks", "artificial-intelligence"], "entities": ["The Menon Lab", "yhavinga/gpt-acc-jax", "OpenAI", "Alethea Power", "Yuri Burda", "Harri Edwards", "Igor Babuschkin", "Vedant Misra"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/grokking-a-777-parameter-transformer-learned-real-math", "markdown": "https://wpnews.pro/news/grokking-a-777-parameter-transformer-learned-real-math.md", "text": "https://wpnews.pro/news/grokking-a-777-parameter-transformer-learned-real-math.txt", "jsonld": "https://wpnews.pro/news/grokking-a-777-parameter-transformer-learned-real-math.jsonld"}}