{"slug": "guide-to-loop-engineering-how-autoresearch-and-bilevel-autoresearch-turn-ai-into", "title": "Guide to Loop Engineering: How ‘autoresearch’ and ‘Bilevel Autoresearch’ Turn AI Agents Into Autonomous Machine Learning ML Research Loops", "summary": "Andrej Karpathy's open-source 'autoresearch' repository, released March 7, 2026, enables AI agents to autonomously run machine learning experiments, achieving an 11% training time reduction for GPT-2 after 700 experiments. The technique, called loop engineering, replaces manual prompting with automated iterative improvement using a verifier, state tracking, and stop conditions, and has been adopted by Shopify CEO Tobi Lütke for a 19% internal model improvement.", "body_md": "Most people still use AI like a 2015 search box. You type, you read, you type again. A newer pattern replaces that manual back-and-forth with a loop. This guide explains **loop engineering** using two verified artifacts. The sources are [Andrej Karpathy’s autoresearch repository](https://github.com/karpathy/autoresearch) and the\n\n[. The framing follows a write-up by](https://arxiv.org/pdf/2603.23420)\n\n`Bilevel Autoresearch`\n\npaper[@0xCodila](https://x.com/0xCodila/status/2072329149520232639).\n\n**What is Loop Engineering?**\n\nTo start, compare two modes. A prompt is one instruction, after which you decide the next step. A loop, by contrast, is a goal the model pursues until it arrives. The model plans, acts, checks its own result, then repeats. You define the objective once, and the loop handles iteration. Crucially, a loop only earns its cost when the work is measurable.\n\n**The Three Parts That Make A Loop Work**\n\nSo what separates a real loop from a chatbot on repeat? Every reliable loop has three components.\n\n- A\n**verifier** grades each attempt. That check can be a passing test, a moving metric, or a build. Without a verifier, the agent simply agrees with itself on repeat. **State** records what was tried, what failed, and what remains. A small side file lets the next run resume instead of restarting.- A\n**stop condition** prevents runaway cost. The loop halts when the goal is met, or after N attempts.\n\n**The Karpathy Loop: Inside ‘autoresearch’**\n\nThese three parts are not theoretical. On March 7, 2026, Karpathy released `autoresearch`\n\n, an open-source repository under the MIT license. It ships three core files and about 630 lines of code. The project went viral within days and now sits near 90,000 GitHub stars. It was latter presented as the pattern “the Karpathy Loop.”\n\nThe design is deliberately small, yet strict. The agent edits only `train.py`\n\n, which holds the GPT model, optimizer (Muon and AdamW), and training loop. It cannot touch the evaluation utilities in `prepare.py`\n\n. That separation stops the agent from making the test easier instead of the model better. Meanwhile, a human writes `program.md`\n\n, the instructions the agent must respect.\n\nEach cycle runs one experiment. The agent reads the code, proposes a change, trains five minutes, then keeps or rolls back. The scoring metric is `val_bpb`\n\n, validation bits per byte, where lower is better. That budget yields roughly 12 experiments per hour, so about 100 run overnight.\n\nThe reported outcomes are concrete. Karpathy pointed it at his already-optimized `nanochat`\n\nGPT-2 training code. It ran for two days and completed about 700 experiments, keeping 20 genuine improvements. Stacked together, those fixes cut GPT-2-quality training time by 11%, from 2.02 to 1.80 hours. One finding was a `QK-Norm`\n\nimplementation missing a scalar multiplier, which had left attention too diffuse across heads.\n\nNotably, humans tire after roughly a dozen experiments, whereas the loop does not. Separately, Shopify CEO Tobi Lütke ran `autoresearch`\n\novernight on an internal model. He reported a 19% improvement after 37 experiments. Karpathy’s takeaway: if you have an objective metric, you are the bottleneck.\n\n**Prompt vs Loop vs Bilevel Loop**\n\nThe differences become clearer side by side.\n\n| Aspect | One-shot prompt | Karpathy loop (`autoresearch` ) | Bilevel Autoresearch |\n|---|---|---|---|\n| You define | Each step | The goal, once | The goal, once |\n| Who iterates | You | Inner agent | Inner + outer agent |\n| Verifier | You, manually | `prepare.py` (`val_bpb` ) | Same metric, two levels |\n| State | Chat only | Experiment log | Log plus injected code |\n| Human role | Engine | Author of `program.md` | Author of `program.md` |\n| Reported result | Varies | 700 runs → 20 fixes, 11% speedup | 5x larger val_bpb drop |\n\n**The Five Building Blocks**\n\nConsequently, AI engineering teams now assemble working loops from **five reusable pieces**:\n\n**Automation** fires the loop on a schedule, event, or trigger.- A\n**skill** stores project knowledge in a markdown file, read on every run. **Sub-agents** split the writer from the reviewer, since one model grades itself too generously.**Connectors** let the loop act inside real tools, like an issue tracker or Slack.- Finally, a\n**verifier** remains the gate that rejects bad work. Claude Code and Codex now ship all five.\n\n**Bilevel Autoresearch: A Loop On Top Of The Loop**\n\nNext, researchers asked a sharper question. If autoresearch is research, can you autoresearch autoresearch? The research paper\n\nanswers yes.[B](https://arxiv.org/pdf/2603.23420)[ilevel Autoresearch: Meta-Autoresearching Itself](https://arxiv.org/pdf/2603.23420)\n\nThe **inner loop** matches Karpathy’s original: propose, train, evaluate, keep or discard. The **outer loop** watches the inner loop and reads its code and traces. It identifies where the search itself keeps stalling. Then it writes new Python mechanisms, injects them at runtime, and reruns the inner loop.\n\nThe result held on Karpathy’s GPT pretraining benchmark. The outer loop cut `val_bpb`\n\n5x more than the single loop (-0.045 vs -0.009). Notably, both loops used the same LLM, so the gain came from architecture, not a smarter model. In practice the design splits into three levels. Level 1 runs the base loop. Level 1.5 tunes search parameters every five iterations. Level 2 generates mechanisms through a four-round session. The reported experiments used an RTX 5090 32GB and a 300-second budget.\n\nThe reason is worth noting. The inner loop kept returning to the same priors, even after they stopped working. The outer loop broke those patterns by forcing unfamiliar exploration.\n\n**Use Cases With Examples**\n\nThese ideas transfer well beyond pretraining. For model work, a loop searches hyperparameters until `val_bpb`\n\ndrops. For software, it refactors until tests, types, and the build pass. For content, it rewrites until every rubric score clears a threshold. For data, it tunes a pipeline until schema checks hold. Each case shares one trait: **an automatic gate that can fail the work.**\n\n**Try It Yourself: A Loop In One Prompt**\n\nTheory aside, you can feel the mechanic without Claude Code or Codex. Paste this into any capable model and watch it self-correct.\n\n```\nYou will work in a loop until the task meets the bar.\n\nTASK:\n[describe exactly what you want produced]\n\nSUCCESS CRITERIA (be strict):\n- [criterion 1]\n- [criterion 2]\n- [criterion 3]\n\nLOOP PROTOCOL, repeat every turn:\n1. PLAN   - state the single next step.\n2. DO     - produce or improve the work.\n3. VERIFY - score the result 1-10 on each criterion. Be honest.\n4. DECIDE - if every criterion is 8+, print FINAL and stop.\n            Otherwise print ITERATING and fix the weakest point first.\n\nRULES:\n- Never call it done until every criterion is 8 or higher.\n- Each pass must fix the weakest score from the last VERIFY.\n- Do not ask questions. Make a sensible assumption and continue.\nBegin.\n```\n\nUnderneath, the control flow is small. The skeleton below shows those three parts in Python: a verifier, a decision, and two stop conditions.\n\n```\ncurrent = baseline\nbest = evaluate(current)                 # verifier: lower val_bpb is better\nfor step in range(MAX_STEPS):            # stop condition 1: experiment budget\n    candidate = propose_change(current)  # agent edits train.py\n    score = train_and_eval(candidate)    # train 5 min, then verify\n    if score < best:                     # keep only real improvements\n        current, best = candidate, score # commit\n    # else: discard candidate, restore baseline\n    if best <= TARGET:                   # stop condition 2: goal met\n        break\n```\n\nBoth versions are limited. You are still the trigger, and closing the tab erases the state. Adding automation, a state file, and a real gate turns this into an autonomous loop.\n\n**See It Run**\n\nThe interactive demo below animates one full loop: propose, train, verify, then keep or roll back. Adjust the target and step limit, and watch `val_bpb`\n\nfall until the stop condition fires.\n\n**Key Takeaways**\n\n- A loop needs three parts: a verifier, persistent state, and a stop condition.\n`autoresearch`\n\nlets an agent edit only`train.py`\n\nand never the evaluator.- Karpathy’s overnight runs kept 20 fixes from 700 experiments, for an 11% speedup.\n`Bilevel Autoresearch`\n\nadds an outer loop for a 5x`val_bpb`\n\ngain.- Loops shift the work to design and review; they do not remove thinking.", "url": "https://wpnews.pro/news/guide-to-loop-engineering-how-autoresearch-and-bilevel-autoresearch-turn-ai-into", "canonical_source": "https://www.marktechpost.com/2026/07/12/guide-to-loop-engineering/", "published_at": "2026-07-12 20:07:02+00:00", "updated_at": "2026-07-12 20:19:43.858440+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-agents", "ai-research", "ai-tools"], "entities": ["Andrej Karpathy", "autoresearch", "nanochat", "Shopify", "Tobi Lütke", "GPT-2", "Bilevel Autoresearch"], "alternates": {"html": "https://wpnews.pro/news/guide-to-loop-engineering-how-autoresearch-and-bilevel-autoresearch-turn-ai-into", "markdown": "https://wpnews.pro/news/guide-to-loop-engineering-how-autoresearch-and-bilevel-autoresearch-turn-ai-into.md", "text": "https://wpnews.pro/news/guide-to-loop-engineering-how-autoresearch-and-bilevel-autoresearch-turn-ai-into.txt", "jsonld": "https://wpnews.pro/news/guide-to-loop-engineering-how-autoresearch-and-bilevel-autoresearch-turn-ai-into.jsonld"}}