AIArticle Open weights, real Runpod jobs, and held-out transfer make meta-RL a forkable template, not theater.
Rachel Goldstein Most agent RL demos stop at tool calling. The interesting systems go one level deeper: the agent’s action is to design and launch a full training job, then get scored on whether the resulting model actually improved. That is the nested loop in ai-trains-ai, a fully open pipeline where an outer agent is RL-trained to write inner RL jobs for small models, and the whole thing was run for roughly $1.3k on real GPUs.
This is not a new paper claiming AGI. It is a reproducible engineering template with LoRA weights, harness, task families, reward code, GPU orchestration, and failure write-ups. For developers building agent training infrastructure, that matters more than another abstract meta-learning claim.
Two loops, two stacks #
The architecture is deliberately split.
Outer loop trains the trainer agent: a Qwen 3.6-35B-A3B model adapted with LoRA. Episodes are “write a training job for this task.” The outer trainer is Tinker with importance-sampling GRPO (group relative policy optimization).
Inner loop is whatever job the agent just wrote: a verifiers environment, rubric, dataset, and prime-rl
config. Small base models (Qwen3-0.6B or 1.7B) train with GRPO on Runpod pods. The hidden-eval score of that checkpoint flows back as the outer episode reward.
One episode is concrete work, not a toy MDP:
- Task spec lands (what to teach, hard constraints, eval interface, a few dev examples).
- The agent edits a sandboxed workspace (
read_file
/write_file
/edit_file
/list_files
) and can callget_baseline_scores
on untrained bases. submit_job
runs a validation probe; failures return with capped retries.- Valid jobs hit a warm pool of pods (up to 16), train with prime-rl, and score pre/post on a hidden eval.
- Outer GRPO updates on the episode reward.
Every outer batch spawns about 40 real inner training jobs. That is the load-bearing detail: the reward is not a proxy LLM judge of a draft config. It is uplift measured after actual GPU training.
Reward that teaches “train better models” #
Episode reward is a weighted sum (live weights 0.35 / 0.60 / 0.05):
Validation(0.35): 1.0 for first-try valid submission, decaying with retries; 0 if never valid. Episodes that never produce a parseable submission also take a −0.1 outer penalty.Job quality(0.60): hybrid of absolute post-training score and signed uplift overbest_pre
(best untrained baseline). Internally:0.25·post + 0.75·uplift_term
. Jobs that die on the GPU score 0 here.Train speed(0.05): small tie-breaker, gated on success.
The agent-facing prompt simplifies this (75/25 uplift/absolute inside job quality, plus a fewer-attempts nudge). The published adapter was trained against that simplified view; the true reward is the 0.35/0.60/0.05 mix. That gap is worth remembering if you fork the prompt.
Task families force multi-step tool use and leave untrained models weak enough that training can help:
| Family | Shape | Untrained best_pre (n=200) |
|---|---|---|
| calc_chain | chained arithmetic | 0.742 | | multi_hop | persona lookups | 0.654 | | string_pipeline | composed transforms | 0.545 | | ledger | stateful bookkeeping | 0.242 | | dispatch | conditional routing | 0.323 |
| triage (held out) | on-call incident triage | 0.352 | Five families train the agent. triage never appears in outer training and is the generalization probe.
What actually improved #
Over 54 outer training steps, episode reward climbed from ~0.0 to a ~0.63 peak, in two distinct rungs rather than a smooth curve. Skill transferred to the held-out triage family. The agent also learned to pick the better base model between the 0.6B and 1.7B options when that choice mattered.
Those three results together are the signal. Climbing reward alone can be reward hacking. Transfer to a family with different surface form (incident correlation across services, deploys, tools) is harder to fake. Base-model selection is a small but real “meta-ML” behavior: the policy is not only writing envs and rubrics, it is making a capacity trade-off under the same reward.
Caveats that still apply: this is GRPO on a constrained task suite with verifiable hidden evals. It is not proof that the same agent will invent good training recipes for open-ended production problems, long-horizon computer use, or noisy human preference data. It is proof that nested, GPU-backed agent RL can move a measurable needle on held-out work at modest cost, with everything open.
Developer angle: when to fork this #
If you already ship agents with tools and you are past pure prompting, this pattern is closer to ART / Unsloth-style multi-turn GRPO, NVIDIA-style RLVR with verifiers, and other GRPO-first agent trainers than it is to classic single-env Q-learning demos. The difference is the action space: configs and training jobs, not only API tool calls. Who should care
- Teams building internal “train a specialist model for task X” workflows and tired of hand-written GRPO configs.
- Researchers who need a real outer/inner loop without standing up two research codebases from scratch.
- Platform engineers who want a reference for warm GPU pools, job validation probes, and reward plumbing that survives real pod failures.
What you actually adopt
- Clone the harness and task template; keep the outer Tinker path and the inner prime-rl path separate. Mixing stacks is how these projects become unmaintainable.
- Define task families the way this one does: untrained baselines must be weak, eval must be hidden from the agent, and at least one family must be held out for transfer.
- Implement reward as code against post-train metrics, not as “LLM grades the YAML.” Validation efficiency + uplift over frozen baselines is a clean default.
- Budget for fan-out. Outer batches that spawn tens of inner jobs need a warm pool and hard failure modes (dead job → quality 0, keep validation term).
- Ship LoRA adapters and reward code together. Weights without the exact reward and prompt view are not reproducible.
Trade-offs
- Cost and latency: every outer step is many full GRPO runs. Fine for offline improvement of a trainer agent; wrong for online user-facing loops.
- Reward design still dominates. RULER-style LLM judges (common in ART-style stacks) are faster to prototype; verifiable uplift after real training is slower and more trustworthy when you can afford it.
- Base model priors matter. Outer 35B-class capacity and inner 0.6B/1.7B bases are a deliberate asymmetry. If your outer model cannot already write coherent training code, GRPO will not invent that skill from a cold start.
- The simplified agent prompt vs true reward decomposition is a footgun when you change either side independently.
If it pans out further
Treat this as a 2025-era “hello world” for agents whose product is model training, not chat. The next useful forks are: more task families with production-shaped verifiers, stronger outer bases, tighter cost controls on inner steps, and comparison against SFT-on-traces then GRPO (the usual path when pure RL from a weak prior fails). Worth your attention now if you are already paying for agent RL experiments; park it if you are still choosing between RAG and SFT for a single workflow.
Where this sits #
The broader stack is converging on GRPO + verifiable rewards for agent reliability: Unsloth/ART for multi-turn trajectories, NeMo Gym-style environments for enterprise RLVR, and assorted recipes for computer-use agents. ai-trains-ai’s contribution is not a new optimizer. It is closing the loop so the reward is “did the trained model get better on a hidden eval,” then open-sourcing the agent that learned to chase that signal.
That is the right unit of progress for practitioners. Nested RL only becomes real when someone else can rerun the outer loop, inspect the failures, and swap in their own task family without reverse-engineering a blog post. This repo is built for that. The ~0.63 peak and held-out transfer are encouraging; the open harness, reward code, and real Runpod path are the parts that change how you work this week.
Sources & further reading #
[Show HN: I RL-trained an agent that trains models with RL (for –$1.3k)](https://github.com/Danau5tin/ai-trains-ai)— github.com -
[Training AI Agents with RL | Unsloth Documentation](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/training-ai-agents-with-rl)— unsloth.ai -
[Training an Agent with Reinforcement Learning](https://tsnewnami.bearblog.dev/training-an-agent-with-reinforcement-learning/)— tsnewnami.bearblog.dev -
Reinforcement Learning: How to Train an RL Agent from Scratch | by Team Datatonic | ${around_the_block}— blog.montrealanalytics.com - Mastering Agentic Techniques: AI Agent Reinforcement Learning | NVIDIA Technical Blog— developer.nvidia.com
Rachel Goldstein· Dev Tools Editor Rachel has been embedded in the developer tooling ecosystem for nearly eight years, covering everything from IDE wars and package-manager drama to the quiet rise of AI-assisted coding. She has a soft spot for open-source maintainers and an unhealthy number of terminal emulators installed on a single laptop.
Discussion 0 #
No comments yet
Be the first to weigh in.