Show HN: I RL-trained an agent that trains models with RL (for โ€“$1.3k) A developer built and open-sourced an AI agent that uses reinforcement learning to train smaller language models via reinforcement learning, achieving a peak reward of 0.63 over 54 training steps and transferring to a held-out task family. The project, costing $1,300, demonstrates a two-loop RL system where the outer agent learns to generate training configurations for inner models. ๐Ÿ”“ Everything is open sourcedincluding: the trained agent's weights LoRA adapter on ๐Ÿค— HF , agent harness, task families, reward code, GPU orchestration, tinker RL training scripts, and retro write-ups of every pilot including the failures . Jump to Getting started โ†“ TL;DR: - I built a pipeline where an AI agent: - Leveraging Tinker https://thinkingmachines.ai/tinker , I then RL-trained the agent itself, rewarding it when it trained better models. - Reward climbed from ~0.0 to a ~0.63 peak over 54 training steps. Transferring to a held-out task family it never trained on . An AI in an RL loop, whose action is training AI in an RL loop. Source: assets/hero.svg. ๐Ÿ” How it works -how-it-works โš–๏ธ Reward design %EF%B8%8F-reward-design ๐Ÿงฉ Task families -task-families ๐Ÿ“ˆ Results -results ๐Ÿ–ฅ๏ธ Infrastructure %EF%B8%8F-infrastructure ๐Ÿ’ฐ Costs -costs ๐Ÿค— Model weights -model-weights ๐Ÿš€ Getting started -getting-started ๐Ÿ”ฎ Future improvements -future-improvements Acknowledgements acknowledgements Two RL loops with two entirely separate training stacks. | who is being trained | what training looks like | stack | | |---|---|---|---| Outer loop | the trainer agent | Tinker https://thinkingmachines.ai/tinker + tinker-cookbook importance-sampling GRPO Inner loop Qwen3-0.6B https://huggingface.co/Qwen/Qwen3-0.6B / 1.7B https://huggingface.co/Qwen/Qwen3-1.7B verifiers https://github.com/PrimeIntellect-ai/verifiers environment + rubric, with a prime rl config table prime-rl https://github.com/PrimeIntellect-ai/prime-rl GRPO on Runpod GPUsTinker trains the agent. The agent writes verifiers envs, rubrics, and prime-rl configs. prime-rl trains the small model. The inner model's hidden-eval score flows back up as the outer loop's reward. One episode, end to end. Source: assets/episode.svg. One episode = one attempt by the trainer agent to produce a valid, high-quality training job for a given task: Task spec โ€” a description of what to train "teach a small model to resolve multi-hop persona queries" , hard constraint bounds, the eval tool interface, and a handful of dev examples. The agent works โ€” it edits a sandboxed workspace through read file / write file / edit file / list files , and can call get baseline scores to see the untrained base models' scores on the hidden eval.โ€” triggers a validation probe. Any failures are returned and the agent gets capped retries. submit job Dispatch โ€” a validated job is queued and picked up by a warm pool of Runpod GPU pods, which run GRPO training with prime-rl and score the checkpoint pre/post on the hidden eval. Reward โ€” combines validation efficiency with the trained model's uplift over the best untrained baseline. The outer loop then RL-trains the agent itself on episode reward, using Tinker https://thinkingmachines.ai/tinker . Every outer-loop batch spawns 40 real inner training jobs across up to 16 GPU pods. Episode reward is a weighted sum live weights 0.35 / 0.60 / 0.05 : Validation โ€” 1.0 for a first-try valid submission, decaying per extra attempt; 0 if the episode never validates. Separately, the outer loop scores an episode โˆ’0.1 when the agent never produces a parseable submission at all. The โˆ’1.0 values you'll see in the CSVs are a "no post score" logging sentinel, not a reward. Job quality โ€” a hybrid of the trained model's absolute post-training score and its signed uplift over best pre , the best untrained model's frozen baseline: 0.25ยทpost + 0.75ยทuplift term . A job that dies on the GPU scores 0 here the episode keeps its validation term . Train speed โ€” a small tie-breaker for faster jobs, gated on job success. Note for close readers: the agent-facing prompt template/INSTRUCTIONS.md gives the agent a simplified view โ€” the 75/25 uplift/absolute split inside job quality, plus a fewer-attempts nudge โ€” not the full 0.35/0.60/0.05 decomposition. The published adapter was trained against that prompt; the reward actually computed is the one above. Six families of tasks, deliberately built so that the untrained models struggled without training, and all require multi-step tool use and reasoning: | family | shape | untrained best pre n=200 | |---|---|---| | calc chain | chained arithmetic where each step feeds the next | 0.742 | | multi hop | persona-world queries needing multiple dependent lookups | 0.654 | | string pipeline | composed string transformations | 0.545 | | ledger | stateful account bookkeeping via tools | 0.242 | | dispatch | conditional routing decisions | 0.323 | triage held out | on-call incident triage: correlating services, incidents and deploys across tools | 0.352 | Five families train the agent; triage is never trained on and serves as the generalisation probe. Setup: Qwen3.6-35B-A3B trainer agent, LoRA rank 8, lr 4e-5, GRPO with group size 8, up to 16 concurrent GPU pods, ~40 real training jobs per batch. Runs: pilot-7 10 steps โ†’ 7b 24 steps, warm-started โ†’ 7c 20 steps, warm-started โ€” 54 steps total. Decomposing the reward shows what was learned, and in what order: Rung 1 โ€” process reliability pilot-7 . The entire early gain came from converting validation failures and dead-on-GPU jobs into completed episodes. Job quality stayed flat while total reward rose to ~0.26. Showing GRPO taking the steepest gradient first. Rung 2 โ€” making better models pilot-7b onward . With reliability saturating ~0.75โ€“0.80 validation , job grade rose 0.30 โ†’ 0.41 and the hidden-eval post-training score went from ~0.04 noise to a sustained 0.22โ€“0.48. The agent started making better models, not just working ones. A task family which the agent never trained on showed performance rises with outer-loop training, then plateaus: | checkpoint | outer steps | mean reward | validated | inner jobs succeeded | worst episode | |---|---|---|---|---|---| | base model | 0 | 0.399 | 9/10 | 5/9 | 0.000 | | pilot-7 final | 10 | 0.438 | 8/10 | 8/8 | 0.000 | | pilot-7b final | 34 | 0.545 | 10/10 | 9/10 | 0.290 | | pilot-7c final | 54 | 0.492 | 10/10 | 8/10 | 0.175 | Early training runs were model-selection blind: 77/79 episodes chose the weaker 0.6B model. After introducing the get baseline scores tool and uplift grading, the policy flipped and remained so during training, deepening across the arc: 1.7B share of job-writing episodes went 42% โ†’ 95% . It also adopted the exposed prime rl config surface 21% โ†’ ~78% of episodes within one warm-start boundary , with a sensible key mix: sampling temperature, optimizer choice, algorithm variant, scheduler, loss. 16x warm GPU pods training at any one time: Inner loop the jobs the agent writes : Runpod warm-pod fleet โ€” a capped pool up to 16 of pods, bootstrap-pinned to exact prime-rl + verifiers revisions so every node is a replica; ~2 min from provision to serving. Idle pods are reaped; the queue is file-backed queued/ โ†’ running/ โ†’ done/ . GPU selection is data-driven โ€” a benchmark matrix over GPU ร— base-model found 2ร— RTX A5000 wins cost at ยฃ0.10/job ~$0.13 ; preference ladders are walked at provision time to take whatever's in stock. What the fleet actually ran on headline arc, ~1,750 jobs โ€” below the nominal 54 steps ร— 40 because episodes that fail validation never dispatch a job : A40 64% 340 GPU-train-hours ยท RTX 4090 32% 151h ยท RTX A6000 3% ยท RTX A5000 1%. The benchmark cost-winner was rarely in stock, so the ladder spent most of the arc on A40s. prime-rl GRPO trains the small model; checkpoints are scored pre/post on the hidden eval with vLLM. Outer loop training the agent : Tinker Thinking Machines' managed RL API trains Qwen3.6-35B-A3B with LoRA via tinker-cookbook's importance-sampling GRPO. A control-inversion bridge runs each episode as a background task behind a queue-backed policy, so the cookbook loop drives episodes turn-by-turn while all harness logic validation retries, nudges, grading stays unchanged. Async off-policy max steps off policy=2 defeats the straggler barrier โ€” one slow episode no longer gates a whole batch. Zero stale discards across the headline arc. Everything is metered. Every LLM call logs tokens and USD; runs/costs.jsonl is a global spend ledger enforced against per-episode budgets. The whole orchestration runs on a CPU box, rented via Nebius https://nebius.com/ . | item | loosely measured cost | |---|---| | one inner training job train + pre/post eval | ~ยฃ0.10โ€“0.15 ~$0.13โ€“0.20 benchmarked on short tasks; ~ยฃ0.15โ€“0.22 ~$0.20โ€“0.30 for the long persona-family jobs | | one outer-loop episode agent tokens, Tinker | ~ยฃ0.11โ€“0.19 ~$0.15โ€“0.25 | | one outer-loop batch 40 real GPU jobs + agent tokens | ~ยฃ11โ€“17 ~$15โ€“23 all-in | | holdout eval arm n=10 | ~ยฃ4โ€“6 ~$5โ€“8 | | Runpod, billed across the arc window | ~ยฃ605 ~$810 โ€” includes some concurrent baseline-seeding and GPU-benchmark pods | | Tinker across the arc invoiced โ€” all agent sampling & training, incl. the holdout-eval episodes | ~ยฃ345 ~$465 | whole headline arc | ~ยฃ950 ~$1,275 all-in ~ยฃ605 Runpod, ~ยฃ345 Tinker | Honesty footnote: ~ยฃ950 is the headline arc, not the project โ€” the pilots, GPU benchmarking, baseline seeding, and blind alleys that got here written up in the docs/ retros cost a few hundred more on top. Benchmark-matrix rows on cold pods ran as high as ~$0.37/job see benchmarks/REPORT.md ; the per-job range above is warm-pool arc jobs. GBP figures at ยฃ0.745/$1 10 Jul 2026 . The trained trainer agent is on Hugging Face: Danau5tin/ai-trains-ai-trainer . It's the LoRA adapter rank 8, ~560MB from the step-34 checkpoint โ€” the held-out transfer peak in the table above โ€” derived from Qwen/Qwen3.6-35B-A3B https://huggingface.co/Qwen/Qwen3.6-35B-A3B and released under Apache-2.0 to match the base model. Load it with PEFT or serve it with vLLM's LoRA support; to run full episodes, drive it through this harness. The model card has usage snippets and honesty notes. If you'd like to reproduce, or extend. The below will get you there .env at repo root with OPENROUTER API KEY=... and TINKER API KEY for outer RL uv sync uv run pytest fully offline: no network, no keys uv run at-episode --task examples/tasks/calc chain v1 fast.json --model qwen3.6-27b run one episode with a frontier agent Each episode prints reward, token usage, and dollar cost, and writes full artifacts trajectory, manifest, the agent's workspace to runs/