{"slug": "how-to-run-an-autoresearch-workflow-with-rl-agent-skills-and-nvidia-nemo", "title": "How to Run an Autoresearch Workflow with RL Agent Skills and NVIDIA NeMo", "summary": "NVIDIA released a guide on running an autoresearch workflow using RL agent skills and the NeMo framework, enabling AI agents to autonomously set up experiments, train models, and iterate on research goals. The workflow, tested with Codex on NeMo RL and NeMo Gym, improved a vision language model's accuracy from 25% to 96.9% on a visual counting task. This approach aims to automate repetitive research tasks while keeping human researchers in control of strategy and decision-making.", "body_md": "Coding [AI agents](https://www.nvidia.com/en-us/ai/) are becoming practical operators for long-running machine learning (ML) workflows. They can inspect repositories, set up runtimes, resolve build issues, launch experiments, monitor execution, analyze metrics, and summarize results.\n\nFor [reinforcement learning (RL)](https://www.nvidia.com/en-us/glossary/reinforcement-learning/) research, this matters because meaningful metrics often appear only after the essential experiment infrastructure is in place. [Autoresearch](https://github.com/karpathy/autoresearch) is an open source Python project by Andrej Karpathy for automating AI and ML model training.\n\nWith this approach, [autonomous AI agents](https://www.nvidia.com/en-us/glossary/ai-agents/) translate high-level goals into hypotheses, edit and test a real codebase, keep metric-improving ideas, and hand results back to the human researcher. This enables teams to build specialized domain agents starting from a performant open model, such as [NVIDIA Nemotron](https://developer.nvidia.com/topics/ai/nemotron), then improve it with RL against measurable domain tasks while maintaining control over data, IP, deployment, and the training workflow.\n\nThis post walks through how to run a lightweight, no-code-from-the-user-side, skill-based autoresearch workflow. It was tested using a frontier coding agent (Codex with GPT 5.5) on [NVIDIA NeMo RL](https://github.com/nvidia-nemo/rl) and [NVIDIA NeMo Gym](https://github.com/NVIDIA-NeMo/Gym), using an [NVIDIA Brev](https://brev.nvidia.com/logout) GPU instance.\n\n## Building an autonomous RL research workflow\n\nThe workflow demonstrates the following three agent capabilities:\n\n**Full-stack autonomy**: Setting up the software stack; resolving dependencies; managing GPU memory, disk space, and checkpoints; launching experiments; monitoring runs; and debugging issues.**Goal-driven autoresearch**: Profiling a baseline, proposing or following hypotheses, launching experiments, analyzing metrics, and iterating toward a research objective.**Paper-to-code**: Reading a paper, forming an implementation plan, translating the algorithm into code, adding tests, and starting validation training.\n\nIn the examples provided in this post, Codex first brings up a full NeMo RL and NeMo Gym stack for a [vision language model (VLM)](https://www.nvidia.com/en-us/glossary/vision-language-models/) RL training smoke test. It then conceptualizes and creates a novel NeMo Gym visual counting environment from scratch, and trains the Qwen3-VL-2B-Instruct model, increasing its accuracy from 25.0% to 96.9% on the task. Finally, it implements an off-policy RL algorithm from a research paper and begins a 10-hour validation training campaign.\n\nThe goal of autoresearch is not to remove the researcher from the loop, but instead to hand off the repetitive setup and iteration work to the agent. Researchers are still responsible for setting goals, reviewing milestones, steering strategy, and making the final decision.\n\n## Using NeMo RL, NeMo Gym, and agent skills for autoresearch\n\n[NeMo RL](https://github.com/nvidia-nemo/rl) and [NeMo Gym](https://github.com/NVIDIA-NeMo/Gym) are open source libraries that are part of the [NVIDIA NeMo framework](https://github.com/NVIDIA-NeMo). NeMo RL is built on [AutoModel](https://github.com/nvidia-nemo/automodel), [Megatron-Bridge](https://github.com/NVIDIA-NeMo/Megatron-Bridge), and [vLLM](https://github.com/vllm-project/vllm) for post-training LLMs and VLMs, orchestrated by [Ray](https://github.com/ray-project/ray). It supports workflows such as GRPO, DPO, SFT, and reward model training, with recipe-driven configuration that can scale from small validation runs to distributed training. NeMo Gym provides environments where models can interact with tasks, receive rewards, and learn through live-generated experience.\n\nTogether, NeMo RL and NeMo Gym provide a useful substrate for agent-led RL research. An agent can inspect recipes, wire up a new environment, run a baseline, adjust training parameters, and compare results in the same repository.\n\nCodex with GPT 5.5 offers strong reasoning, code navigation, and tool-use capabilities, but it does not automatically know every local operating convention. It may not know where checkpoints should live, which metrics are authoritative, or how to recover the campaign objective after a long session, context compaction, or disconnect.\n\nFor this reason, three complementary [agent skills](https://github.com/nvidia/skills) were also used:\n\n**Brev-etiquette**: Operating guidance for NVIDIA Brev GPU instances. It keeps the repository clean, stores large artifacts such as checkpoints and caches on the intended volume, and handles secrets safely.**Session-memory**: A durable session diary for long-running work. It records the overall goal, subtasks, loaded skills, important files, decisions, progress, and handoff notes.**Autoresearch**: The experiment loop. It preserves the user’s objective, establishes baselines, creates a branch for each hypothesis, logs attempts in a ledger, monitors stop rules, and summarizes results for human review.\n\nThese skills act as structured, reusable workflow instructions. They encode operating context and institutional knowledge so the agent can execute the research loop more reproducibly.\n\n## Prerequisites\n\nThis walkthrough uses the following setup:\n\n- Visual Studio Code\n- Codex plugin with the GPT 5.5 model\n- A Brev instance with one NVIDIA L40S 48 GB GPU\n[NeMo RL repository](https://github.com/NVIDIA-NeMo/RL)(for the Brev launchable; precloned under /home/ubuntu/RL)- Hugging Face and Weights & Biases credentials, if the workflow requires authenticated model, dataset, or logging access\n\nThe following sections validate the three agent capabilities. First, set up the machine and validate the stack. Next, run a goal-driven autoresearch campaign. Finally, use the same agent workflow to implement a research paper and begin a longer validation training run.\n\n## Full-stack autonomy\n\nThis section demonstrates the agent’s ability to automanage the full hardware and software stack. Before starting an autoresearch loop, first validate that the machine, repository, dependencies, model access, and training loop work end to end. This setup validation is the foundation for longer agent-led research runs.\n\n### Step 1: Launch a Brev instance\n\nLaunch a Brev instance with one NVIDIA L40S 48 GB GPU using the [NeMo-RL Autoresearch Launchable](https://brev.nvidia.com/launchable/deploy?launchableID=env-3ECKNnhrpAkQVmcfgA88kFWW7v7).\n\n### Step 2: Connect VS Code to the remote instance\n\nOn Windows, open the SSH config file, typically under `C:\\Users\\<YOUR_USER_NAME>\\.ssh\\config`\n\n, and add the following:\n\n```\nHost Brev-Ubuntu\n    HostName <BREV_IP_ADDRESS>\n\tUser ubuntu\n    IdentityFile C:/Users/<YOUR_USER_NAME>/.brev/brev.pem\n```\n\nFind `BREV_IP_ADDRESS`\n\nlisted at the top of the Brev instance page. In VS Code, select Connect to Host and choose the configured host name, such as Brev-Ubuntu. On macOS and Linux, the Brev key is typically under `~/.brev/brev.pem`\n\n, and the SSH configuration is similar.\n\n### Step 3: Install and configure Codex\n\nFrom VS Code, open the NeMo RL directory under `/home/ubuntu/RL`\n\n. Then install the Codex IDE extension from the VS Code extensions marketplace. After installation, Codex appears in the chat window. Sign in using an API key or ChatGPT account.\n\nCodex can run in several permission modes: Default, Auto-Review, and Full Access. For this workflow, Auto-Review is a reasonable starting point. It allows Codex to run sandboxed commands while adding review for elevated actions. Full access gives broader control over the instance and should only be used when you are comfortable with the security settings.\n\n### Step 4: Prepare the NeMo RL workspace\n\nOpen a VS Code terminal in `/home/ubuntu/RL`\n\n. If needed, create a `.env`\n\nfile for credentials:\n\n```\nexport WANDB_API_KEY=wandb_v1xxx\nexport HF_TOKEN=hf_mqyyy\n```\n\nFamiliarize yourself with the Brev environment. For example, type `df -h`\n\nto find out more about the storage configuration on the virtual instance.\n\n### Step 5: Run a NeMo RL smoke test\n\nNow prompt Codex to set up NeMo RL and run a small RL training smoke test:\n\n```\n/goal Use the nemo-rl-brev-etiquette skill under `./skills`, set up NemoRL to \ndo a smoke test RL training of the Qwen3-VL-2B-Instruct model\n```\n\nThe `/goal`\n\ncommand sets a persistent objective for Codex to work toward. In this run, Codex should load the Brev operating skill, initialize missing NeMo RL submodules, download the Qwen3-VL model from Hugging Face, route large assets such as Hugging Face, Torch, Triton, uv, Ray, logs, and worker caches to `/ephemeral`\n\n, and resolve dependency or configuration issues until a training step completes.\n\nThe elapsed time is approximately 40 minutes to an hour. The goal is not the most optimized NeMo RL setup, but a validated end-to-end path through repository setup, model access, dependency resolution, and training execution.\n\n### Step 6: Persist the setup with session memory\n\nAfter the stack is working, persist the setup state with the following prompt:\n\n```\nNow use the nemo-rl-session-memory skill under `./skills`, persist this setup information\n```\n\nTo resume after a network disconnect or VS Code restart, you can load the latest session with the following prompt:\n\n```\nUse the nemo-rl-session-memory skill under `./skills`, load the latest session, and provide me with a summary\n```\n\nThis provides Codex with the previous goal, environment assumptions, loaded skills, important paths, and recent command outcomes before continuing.\n\n## Goal-driven autoresearch\n\nWith the stack validated, the next step is to move from setup to a goal-driven campaign. In this mode, the human provides the objective and budget, and Codex manages the loop: baseline, hypotheses, experiments, metrics, and summary.\n\n### Step 7: Create a new NeMo Gym environment\n\nBefore launching the campaign, ask Codex to implement a new RL environment. The example below creates a visual counting task for a VLM:\n\n```\n/goal Now using the Nemo gym circle click environment as a reference, \nimplement a new Gym environment named \"star count\", that teaches \na model to count the number of stars of different colors on a canvas of \ndifferent resolutions. Show me several examples\n```\n\nThe resulting environment generates images with colored stars at varying canvas resolutions and asks the model to count the number of stars by color.\n\n### Step 8: Start the autoresearch campaign\n\nNext, start a time-bounded campaign using the following prompt:\n\n```\nGreat. /goal Now use the nemo-rl-autoresearch and nemo-rl-session-memory skill under `./skills`, start a new \nauto research session, goal: Train the Qwen3-VL-2B-Instruct model to high accuracy, using SFT algorithm \non the star count gym environment. Start with profiling the model out of the box accuracy. Time budget: 5h\n```\n\nThe prompt must contain the objective, method, task environment, baseline requirement, and time budget. The autoresearch skill translates these into a campaign with measurable stop rules and an experiment ledger.\n\nIn this run, Codex wires up the [synthetically generated data](https://www.nvidia.com/en-us/glossary/synthetic-data-generation/) from the star count environment into the NeMo RL SFT path. SFT is used in this case because a prior autoresearch session found that GRPO did not improve accuracy on this task, while the Codex proposed SFT route later proved effective.\n\nIn one experiment, Codex achieved the best result in under an hour. The Codex final report when prompted to provide a brief summary is shown below:\n\n```\nThe star_count SFT campaign trained Qwen/Qwen3-VL-2B-Instruct on the new NeMo Gym visual \ncounting environment using LoRA SFT. We first profiled the model out of the box on a fixed 64-example \nheld-out eval split, where it reached 25.0% exact accuracy with mean reward 0.705208. After setting up the \nenvironment, data generation, evaluator, and SFT recipe, the first LoRA smoke run improved accuracy to 78.125%.\nScaling the SFT data and training steps produced the main gains. A 2048-train / 256-val, 160-step run \nreached 95.3125% exact accuracy, and the best 4096-train / 512-val, 320-step run reached 96.875% exact \naccuracy with mean reward 0.992188. The best checkpoint was step_240, which slightly beat the final checkpoint \non partial-credit reward while tying exact accuracy.\nOverall, accuracy improved from 25.0% to 96.875%, a gain of 71.875 percentage points. The remaining errors \nwere not formatting failures; they were near-miss off-by-one count mistakes on individual colors, suggesting the \nnext best improvement would be targeted hard-case augmentation for dense or visually ambiguous scenes.\n```\n\nThe remaining errors were near-miss off-by-one count mistakes on individual colors. This suggests targeted hard-case augmentation for dense or visually ambiguous scenes as the next experiment.\n\nThis captures the core value of the agent-led research loop. Specifically, Codex did not run only one command. Instead, it profiled a baseline, built the environment and data path, ran a validation experiment, scaled data and training steps, evaluated checkpoints, analyzed errors, and proposed a plausible next direction for human review.\n\n## Paper-to-code\n\nPaper-to-code is an idea pioneered by OpenAI [PaperBench](https://openai.com/index/paperbench/) to auto-implement research ideas from papers. Speeding this process, Codex reads a paper, derives an implementation plan, writes and tests the code, and then kicks off training.\n\n### Step 9: Give Codex a paper and an implementation target\n\nFor this example, the target paper is [LLMs Can Learn to Reason via Off-Policy RL](https://arxiv.org/abs/2602.19362). The paper introduces Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL), a long-lag off-policy RL method. Instead of treating the training-inference policy mismatch as a bug to eliminate, OAPL uses the lagged generation policy as part of the objective, enabling stable and efficient training even when the training and generation models are substantially out of sync.\n\nThis is a useful test case because it requires both algorithmic implementation and system-level judgment. The loss computation changes, and the long-lag setup also affects how training and generation are coordinated.\n\nThe composite prompt is:\n\n```\n/goal Start a new session with nemo-rl-session-memory skill under `./skills`. First, read \nthis paper https://arxiv.org/abs/2602.19362 in pdf format, form a plan then implement it. \nWrite and run unit tests to satisfaction. Write document. Then start a new autoresearch \nsession with the nemo-rl-autoresearch skill, train a Qwen3-1.7B model with this algorithm \non the DAPO-math-17K dataset. Time budget: 10h\n```\n\nCodex breaks the request into the following stages:\n\n- Read the PDF\n- Extract the algorithm\n- Inspect NeMo RL hook points\n- Produce an implementation plan\n- Translate the algorithm into code\n- Add unit tests\n- Write documentation\n- Start a validation campaign on Qwen3-1.7B with DAPO-math-17K dataset\n\nThis prompt is intentionally demanding. In practice, the safer workflow is staged. First use Codex for reading and planning with human review, then approve implementation, documentation, and unit tests, and only then start a longer autoresearch validation campaign.\n\nAdditional validation can include self-consistency, where Codex implements the algorithm twice on separate branches and compares decisions, or an agentic cross-check, where another frontier coding agent reviews the paper, implementation, tests, and results.\n\n### Step 10: Training validation and follow-up research question\n\nFigure 3 shows the training progress of the OAPL algorithm compared with a GRPO variant, DAPO, on the Qwen3-1.7B model and the DAPO-math-17K dataset.\n\nIn this run, OAPL reached higher accuracy with fewer training steps, consistent with the paper’s central claim about effective long-lag off-policy training. The right panel also shows that OAPL used significantly more tokens than DAPO, which creates a natural follow-up research question: Why does OAPL become more verbose, and what algorithmic changes could promote brevity without sacrificing accuracy?\n\n## Tips and best practices for autoresearch workflows\n\nWhile increasingly powerful, frontier agents still exhibit a few failure modes to watch out for, including:\n\n**Context drift:** Long sessions can pass through multiple context compaction cycles. Without durable memory, the agent may forget the objective, loaded skills, or stop rules.**Local housekeeping:** Logs, checkpoints, datasets, and caches can scatter across the filesystem unless the machine-specific etiquette is explicit.**Steering drift:** Mid-run instructions are useful, but follow-up questions sometimes derail an autoresearch session, leading to early termination.**Low-signal loops:** Sometimes Codex gets stuck in a trial loop. For example, training for a small number of steps and/or using a tiny batch, which cannot provide enough signal. A human researcher should easily spot this unproductive behavior and steer Codex away.\n\nSeveral practices we have found useful during the building and testing of the autoresearch skills for NeMo RL and NeMo Gym include the following:\n\n- Treat autoresearch skills as evolving workflow instructions. Codex can help update them over time to reflect institutional conventions, best practices, and lessons learned from previous runs.\n- While Codex is resourceful and helpful, for tough research questions, human guidance and judgment are still mission-critical.\n- It’s useful to go through an explicit brainstorming and planning process to validate ideas before a long-running autoresearch execution phase.\n- Think of Codex as a capable intern or junior colleague. This implies that the human researcher doesn’t necessarily need to maintain a deep, single-line-of-code-level understanding of every implementation, but still has methodologies to validate to be confident in the work carried out.\n- Set explicit budget constraints before starting a long-running autoresearch session, such as a time limit, GPU-hour budget, or maximum number of experiments. This keeps the agent focused on the highest-value hypotheses and prevents overly broad experiment sweeps.\n\n## Get started with autoresearch\n\nAgents are not replacing researchers, but rather supporting a productive division of labor. The agent handles much of the setup, debugging, logging, and experiment loop while also providing insights and proposals for the next steps. The researcher defines the goal, provides domain judgment, steers the campaign, and decides which results are meaningful.\n\nWith NeMo RL, NeMo Gym, Brev, Codex, and a small set of operating skills, researchers can reduce the time and compute cost spent on setup, debugging, and repeated experiment execution. This shifts more of the workflow toward deciding which experiments are worth running next and how to interpret the results.\n\nTo get started with autoresearch for reinforcement learning, check out these related resources:\n\n- Try the environment from the tutorial with the\n[Brev Launchable](https://brev.nvidia.com/launchable/deploy?launchableID=env-3ECKNnhrpAkQVmcfgA88kFWW7v7). - Review the\n[NVIDIA-verified Agent Skills documentation](https://docs.nvidia.com/skills)and browse the[NVIDIA/skills GitHub repo](https://github.com/NVIDIA/skills), including the[NeMo RL Auto Research skill](https://github.com/NVIDIA/skills/tree/main/skills/nemo-rl-auto-research). - Explore\n[NVIDIA NeMo RL](https://github.com/nvidia-nemo/rl)and[NVIDIA NeMo Gym](https://github.com/NVIDIA-NeMo/Gym)for RL training and gym environments.", "url": "https://wpnews.pro/news/how-to-run-an-autoresearch-workflow-with-rl-agent-skills-and-nvidia-nemo", "canonical_source": "https://developer.nvidia.com/blog/how-to-run-an-autoresearch-workflow-with-rl-agent-skills-and-nvidia-nemo/", "published_at": "2026-07-14 16:00:00+00:00", "updated_at": "2026-07-14 16:21:02.492810+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-research", "ai-tools"], "entities": ["NVIDIA", "NeMo RL", "NeMo Gym", "Codex", "GPT 5.5", "Andrej Karpathy", "Qwen3-VL-2B-Instruct", "NVIDIA Brev"], "alternates": {"html": "https://wpnews.pro/news/how-to-run-an-autoresearch-workflow-with-rl-agent-skills-and-nvidia-nemo", "markdown": "https://wpnews.pro/news/how-to-run-an-autoresearch-workflow-with-rl-agent-skills-and-nvidia-nemo.md", "text": "https://wpnews.pro/news/how-to-run-an-autoresearch-workflow-with-rl-agent-skills-and-nvidia-nemo.txt", "jsonld": "https://wpnews.pro/news/how-to-run-an-autoresearch-workflow-with-rl-agent-skills-and-nvidia-nemo.jsonld"}}