How to Run an Autoresearch Workflow with RL Agent Skills and NVIDIA NeMo

NVIDIA released a guide on running an autoresearch workflow using RL agent skills and the NeMo framework, enabling AI agents to autonomously set up experiments, train models, and iterate on research goals. The workflow, tested with Codex on NeMo RL and NeMo Gym, improved a vision language model's accuracy from 25% to 96.9% on a visual counting task. This approach aims to automate repetitive research tasks while keeping human researchers in control of strategy and decision-making.

Coding AI agents https://www.nvidia.com/en-us/ai/ are becoming practical operators for long-running machine learning ML workflows. They can inspect repositories, set up runtimes, resolve build issues, launch experiments, monitor execution, analyze metrics, and summarize results. For reinforcement learning RL https://www.nvidia.com/en-us/glossary/reinforcement-learning/ research, this matters because meaningful metrics often appear only after the essential experiment infrastructure is in place. Autoresearch https://github.com/karpathy/autoresearch is an open source Python project by Andrej Karpathy for automating AI and ML model training. With this approach, autonomous AI agents https://www.nvidia.com/en-us/glossary/ai-agents/ translate high-level goals into hypotheses, edit and test a real codebase, keep metric-improving ideas, and hand results back to the human researcher. This enables teams to build specialized domain agents starting from a performant open model, such as NVIDIA Nemotron https://developer.nvidia.com/topics/ai/nemotron , then improve it with RL against measurable domain tasks while maintaining control over data, IP, deployment, and the training workflow. This post walks through how to run a lightweight, no-code-from-the-user-side, skill-based autoresearch workflow. It was tested using a frontier coding agent Codex with GPT 5.5 on NVIDIA NeMo RL https://github.com/nvidia-nemo/rl and NVIDIA NeMo Gym https://github.com/NVIDIA-NeMo/Gym , using an NVIDIA Brev https://brev.nvidia.com/logout GPU instance. Building an autonomous RL research workflow The workflow demonstrates the following three agent capabilities: Full-stack autonomy : Setting up the software stack; resolving dependencies; managing GPU memory, disk space, and checkpoints; launching experiments; monitoring runs; and debugging issues. Goal-driven autoresearch : Profiling a baseline, proposing or following hypotheses, launching experiments, analyzing metrics, and iterating toward a research objective. Paper-to-code : Reading a paper, forming an implementation plan, translating the algorithm into code, adding tests, and starting validation training. In the examples provided in this post, Codex first brings up a full NeMo RL and NeMo Gym stack for a vision language model VLM https://www.nvidia.com/en-us/glossary/vision-language-models/ RL training smoke test. It then conceptualizes and creates a novel NeMo Gym visual counting environment from scratch, and trains the Qwen3-VL-2B-Instruct model, increasing its accuracy from 25.0% to 96.9% on the task. Finally, it implements an off-policy RL algorithm from a research paper and begins a 10-hour validation training campaign. The goal of autoresearch is not to remove the researcher from the loop, but instead to hand off the repetitive setup and iteration work to the agent. Researchers are still responsible for setting goals, reviewing milestones, steering strategy, and making the final decision. Using NeMo RL, NeMo Gym, and agent skills for autoresearch NeMo RL https://github.com/nvidia-nemo/rl and NeMo Gym https://github.com/NVIDIA-NeMo/Gym are open source libraries that are part of the NVIDIA NeMo framework https://github.com/NVIDIA-NeMo . NeMo RL is built on AutoModel https://github.com/nvidia-nemo/automodel , Megatron-Bridge https://github.com/NVIDIA-NeMo/Megatron-Bridge , and vLLM https://github.com/vllm-project/vllm for post-training LLMs and VLMs, orchestrated by Ray https://github.com/ray-project/ray . It supports workflows such as GRPO, DPO, SFT, and reward model training, with recipe-driven configuration that can scale from small validation runs to distributed training. NeMo Gym provides environments where models can interact with tasks, receive rewards, and learn through live-generated experience. Together, NeMo RL and NeMo Gym provide a useful substrate for agent-led RL research. An agent can inspect recipes, wire up a new environment, run a baseline, adjust training parameters, and compare results in the same repository. Codex with GPT 5.5 offers strong reasoning, code navigation, and tool-use capabilities, but it does not automatically know every local operating convention. It may not know where checkpoints should live, which metrics are authoritative, or how to recover the campaign objective after a long session, context compaction, or disconnect. For this reason, three complementary agent skills https://github.com/nvidia/skills were also used: Brev-etiquette : Operating guidance for NVIDIA Brev GPU instances. It keeps the repository clean, stores large artifacts such as checkpoints and caches on the intended volume, and handles secrets safely. Session-memory : A durable session diary for long-running work. It records the overall goal, subtasks, loaded skills, important files, decisions, progress, and handoff notes. Autoresearch : The experiment loop. It preserves the user’s objective, establishes baselines, creates a branch for each hypothesis, logs attempts in a ledger, monitors stop rules, and summarizes results for human review. These skills act as structured, reusable workflow instructions. They encode operating context and institutional knowledge so the agent can execute the research loop more reproducibly. Prerequisites This walkthrough uses the following setup: - Visual Studio Code - Codex plugin with the GPT 5.5 model - A Brev instance with one NVIDIA L40S 48 GB GPU NeMo RL repository https://github.com/NVIDIA-NeMo/RL for the Brev launchable; precloned under /home/ubuntu/RL - Hugging Face and Weights & Biases credentials, if the workflow requires authenticated model, dataset, or logging access The following sections validate the three agent capabilities. First, set up the machine and validate the stack. Next, run a goal-driven autoresearch campaign. Finally, use the same agent workflow to implement a research paper and begin a longer validation training run. Full-stack autonomy This section demonstrates the agent’s ability to automanage the full hardware and software stack. Before starting an autoresearch loop, first validate that the machine, repository, dependencies, model access, and training loop work end to end. This setup validation is the foundation for longer agent-led research runs. Step 1: Launch a Brev instance Launch a Brev instance with one NVIDIA L40S 48 GB GPU using the NeMo-RL Autoresearch Launchable https://brev.nvidia.com/launchable/deploy?launchableID=env-3ECKNnhrpAkQVmcfgA88kFWW7v7 . Step 2: Connect VS Code to the remote instance On Windows, open the SSH config file, typically under C:\Users\<YOUR USER NAME \.ssh\config , and add the following: Host Brev-Ubuntu HostName <BREV IP ADDRESS User ubuntu IdentityFile C:/Users/<YOUR USER NAME /.brev/brev.pem Find BREV IP ADDRESS listed at the top of the Brev instance page. In VS Code, select Connect to Host and choose the configured host name, such as Brev-Ubuntu. On macOS and Linux, the Brev key is typically under ~/.brev/brev.pem , and the SSH configuration is similar. Step 3: Install and configure Codex From VS Code, open the NeMo RL directory under /home/ubuntu/RL . Then install the Codex IDE extension from the VS Code extensions marketplace. After installation, Codex appears in the chat window. Sign in using an API key or ChatGPT account. Codex can run in several permission modes: Default, Auto-Review, and Full Access. For this workflow, Auto-Review is a reasonable starting point. It allows Codex to run sandboxed commands while adding review for elevated actions. Full access gives broader control over the instance and should only be used when you are comfortable with the security settings. Step 4: Prepare the NeMo RL workspace Open a VS Code terminal in /home/ubuntu/RL . If needed, create a .env file for credentials: export WANDB API KEY=wandb v1xxx export HF TOKEN=hf mqyyy Familiarize yourself with the Brev environment. For example, type df -h to find out more about the storage configuration on the virtual instance. Step 5: Run a NeMo RL smoke test Now prompt Codex to set up NeMo RL and run a small RL training smoke test: /goal Use the nemo-rl-brev-etiquette skill under ./skills , set up NemoRL to do a smoke test RL training of the Qwen3-VL-2B-Instruct model The /goal command sets a persistent objective for Codex to work toward. In this run, Codex should load the Brev operating skill, initialize missing NeMo RL submodules, download the Qwen3-VL model from Hugging Face, route large assets such as Hugging Face, Torch, Triton, uv, Ray, logs, and worker caches to /ephemeral , and resolve dependency or configuration issues until a training step completes. The elapsed time is approximately 40 minutes to an hour. The goal is not the most optimized NeMo RL setup, but a validated end-to-end path through repository setup, model access, dependency resolution, and training execution. Step 6: Persist the setup with session memory After the stack is working, persist the setup state with the following prompt: Now use the nemo-rl-session-memory skill under ./skills , persist this setup information To resume after a network disconnect or VS Code restart, you can load the latest session with the following prompt: Use the nemo-rl-session-memory skill under ./skills , load the latest session, and provide me with a summary This provides Codex with the previous goal, environment assumptions, loaded skills, important paths, and recent command outcomes before continuing. Goal-driven autoresearch With the stack validated, the next step is to move from setup to a goal-driven campaign. In this mode, the human provides the objective and budget, and Codex manages the loop: baseline, hypotheses, experiments, metrics, and summary. Step 7: Create a new NeMo Gym environment Before launching the campaign, ask Codex to implement a new RL environment. The example below creates a visual counting task for a VLM: /goal Now using the Nemo gym circle click environment as a reference, implement a new Gym environment named "star count", that teaches a model to count the number of stars of different colors on a canvas of different resolutions. Show me several examples The resulting environment generates images with colored stars at varying canvas resolutions and asks the model to count the number of stars by color. Step 8: Start the autoresearch campaign Next, start a time-bounded campaign using the following prompt: Great. /goal Now use the nemo-rl-autoresearch and nemo-rl-session-memory skill under ./skills , start a new auto research session, goal: Train the Qwen3-VL-2B-Instruct model to high accuracy, using SFT algorithm on the star count gym environment. Start with profiling the model out of the box accuracy. Time budget: 5h The prompt must contain the objective, method, task environment, baseline requirement, and time budget. The autoresearch skill translates these into a campaign with measurable stop rules and an experiment ledger. In this run, Codex wires up the synthetically generated data https://www.nvidia.com/en-us/glossary/synthetic-data-generation/ from the star count environment into the NeMo RL SFT path. SFT is used in this case because a prior autoresearch session found that GRPO did not improve accuracy on this task, while the Codex proposed SFT route later proved effective. In one experiment, Codex achieved the best result in under an hour. The Codex final report when prompted to provide a brief summary is shown below: The star count SFT campaign trained Qwen/Qwen3-VL-2B-Instruct on the new NeMo Gym visual counting environment using LoRA SFT. We first profiled the model out of the box on a fixed 64-example held-out eval split, where it reached 25.0% exact accuracy with mean reward 0.705208. After setting up the environment, data generation, evaluator, and SFT recipe, the first LoRA smoke run improved accuracy to 78.125%. Scaling the SFT data and training steps produced the main gains. A 2048-train / 256-val, 160-step run reached 95.3125% exact accuracy, and the best 4096-train / 512-val, 320-step run reached 96.875% exact accuracy with mean reward 0.992188. The best checkpoint was step 240, which slightly beat the final checkpoint on partial-credit reward while tying exact accuracy. Overall, accuracy improved from 25.0% to 96.875%, a gain of 71.875 percentage points. The remaining errors were not formatting failures; they were near-miss off-by-one count mistakes on individual colors, suggesting the next best improvement would be targeted hard-case augmentation for dense or visually ambiguous scenes. The remaining errors were near-miss off-by-one count mistakes on individual colors. This suggests targeted hard-case augmentation for dense or visually ambiguous scenes as the next experiment. This captures the core value of the agent-led research loop. Specifically, Codex did not run only one command. Instead, it profiled a baseline, built the environment and data path, ran a validation experiment, scaled data and training steps, evaluated checkpoints, analyzed errors, and proposed a plausible next direction for human review. Paper-to-code Paper-to-code is an idea pioneered by OpenAI PaperBench https://openai.com/index/paperbench/ to auto-implement research ideas from papers. Speeding this process, Codex reads a paper, derives an implementation plan, writes and tests the code, and then kicks off training. Step 9: Give Codex a paper and an implementation target For this example, the target paper is LLMs Can Learn to Reason via Off-Policy RL https://arxiv.org/abs/2602.19362 . The paper introduces Optimal Advantage-based Policy Optimization with Lagged Inference policy OAPL , a long-lag off-policy RL method. Instead of treating the training-inference policy mismatch as a bug to eliminate, OAPL uses the lagged generation policy as part of the objective, enabling stable and efficient training even when the training and generation models are substantially out of sync. This is a useful test case because it requires both algorithmic implementation and system-level judgment. The loss computation changes, and the long-lag setup also affects how training and generation are coordinated. The composite prompt is: /goal Start a new session with nemo-rl-session-memory skill under ./skills . First, read this paper https://arxiv.org/abs/2602.19362 in pdf format, form a plan then implement it. Write and run unit tests to satisfaction. Write document. Then start a new autoresearch session with the nemo-rl-autoresearch skill, train a Qwen3-1.7B model with this algorithm on the DAPO-math-17K dataset. Time budget: 10h Codex breaks the request into the following stages: - Read the PDF - Extract the algorithm - Inspect NeMo RL hook points - Produce an implementation plan - Translate the algorithm into code - Add unit tests - Write documentation - Start a validation campaign on Qwen3-1.7B with DAPO-math-17K dataset This prompt is intentionally demanding. In practice, the safer workflow is staged. First use Codex for reading and planning with human review, then approve implementation, documentation, and unit tests, and only then start a longer autoresearch validation campaign. Additional validation can include self-consistency, where Codex implements the algorithm twice on separate branches and compares decisions, or an agentic cross-check, where another frontier coding agent reviews the paper, implementation, tests, and results. Step 10: Training validation and follow-up research question Figure 3 shows the training progress of the OAPL algorithm compared with a GRPO variant, DAPO, on the Qwen3-1.7B model and the DAPO-math-17K dataset. In this run, OAPL reached higher accuracy with fewer training steps, consistent with the paper’s central claim about effective long-lag off-policy training. The right panel also shows that OAPL used significantly more tokens than DAPO, which creates a natural follow-up research question: Why does OAPL become more verbose, and what algorithmic changes could promote brevity without sacrificing accuracy? Tips and best practices for autoresearch workflows While increasingly powerful, frontier agents still exhibit a few failure modes to watch out for, including: Context drift: Long sessions can pass through multiple context compaction cycles. Without durable memory, the agent may forget the objective, loaded skills, or stop rules. Local housekeeping: Logs, checkpoints, datasets, and caches can scatter across the filesystem unless the machine-specific etiquette is explicit. Steering drift: Mid-run instructions are useful, but follow-up questions sometimes derail an autoresearch session, leading to early termination. Low-signal loops: Sometimes Codex gets stuck in a trial loop. For example, training for a small number of steps and/or using a tiny batch, which cannot provide enough signal. A human researcher should easily spot this unproductive behavior and steer Codex away. Several practices we have found useful during the building and testing of the autoresearch skills for NeMo RL and NeMo Gym include the following: - Treat autoresearch skills as evolving workflow instructions. Codex can help update them over time to reflect institutional conventions, best practices, and lessons learned from previous runs. - While Codex is resourceful and helpful, for tough research questions, human guidance and judgment are still mission-critical. - It’s useful to go through an explicit brainstorming and planning process to validate ideas before a long-running autoresearch execution phase. - Think of Codex as a capable intern or junior colleague. This implies that the human researcher doesn’t necessarily need to maintain a deep, single-line-of-code-level understanding of every implementation, but still has methodologies to validate to be confident in the work carried out. - Set explicit budget constraints before starting a long-running autoresearch session, such as a time limit, GPU-hour budget, or maximum number of experiments. This keeps the agent focused on the highest-value hypotheses and prevents overly broad experiment sweeps. Get started with autoresearch Agents are not replacing researchers, but rather supporting a productive division of labor. The agent handles much of the setup, debugging, logging, and experiment loop while also providing insights and proposals for the next steps. The researcher defines the goal, provides domain judgment, steers the campaign, and decides which results are meaningful. With NeMo RL, NeMo Gym, Brev, Codex, and a small set of operating skills, researchers can reduce the time and compute cost spent on setup, debugging, and repeated experiment execution. This shifts more of the workflow toward deciding which experiments are worth running next and how to interpret the results. To get started with autoresearch for reinforcement learning, check out these related resources: - Try the environment from the tutorial with the Brev Launchable https://brev.nvidia.com/launchable/deploy?launchableID=env-3ECKNnhrpAkQVmcfgA88kFWW7v7 . - Review the NVIDIA-verified Agent Skills documentation https://docs.nvidia.com/skills and browse the NVIDIA/skills GitHub repo https://github.com/NVIDIA/skills , including the NeMo RL Auto Research skill https://github.com/NVIDIA/skills/tree/main/skills/nemo-rl-auto-research . - Explore NVIDIA NeMo RL https://github.com/nvidia-nemo/rl and NVIDIA NeMo Gym https://github.com/NVIDIA-NeMo/Gym for RL training and gym environments.