Turn production evidence into prioritized issues and custom evaluations that help you improve prompts, tools, models, memory, skills, and harness code.
When you build an agent for your domain, generic benchmark scores may not explain its failures. An agent might choose the wrong tool, follow the wrong workflow branch, miss relevant context, use an unnecessarily expensive model, or complete a task without meeting a domain-specific requirement.
Improving an agent requires more than model tuning. Prompts, tools, routing, memory, skills, control flow, and other code all shape production reliability. Traces help you investigate failures, while evaluations help you measure whether a change improves agent behavior.
This tutorial walks you through one complete optimization cycle: use NVIDIA NeMo Compass to identify a failure in recorded traces, create an evaluation with NVIDIA NeMo Eval Author, update the agent’s instructions, and compare results on development and held-out tasks. Follow along with the NVIDIA NemoClaw blueprint for agent optimization.
The blueprint uses Hermes Agent in an NVIDIA OpenShell sandbox, with recorded traces from a fictional company.
The agent harness is the optimization surface #
One wrong tool call can derail an entire task. The cause may be an unclear tool description, a routing rule, or missing context in the agent harness. A trace helps identify where the run went wrong.
Our example agent is a Hermes enterprise assistant for a fictional company, connected through Model Context Protocol (MCP) to the company’s chat, email, files, and tickets. We asked it to “draft a message with project updates.” It drafted the message but also called the send_message tool without user confirmation. Addressing this error requires examining both the harness instructions and tool execution.
Prerequisites
Use a Linux machine running Docker natively. The recommended configuration is Ubuntu 24.04 with 8 vCPUs and 32 GB of RAM. You also need:
- OpenShell 0.1.2 and Harbor 0.22.0
- An NVIDIA API key from build.nvidia.com
- A coding agent that NeMo Eval Author supports, such as Codex or Claude Code
Follow the full step-by-step tutorial on GitHub or get started with an NVIDIA Brev launchable.
NeMo Compass: Find what to fix first #
The NemoClaw blueprint for agent optimization uses three libraries: NeMo Compass, NeMo Eval Author, and NVIDIA NeMo Relay.
NeMo Compass analyzes stored agent traces and evaluation evidence to find recurring problems, identify evaluation gaps, and prioritize improvements. Rather than requiring you to inspect each run, NeMo Compass reduces a large trace corpus to a structured backlog of issues, supporting examples, and recommended areas for improvement or experimentation.
Each Hermes run in the example is recorded as one JSON trace per line. We pointed NeMo Compass at the 42 traces in the repository, along with ETHOS.md, a short file that defines the assistant’s expected behavior.
compass --config configs/trace-analyst.yaml \
--trace.filesystem.path traces/world-v3/production/insights.jsonl
NeMo Compass returns findings based on multiple signals:
- Signs of unusual behavior
- How the agent uses tools
- Divergence from the agent’s intended behavior
NeMo Compass also provides trace IDs that link each signal to the relevant agent runs.
These trace IDs support the overall finding. In these runs, the agent sent messages without user confirmation. NeMo Compass identified this error by comparing the agent’s actions with the rules in ETHOS.md.
In our internal benchmarks, running the full set of NeMo Compass signals against a dataset of thousands of traces identified anomalies that a coding agent assigned a similar task missed.
Build evaluations around real agent work with NeMo Eval Author #
After finding an issue with NeMo Compass, create a test that reproduces the problem. NeMo Eval Author provides coding agent skills and workflows to help you create these tests. NeMo Eval Author supports two agent evaluation formats: Harbor evaluations and NVIDIA NeMo Gym evaluations.
Install NeMo Eval Author’s skills in your coding agent and instruct it to use them. Provide the NeMo Compass findings, raw traces, the agent’s ETHOS.md file, and access to your agent repository as context. The evaluations should measure agent behavior relevant to your agent’s requirements. Review and tune the generated evaluations to make sure they reflect those requirements.
In our example, the coding agent created four Harbor evaluations with tasks similar to “draft a response.” NeMo Eval Author also creates verifiers that determine whether an agent meets specific criteria. In this example, the verifier checks whether the agent called the send_message tool without human confirmation and marks that run as a failure.
NeMo Eval Author can help you create custom evaluations before production data exists, using agent requirements from ETHOS.md and code details such as tool definitions and existing workflows. As real traces and known failures become available, use NeMo Eval Author to expand your evaluation suite, so it continues to reflect how your agent is used.
Change one part of the harness #
After identifying an issue and writing tests for it, update the harness. In our example, we changed the Hermes agent’s SOUL.md (a Hermes concept similar to a system prompt):
Requests to write, draft, compose, prepare, or suggest a message authorize preparation only. Provide the draft for review and stop before sending it. A token returned by a preparation tool is a technical capability to execute an action. It does not establish the user's consent to that action.
Harness changes can range from simple fixes like this to more complex middleware implementations, plugin extensions, and other agent harness patterns. Provide the coding agent with the NeMo Compass outputs, your agent’s ETHOS.md file, and your agent implementation. This context grounds the coding agent’s proposal. Earlier posts cover harness optimizations with LangChain Deep Agents and Nous Research Hermes.
Measure before and after #
After updating the agent, we reran the evaluation tasks created by NeMo Eval Author. Two tasks, labeled “Held out,” were withheld from the coding agents proposing harness changes. These held-out tasks help assess whether the changes generalize beyond the development tasks.
| Evaluation set | Baseline passes/runs | Updated-agent passes/runs | Tool calls, before/after |
|---|---|---|---|
| Development, 2 tasks x 3 runs | 3/6 | 6/6 | 10 to 6 |
| Held out, 2 tasks x 3 runs | 4/6 | 6/6 | 28 to 15 |
Table 1. Evaluation results before and after the harness change
In our example, we started with a small set of evaluation tasks focused narrowly on the NeMo Compass findings. For your production agent, build an evaluation suite that covers a broad range of behaviors. In our experience, production suites often contain hundreds to a few thousand examples. Accept changes only when they improve both the test cases and the held-out set. If results are inconclusive, or a change fixes one problem but introduces a regression, run NeMo Compass on the new traces from the proposed agent. The coding agent can use these new insights to refine the change. As you iterate on this process, ensure that the held-out cases remain hidden from the coding agent to avoid overfitting.
Run the loop again #
After a successful harness change, repeat the analysis to identify the next issue. We ran NeMo Compass on the updated agent’s scored traces, and it no longer reported the sending-without-confirmation finding. It identified another issue: for some queries, the agent found the answer but failed to ground its response in the retrieved evidence. That finding is the focus of the next round of testing and fixes.
Apply the optimization loop to an engineering agent #
The same trace-to-evaluation loop can be applied to agents that perform more complex technical tasks. In an engineering workflow, evaluations need to measure engineering outcomes as well as the efficiency of the agent’s tool use.
A car door design example illustrates how NVIDIA NeMo tools can support this process. The example uses LEO, the engineering Virtual Companion from Dassault Systèmes, to propose design changes and test them through simulation with the goal of creating a stronger, safer car door.
NeMo Compass analyzes the agent’s traces to identify areas for improvement. A coding agent uses the findings to make targeted prompt and configuration changes, and evaluations measure the updated agent’s performance in the engineering environment. Additional scenarios check whether those changes introduce regressions.
In this example, NeMo Compass identified a recurring issue in which the agent placed QueryModel arguments outside the expected options field. It returned an empty error message, leaving the agent unable to recover. The issue appeared across multiple traces. Based on this finding, the coding agent added a validation step that catches misplaced arguments and names in tool calls. Compared with the baseline, malformed tool calls fell from 142 to zero. Accuracy was unchanged, and no regressions were observed in the remaining scenarios.
During the initial pass, the optimization loop ran overnight without human intervention. In approximately 12 hours, the reported results showed a 28% relative improvement in the door design workflow’s overall evaluation score and 35% fewer tool calls per run across six tested scenarios, with no regressions reported in the other scenarios.
The example connects the NeMo workflow to two measures of agent performance: task quality and execution efficiency. It follows the process described in this tutorial—use traces to identify what to change, then use custom evaluations to assess the targeted improvement and check for regressions before adopting the change.
Try the NemoClaw blueprint for agent optimization #
Start with the NemoClaw blueprint for agent optimization, the workplace assistant example from this post, or the CAD guide.
Run the example agent and establish a baseline score. Then complete one optimization cycle: find an issue with NeMo Compass, build a test for it with NeMo Eval Author, and try one change to the harness. After that, point the same loop at your own agent.