You ship an agent. It mostly works. Then the feedback loop becomes a support ticket: a user got the wrong answer, a tool call looped four times, a refund went out that shouldn’t have. Standard observability tells you the system was up and the latency was fine. It can’t tell you the agent chose the wrong tool, used stale context, or made a frustrated user give up. And when you do find the problem, writing realistic test cases to validate a fix means hours of digging through traces or inventing scenarios you hope resemble reality.
Today we’re launching two capabilities, part of DigitalOcean Agent Learning Services, that address both halves of that problem. Signals watches production traffic on DigitalOcean Managed Agents and DigitalOcean Inference Engine, and automatically highlights potential problems: frustrated users, failing tools, infrastructure constraints. Simulations, a new capability of DigitalOcean Evaluations, lets you test a change through realistic multi-turn scenarios, grounded in that same production traffic, on the same harness and tools your agent uses in production, before it reaches a user.
Both run natively on the infrastructure you already use to operate models and agents on DigitalOcean, and both feed the same Evaluations grader you already trust. Signals shows you what went wrong, and Simulations lets you test the fix before it ships.
Agent and inference failures often look technically successful. The tool call returned 200; it just returned the wrong region. The conversation completed; the user left annoyed. Signals catches that class of failure at scale by leveraging the infrastructure provided by DigitalOcean Inference Engine, Harness Runtime and Action Gateway. Signals stitches conversation turns into sessions and classifies each one across three dimensions. Interaction signals that track user sentiment (corrections, escalations, users quitting), Execution signals that track tool reliability (invalid arguments, retry loops), and Environmental signals that track infrastructure (timeouts, rate limits, context overflow). Trends show up on the Signals view on Harness Runtime and Serverless Inference, and you can drill from any trend into the sessions behind it.
Raw trace volume can exceed 100 GB/s, so grading every session with an LLM judge is prohibitively expensive and manual review doesn’t scale at all. Signals instead runs an inexpensive first-pass triage to find the sessions worth a closer look. The method, described in our paper Signals: Trajectory Sampling and Triage for Agentic Interactions, achieved an 82% informativeness rate and a 1.52x efficiency gain over random sampling in a controlled study on the τ-bench benchmark. Flagged sessions become evaluation scenarios in one click, so a cluster of wrong-region database queries becomes a targeted eval with no manual test-data prep. Signals is opt-in, is designed to redact PII before processing, and keeps traces and trace-derived evaluations within your DigitalOcean Team boundary; they’re never used for cross-customer model training. It works best when your agent runs on Managed Agents and your models use DigitalOcean Inference Engine.
Evaluations scores agent and model performance against defined test cases, which is the right check for single responses. But agents fail across turns: a prompt tweak fixes one step and quietly breaks a later one, a fine-tuned checkpoint scores well on its training objective and regresses on real multi-turn behavior. Hand-built test cases are narrow, synthetic sets aren’t grounded in your agent or your users, and live traffic is realistic at the cost of real mistakes. Simulations give you that validation layer without the tedious build. It starts with crafting the right scenario to test your model or agent candidate: every scenario starts from your agent’s configuration, so it only exercises goals and tools the agent actually supports. When Signals is enabled, scenario generation is weighted toward where production traffic already shows friction and toward what your change is most likely to affect; when there’s no production data yet, guided refinement builds a baseline set from your configuration. Or you can just describe one in plain language: “a refund request from an angry customer with missing order details.”
A user simulator plays the persona turn by turn against your candidate, which runs in an isolated sandbox on Harness Runtime with the same configuration and Action Gateway tools it uses in production. Tools that write data or cost money can run as mocks, so tests don’t need to touch your live systems. Each scenario plays several times, and an LLM judge scores every trajectory on Evaluations metrics such as user-goal completion, knowledge retention, and tool-use accuracy, plus any custom metrics you define. Because each trajectory records every tool call, Simulations can tell a reasoning failure from a tool-use failure, and running baseline against candidate shows exactly what improved and what regressed. It can be used to validate any change, from instructions and tool selection to fine-tuned or RL-trained checkpoints, on any harness supported by Harness Runtime, including Claude Code, Codex, and OpenCode. Third-party agents such as LangChain, Hermes, and CrewAI aren’t supported in this preview, and if you only want to compare two models on fixed prompts, single-turn Evaluations remains the right tool.
At a high level, the loop runs like this:
DigitalOcean Signals and Simulations are now available in Public Preview. Turn on Signals for an agent you’re already running, and run your next change through Simulations before your users do.
Enable Signals
Run a Simulation