Bringing AI-driven tooling closer to your robotics data
Foxglove Automations run pre-defined agent prompts on your data in response to triggers like imported recordings, updated events, scheduled jobs, API calls, and Slack requests. They run in the same infrastructure as your data and work with Foxglove’s ever-expanding suite of tools. They can review your data, annotate with typed events, add episodes to Datasets, and interact with external services to file tickets and upload reports where your team works.
Automations are available on Foxglove-hosted and Bring Your Own Storage (BYOS) projects, on Pro and Enterprise plans.
A motivating example #
An engineer is collecting instances of “missed grasps” for a robotic arm in order to improve their control system. The arm has sensors which detect when the fingers are all the way closed, and when this happens during a grasp attempt, it’s a reasonable signal that the grasp missed. However, some of these signals are false positives – for example when picking up a piece of paper or cloth – and need to be filtered out using other signals from the logs.
In a separate technical walkthrough, we illustrate in detail how Automations can process these recordings and events as they’re imported into Foxglove. There, we define an “annotator” Automation that creates an event whenever it detects a missed grasp. Each of those events in turn kicks off a second “reviewer” Automation that looks at the camera views and gripper telemetry around the missed grasp and files an issue with evidence attached and an initial analysis of the problem.
In one run, the gripper signal showed the fingers closing all the way, indicating a possible miss. But after reviewing the image data, gripper signal, and control system state together, the reviewer determined that the gripper had in fact closed on something thin, and noted that in the issue. This kind of judgment is hard to write as a deterministic check. It’s easier to describe in natural language and let the agent decide from context whether the miss was real.
Evolution of the product concept #
Automations grew out of a problem we kept hearing from robotics teams: their robots generate more data than anyone can fully review. Triage queues keep growing, most of what gets flagged turns out to be a false positive, and hiring more reviewers doesn’t scale.
The underlying use case isn’t new. Foxglove’s great strength has always been making triage of robot data faster and more insightful, which helps teams build better robots. But someone still has to do the triage. AI is starting to change that: agents can take the first pass through new data, and engineers can spend their time reviewing what the agents find. Our original plan was a triage product with carefully managed decision trees, image-model checks, and a review inbox.
Meanwhile, our design partners were describing similar needs in other parts of their work, outside of triage. There was often a queue, a repeatable decision or analysis task, and a tracking system like Jira that lives outside Foxglove. We saw this pattern in detection on ingest, enriching tickets with evidence, repeatable fleet digests, and gates that accept or reject episodes for ML training datasets. A triage-specific system would not have served these scenarios well, so we generalized to a prompt-and-trigger mechanism which can support a wider range of automated tasks, including triage. We still plan to build improved tooling and functionality to better support each individual use case, but we wanted to start with a broader foundation.
Starting with the fundamentals #
Earlier this year we released the AI agent sidebar and began instrumenting Foxglove with MCP tooling for agentic use. In broad strokes, Automation runs are implemented as chats without a human in the loop, using a harness and system prompt that are mostly shared with the agent sidebar. On top of this we added triggers that kick off Automation runs, and input prompts that take context from the triggers. We also built tooling specifically for Automations, including a headless Foxglove visualizer that runs in a sandboxed VM so the agent can render layouts and screenshot panels to check its work. Outputs are primarily handled as side effects such as creating events, drafting Datasets, and making updates in external systems like Jira and Linear. Automations can both generate open-ended reports and make decisions and categorizations.
Like the AI agent sidebar, Automations can use the same powerful suite of tools a robotics engineer has in Foxglove, as needed. They can read data, interact with events, run user scripts, and use headless instances of the Foxglove app for deep analysis. An Automation run performs actions as its owner, in one project, right beside the data. Runs are billed as normal AI usage, and run data is processed by Anthropic under our terms of service.
Creating Automations today #
As with any LLM prompt, writing an effective Automation is a lot like creating a guide for a new teammate. The clearer the instructions and the smaller the steps, the better the results.
Some tips:
- It helps to tell the agent what it’s looking at. Topic names rarely explain themselves, so a sentence or two about which topics and fields matter (and what normal looks like) goes a long way. You can load details from external MCP services to help populate this kind of context. The less the agent has to re-discover each run, the faster and cheaper each run will be.
- Prepare a reference layout with appropriate user scripts andpanels supporting the task, and mention the layout in the prompt. For example, if your answer relies on a rolling windowed average, compute this as a derived topic in a layout so the agent can read the answer deterministically. If you want the agent to interpret particular camera angles or 3D views, configure those panels in advance for the task at hand.
- Narrowly focused Automations do better than broad, ambitious ones. Decompose complex tasks so each step answers one clear and well-defined question. When a job has several steps, chain them together. For example, a first Automation might flag candidate events, and a second verifies them.
- Add screenshots and images to help the agent distinguish common patterns.
We recommend using the built-in AI agent sidebar or your favorite MCP client to develop your prompt and reference layout. Start by describing clearly what a positive and a negative case look like. Get the task working on a single instance, then run it against known positive and negative examples to see how it generalizes. After a few runs, have the agent review them: it can point out where the Automation struggled and suggest what to add to the prompt, like which topics to check first or which thresholds matter.
Where things are headed #
Modern LLMs allow you to articulate sophisticated policies about how to interpret real-world data using natural language, making them remarkably expressive and flexible. That said, running raw data directly through the models is not an effective or efficient way to process it. We already provide a wide range of deterministic tools to the Automation, and over time we plan to progressively push more work into sandboxed code, richer MCP tools, FoxQL queries, and sub-agents running smaller decision models. Your Automations will get cheaper, faster, and more repeatable while keeping the same expressive power.
We’d love to hear how Automations is working for you, and how we can make it better. Start with the technical walkthrough or the Automations docs, and join our community to share feedback.