Evaluating Proactive Agents A new guide argues that proactive agents — such as Meta's Muse, SpaceXAI's Grok Bot and Instinct — must be evaluated by tracing their `worth_acting_on` decision step, logging the triggering event, retrieved context, outcome (act now, wait or stay silent), how far the agent goes (notify, suggest, ask for approval or act) and its rationale. The guide distinguishes proactive agents, which hold a standing goal and treat triggers as situations to assess, from reactive agents, which treat a trigger as an instruction, and recommends splitting the decision into a cheap filter using rules, embeddings or a decision model like Jev. Evaluating proactive agents Proactive agents decide on their own whether, when, and how far to act. Learn how to trace that decision, measure it like a classifier, and build a dataset of moments to evaluate it. My personal Muse agent, Dr. Doodle, lives in my WhatsApp and slides into my DMs multiple times a day. Dr. Doodle is part of the newest wave of agents that doesn't wait to be asked. Muse https://about.fb.com/news/2026/09/introducing-muse-personal-ai-agent/amp/ from Meta, Grok Bot https://x.ai/news/grok-bot-1-0 from SpaceXAI and Instinct https://spyglass.org/proactive-ai-is-the-key/ watch your inbox, goals, calendar and projects, and step in when they think something needs doing. "Proactive" is the label everyone uses. But what actually makes an agent proactive? And how do you evaluate whether it acts on the right things? What makes an agent proactive what-makes-an-agent-proactive Proactive agent: an agent with a standing goal that monitors its context for triggers and decides on its own whether, when and how far to act, without being asked each time. Reactive agent: an agent that acts when triggered, on a task it was given. Under the hood, both are built from the same parts: instructions, tools, an agent loop, memory and a trigger. The difference is the goal and the decision. Some examples: | Case | Proactive? | Why | |---|---|---| | Chatbot with memory | No | No standing goal. The user decides when it acts. | | Nightly job that always runs | No | No decision. The schedule decides. | | Webhook that triages every new ticket | No | It acts on every trigger. | | Agent that checks your inbox and only pings you about what matters | Yes | It has a goal, watches context, and can decide to do nothing. | | Agent that rebooks your flight after a delay | Yes | It assesses the delay, decides it needs solving, then asks or just rebooks depending on your instructions. | The most important difference between both is how they handle triggers. A reactive agent treats the trigger as an instruction, a proactive agent treats it as a situation to assess in light of the larger goal. A nightly job that always runs the same task is automation, not proactivity. What it looks like in code what-it-looks-like-in-code The whole distinction fits in two lines: - Reactive: trigger → act - Proactive: trigger → decide whether, when and how far → act or not In code, it's one extra step before the agent runs: worth acting on answers three questions. Say an email arrives: your flight is delayed by two hours. 1. Whether to act: does the delay break your connection, or is it harmless? 2. When to act: tell you now, or wait an hour to see if the airline rebooks you? Waiting means the agent schedules its own next trigger. 3. How far to go: tell you, suggest a new flight, ask to book it, or book it on its own. All three depend on memory. The agent has to know your plans and preferences to judge what deserves your attention. Everything interesting about a proactive agent lives inside worth acting on . That's the part you need to see and evaluate, and it's what the rest of this post is about. Trace the decision, not just the action trace-the-decision With a chatbot, every output has a reader: the person who asked. That's free quality control. With a proactive agent, most runs end in "do nothing" or finish in the background, and nobody looks at them. So the decision itself needs to be in the trace https://langfuse.com/docs/observability/best-practices . For every event that reaches the decision step, log: - The event: what woke the agent up. - The context: what it retrieved from memory and connected apps. - The outcome: act now, wait, or stay silent. - How far: notify, suggest, ask for approval, or act. - The rationale: why it decided this. If you split worth acting on into a cheap filter rules, embeddings, or a decision model like Jev https://langfuse.com/blog/2026-09-18-using-typesafes-jev-for-evals and an expensive judgment call, as Navan https://navan.com/blog/build-proactive-ai-agents recommends, trace both steps https://langfuse.com/docs/observability/features/observation-types . Otherwise you can't tell whether a missed alert was dropped by the filter or judged harmless. Keep the runs that end in "do nothing". A decision not to act produces no output, so it never shows up in a review queue https://langfuse.com/docs/evaluation/evaluation-methods/annotation-queues unless you put it there. Navan https://navan.com/blog/build-proactive-ai-agents makes the same point: staying silent is a valid outcome, and often the right one. That only holds if you can check it. Measure it like a classifier measure-it-like-a-classifier worth acting on takes an event and returns a decision given the overall context. That makes it a classifier, so evaluate it https://langfuse.com/docs/evaluation/overview like one. | Failure | What it looks like | Signals in production | |---|---|---| | Low precision: too eager | The agent pings you about things you don't care about. Users mute it. | Dismissed suggestions, ignored notifications, denied approvals | | Low recall: too cautious | The agent misses what you actually needed. Nobody notices, because nothing happened. | Users doing the task themselves, "why didn't you tell me?" | | Wrong depth: went too far | The agent acts where it should have asked. | Undone actions, complaints, support tickets | Being too eager or going too far is a core risk, it is Dr. Doodle reminding me 5x about a package I already picked up. But it can also have security consequences. One user reported https://proton.me/blog/proactive-ai that Instinct hit a login wall while buying something and reset the user's password instead of asking. The costs are not symmetric. A missed nudge costs little. A wrong purchase costs money and trust. So don't tune one global threshold. Set a stricter bar for actions that are expensive or hard to undo, and a looser one for a suggestion the user can ignore. To turn dismissals, ignored notifications and undone actions into scores on your traces, see Capturing user signals https://langfuse.com/academy/monitoring/capturing-signals . Build a dataset of moments and run evals build-a-dataset-of-moments For a chatbot, a dataset https://langfuse.com/docs/evaluation/experiments/datasets item is a question with an expected answer. For a proactive agent, it's a moment with an expected decision . - Input: the trigger plus a snapshot of the user's memory and instructions. Without it, you can't judge the decision: the same flight delay means "stay silent" for one user and "rebook" for another. - Expected output: act, wait or stay silent, and how far, plus a short note on why. - Sources if based on real failures : flagged traces user feedback https://langfuse.com/docs/observability/features/user-feedback like dismissals, undos, "why didn't you tell me?" , a random sample including runs where nothing happened, and pairs with the same trigger but different users. - Experiments: rerun https://langfuse.com/docs/evaluation/experiments/experiments-via-sdk on every change to prompt, model, memory or thresholds, or in CI https://langfuse.com/docs/evaluation/experiments/experiments-ci-cd . Track precision and recall separately https://langfuse.com/docs/evaluation/experiments/compare-experiments , since more helpful often means more annoying. When a run gets it wrong, check whether it's a judgment bug or a memory bug. It's the same AI engineering loop https://langfuse.com/academy/ai-engineering-loop as for any agent, with decisions as the unit instead of answers. The decision is the product the-decision-is-the-product The skill, the tools, the loop, the memory and the trigger are parts every agent has. What makes a proactive agent good or bad is one function: whether this moment is worth acting on, when, and how far. That function runs mostly where nobody is watching. Make it visible, and it becomes something you can measure and improve. Sources sources - Introducing Muse: The World's First Personal AI Agent Built for Everyone https://about.fb.com/news/2026/09/introducing-muse-personal-ai-agent/amp/ , Meta - Introducing Grok Bot 1.0 https://x.ai/news/grok-bot-1-0 , SpaceXAI - Profoundly Proactive https://spyglass.org/proactive-ai-is-the-key/ , Spyglass - How to build proactive AI agents that only act when needed https://navan.com/blog/build-proactive-ai-agents , Navan - What is proactive AI https://proton.me/blog/proactive-ai , Proton Further reading further-reading - Using TypeSafe's Jev for evals https://langfuse.com/blog/2026-09-18-using-typesafes-jev-for-evals : a decision model for cheap classification and eval verdicts. - Catching conversation signals in Langfuse https://langfuse.com/blog/2026-09-23-catching-conversation-signals-in-langfuse : score every conversation for frustration, corrections and resolution with Jev. - LLM evaluation: methods, best practices, and a practical roadmap https://langfuse.com/blog/2025-11-12-evals : the evaluation methods behind this post. - Steal our eval setup for chat agents https://langfuse.com/blog/2026-07-16-steal-our-eval-setup : datasets, evaluators and experiments we run on our own agent. - Systematic evaluation of AI agents https://langfuse.com/blog/2025-11-06-experiment-interpretation : how to run and interpret experiments. - Comparing open-source AI agent frameworks https://langfuse.com/blog/2025-03-19-ai-agent-comparison : the harnesses proactive agents are built on, and how to trace them. Ready to get started with Langfuse? Join thousands of teams building better LLM applications with Langfuse's open-source observability platform. No credit card required, free tier available, self-hosting option Get started today Interested in tracing and improving your AI agents? Get started with Langfuse in minutes.