{"slug": "evaluating-proactive-agents", "title": "Evaluating Proactive Agents", "summary": "A new guide argues that proactive agents — such as Meta's Muse, SpaceXAI's Grok Bot and Instinct — must be evaluated by tracing their `worth_acting_on` decision step, logging the triggering event, retrieved context, outcome (act now, wait or stay silent), how far the agent goes (notify, suggest, ask for approval or act) and its rationale. The guide distinguishes proactive agents, which hold a standing goal and treat triggers as situations to assess, from reactive agents, which treat a trigger as an instruction, and recommends splitting the decision into a cheap filter using rules, embeddings or a decision model like Jev.", "body_md": "# Evaluating proactive agents\n\nProactive agents decide on their own whether, when, and how far to act. Learn how to trace that decision, measure it like a classifier, and build a dataset of moments to evaluate it.\n\nMy personal Muse agent, Dr. Doodle, lives in my WhatsApp and slides into my DMs multiple times a day. Dr. Doodle is part of the newest wave of agents that doesn't wait to be asked. [Muse](https://about.fb.com/news/2026/09/introducing-muse-personal-ai-agent/amp/) from Meta, [Grok Bot](https://x.ai/news/grok-bot-1-0) from SpaceXAI and [Instinct](https://spyglass.org/proactive-ai-is-the-key/) watch your inbox, goals, calendar and projects, and step in when they think something needs doing.\n\n\"Proactive\" is the label everyone uses. But what actually makes an agent proactive? And how do you evaluate whether it acts on the right things?\n\n## [What makes an agent proactive](#what-makes-an-agent-proactive)\n\n**Proactive agent:** an agent with a standing goal that monitors its context for triggers and decides on its own whether, when and how far to act, without being asked each time.\n\n**Reactive agent:** an agent that acts when triggered, on a task it was given.\n\nUnder the hood, both are built from the same parts: instructions, tools, an agent loop, memory and a trigger. The difference is the goal and the decision. Some examples:\n\n| Case | Proactive? | Why | \n|---|---|---|\n| Chatbot with memory | No | No standing goal. The user decides when it acts. | \n| Nightly job that always runs | No | No decision. The schedule decides. | \n| Webhook that triages every new ticket | No | It acts on every trigger. | \n| Agent that checks your inbox and only pings you about what matters | Yes | It has a goal, watches context, and can decide to do nothing. | \n| Agent that rebooks your flight after a delay | Yes | It assesses the delay, decides it needs solving, then asks or just rebooks depending on your instructions. | \n\nThe most important difference between both is how they handle triggers. **A reactive agent treats the trigger as an instruction, a proactive agent treats it as a situation to assess in light of the larger goal.** A nightly job that always runs the same task is automation, not proactivity.\n\n## [What it looks like in code](#what-it-looks-like-in-code)\n\nThe whole distinction fits in two lines:\n\n- **Reactive:** trigger → act\n- **Proactive:** trigger → decide*whether, when and how far* → act (or not)\n\nIn code, it's one extra step before the agent runs:\n\n`worth_acting_on` answers three questions. Say an email arrives: your flight is delayed by two hours.\n\n1. **Whether** to act: does the delay break your connection, or is it harmless?\n2. **When** to act: tell you now, or wait an hour to see if the airline rebooks you? Waiting means the agent schedules its own next trigger.\n3. **How far** to go: tell you, suggest a new flight, ask to book it, or book it on its own.\n\nAll three depend on memory. The agent has to know your plans and preferences to judge what deserves your attention.\n\n**Everything interesting about a proactive agent lives inside `worth_acting_on`.** That's the part you need to see and evaluate, and it's what the rest of this post is about.\n\n## [Trace the decision, not just the action](#trace-the-decision)\n\nWith a chatbot, every output has a reader: the person who asked. That's free quality control. With a proactive agent, most runs end in \"do nothing\" or finish in the background, and nobody looks at them.\n\nSo the decision itself needs to be [in the trace](https://langfuse.com/docs/observability/best-practices). For every event that reaches the decision step, log:\n\n- **The event:** what woke the agent up.\n- **The context:** what it retrieved from memory and connected apps.\n- **The outcome:** act now, wait, or stay silent.\n- **How far:** notify, suggest, ask for approval, or act.\n- **The rationale:** why it decided this.\n\nIf you split `worth_acting_on` into a cheap filter (rules, embeddings, or a decision model like [Jev](https://langfuse.com/blog/2026-09-18-using-typesafes-jev-for-evals)) and an expensive judgment call, as [Navan](https://navan.com/blog/build-proactive-ai-agents) recommends, [trace both steps](https://langfuse.com/docs/observability/features/observation-types). Otherwise you can't tell whether a missed alert was dropped by the filter or judged harmless.\n\n**Keep the runs that end in \"do nothing\".** A decision not to act produces no output, so it never shows up in a [review queue](https://langfuse.com/docs/evaluation/evaluation-methods/annotation-queues) unless you put it there. [Navan](https://navan.com/blog/build-proactive-ai-agents) makes the same point: staying silent is a valid outcome, and often the right one. That only holds if you can check it.\n\n## [Measure it like a classifier](#measure-it-like-a-classifier)\n\n`worth_acting_on` takes an event and returns a decision given the overall context. **That makes it a classifier, so [evaluate it](https://langfuse.com/docs/evaluation/overview) like one.**\n\n| Failure | What it looks like | Signals in production | \n|---|---|---|\n| **Low precision:** too eager | The agent pings you about things you don't care about. Users mute it. | Dismissed suggestions, ignored notifications, denied approvals | \n| **Low recall:** too cautious | The agent misses what you actually needed. Nobody notices, because nothing happened. | Users doing the task themselves, \"why didn't you tell me?\" | \n| **Wrong depth:** went too far | The agent acts where it should have asked. | Undone actions, complaints, support tickets | \n\nBeing too eager or going too far is a core risk, it is Dr. Doodle reminding me 5x about a package I already picked up. But it can also have security consequences. [One user reported](https://proton.me/blog/proactive-ai) that Instinct hit a login wall while buying something and reset the user's password instead of asking.\n\n**The costs are not symmetric.** A missed nudge costs little. A wrong purchase costs money and trust. So don't tune one global threshold. Set a stricter bar for actions that are expensive or hard to undo, and a looser one for a suggestion the user can ignore.\n\nTo turn dismissals, ignored notifications and undone actions into scores on your traces, see [Capturing user signals](https://langfuse.com/academy/monitoring/capturing-signals).\n\n## [Build a dataset of moments and run evals](#build-a-dataset-of-moments)\n\nFor a chatbot, a [dataset](https://langfuse.com/docs/evaluation/experiments/datasets) item is a question with an expected answer. For a proactive agent, **it's a moment with an expected decision**.\n\n- **Input:** the trigger plus a snapshot of the user's memory and instructions. Without it, you can't judge the decision: the same flight delay means \"stay silent\" for one user and \"rebook\" for another.\n- **Expected output:** act, wait or stay silent, and how far, plus a short note on why.\n- **Sources** (if based on real failures): flagged traces ([user feedback](https://langfuse.com/docs/observability/features/user-feedback) like dismissals, undos, \"why didn't you tell me?\"), a random sample including runs where nothing happened, and pairs with the same trigger but different users.\n- **Experiments:**[rerun](https://langfuse.com/docs/evaluation/experiments/experiments-via-sdk) on every change to prompt, model, memory or thresholds, or[in CI](https://langfuse.com/docs/evaluation/experiments/experiments-ci-cd) .[Track precision and recall separately](https://langfuse.com/docs/evaluation/experiments/compare-experiments) , since more helpful often means more annoying. When a run gets it wrong, check whether it's a judgment bug or a memory bug.\n\nIt's the same [AI engineering loop](https://langfuse.com/academy/ai-engineering-loop) as for any agent, with decisions as the unit instead of answers.\n\n## [The decision is the product](#the-decision-is-the-product)\n\nThe skill, the tools, the loop, the memory and the trigger are parts every agent has. What makes a proactive agent good or bad is one function: whether this moment is worth acting on, when, and how far.\n\nThat function runs mostly where nobody is watching. **Make it visible, and it becomes something you can measure and improve.**\n\n## [Sources](#sources)\n\n- [Introducing Muse: The World's First Personal AI Agent Built for Everyone](https://about.fb.com/news/2026/09/introducing-muse-personal-ai-agent/amp/) , Meta\n- [Introducing Grok Bot 1.0](https://x.ai/news/grok-bot-1-0) , SpaceXAI\n- [Profoundly Proactive](https://spyglass.org/proactive-ai-is-the-key/) , Spyglass\n- [How to build proactive AI agents that only act when needed](https://navan.com/blog/build-proactive-ai-agents) , Navan\n- [What is proactive AI](https://proton.me/blog/proactive-ai) , Proton\n\n## [Further reading](#further-reading)\n\n- [Using TypeSafe's Jev for evals](https://langfuse.com/blog/2026-09-18-using-typesafes-jev-for-evals) : a decision model for cheap classification and eval verdicts.\n- [Catching conversation signals in Langfuse](https://langfuse.com/blog/2026-09-23-catching-conversation-signals-in-langfuse) : score every conversation for frustration, corrections and resolution with Jev.\n- [LLM evaluation: methods, best practices, and a practical roadmap](https://langfuse.com/blog/2025-11-12-evals) : the evaluation methods behind this post.\n- [Steal our eval setup for chat agents](https://langfuse.com/blog/2026-07-16-steal-our-eval-setup) : datasets, evaluators and experiments we run on our own agent.\n- [Systematic evaluation of AI agents](https://langfuse.com/blog/2025-11-06-experiment-interpretation) : how to run and interpret experiments.\n- [Comparing open-source AI agent frameworks](https://langfuse.com/blog/2025-03-19-ai-agent-comparison) : the harnesses proactive agents are built on, and how to trace them.\n\n### Ready to get started with Langfuse?\n\nJoin thousands of teams building better LLM applications with Langfuse's open-source observability platform.\n\nNo credit card required, free tier available, self-hosting option\n\nGet started today\n\nInterested in tracing and improving your AI agents? Get started with Langfuse in minutes.", "url": "https://wpnews.pro/news/evaluating-proactive-agents", "canonical_source": "https://langfuse.com/blog/2026-11-19-evaluating-proactive-agents", "published_at": "2026-11-19 17:25:04+00:00", "updated_at": "2026-11-19 17:45:49.820550+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "artificial-intelligence"], "entities": ["Meta", "Muse", "SpaceXAI", "Grok Bot", "Instinct", "Dr. Doodle", "Jev"], "also_reported_by": [], "takeaway": "Log every event reaching a proactive agent's worth_acting_on step — the trigger, retrieved context, act/wait/silent outcome, escalation level and rationale — and split the decision into a cheap filter (rules, embeddings or a decision model) before the full agent runs.", "recommendation": {"advice": "Instrument proactive agents to trace each worth_acting_on decision — event, context, outcome, escalation level and rationale — and gate the full agent behind a cheap rules, embedding or decision-model filter.", "playbook": "agent-architecture", "playbook_url": "https://wpnews.pro/playbooks/agent-architecture"}, "alternates": {"html": "https://wpnews.pro/news/evaluating-proactive-agents", "markdown": "https://wpnews.pro/news/evaluating-proactive-agents.md", "text": "https://wpnews.pro/news/evaluating-proactive-agents.txt", "jsonld": "https://wpnews.pro/news/evaluating-proactive-agents.jsonld"}}