cd /news/ai-agents/evaluating-proactive-agents · home › topics › ai-agents › article
[ARTICLE · art-172721] src=langfuse.com ↗ pub= topic=ai-agents verified=true sentiment=· neutral

Evaluating Proactive Agents

A new guide argues that proactive agents — such as Meta's Muse, SpaceXAI's Grok Bot and Instinct — must be evaluated by tracing their `worth_acting_on` decision step, logging the triggering event, retrieved context, outcome (act now, wait or stay silent), how far the agent goes (notify, suggest, ask for approval or act) and its rationale. The guide distinguishes proactive agents, which hold a standing goal and treat triggers as situations to assess, from reactive agents, which treat a trigger as an instruction, and recommends splitting the decision into a cheap filter using rules, embeddings or a decision model like Jev.

recommendation

Instrument proactive agents to trace each worth_acting_on decision — event, context, outcome, escalation level and rationale — and gate the full agent behind a cheap rules, embedding or decision-model filter.

more in playbook: Designing agent systems
read7 min views6 publishedNov 19, 2026
Evaluating Proactive Agents
Image: source

Proactive agents decide on their own whether, when, and how far to act. Learn how to trace that decision, measure it like a classifier, and build a dataset of moments to evaluate it.

My personal Muse agent, Dr. Doodle, lives in my WhatsApp and slides into my DMs multiple times a day. Dr. Doodle is part of the newest wave of agents that doesn't wait to be asked. Muse from Meta, Grok Bot from SpaceXAI and Instinct watch your inbox, goals, calendar and projects, and step in when they think something needs doing.

"Proactive" is the label everyone uses. But what actually makes an agent proactive? And how do you evaluate whether it acts on the right things?

What makes an agent proactive #

Proactive agent: an agent with a standing goal that monitors its context for triggers and decides on its own whether, when and how far to act, without being asked each time.

Reactive agent: an agent that acts when triggered, on a task it was given.

Under the hood, both are built from the same parts: instructions, tools, an agent loop, memory and a trigger. The difference is the goal and the decision. Some examples:

Case Proactive? Why
Chatbot with memory No No standing goal. The user decides when it acts.
Nightly job that always runs No No decision. The schedule decides.
Webhook that triages every new ticket No It acts on every trigger.
Agent that checks your inbox and only pings you about what matters Yes It has a goal, watches context, and can decide to do nothing.
Agent that rebooks your flight after a delay Yes It assesses the delay, decides it needs solving, then asks or just rebooks depending on your instructions.

The most important difference between both is how they handle triggers. A reactive agent treats the trigger as an instruction, a proactive agent treats it as a situation to assess in light of the larger goal. A nightly job that always runs the same task is automation, not proactivity.

What it looks like in code #

The whole distinction fits in two lines:

- **Reactive:** trigger → act
- **Proactive:** trigger → decide*whether, when and how far* → act (or not)

In code, it's one extra step before the agent runs:

worth_acting_on answers three questions. Say an email arrives: your flight is delayed by two hours.

  1. Whether to act: does the delay break your connection, or is it harmless?
  2. When to act: tell you now, or wait an hour to see if the airline rebooks you? Waiting means the agent schedules its own next trigger.
  3. How far to go: tell you, suggest a new flight, ask to book it, or book it on its own.

All three depend on memory. The agent has to know your plans and preferences to judge what deserves your attention.

Everything interesting about a proactive agent lives inside worth_acting_on. That's the part you need to see and evaluate, and it's what the rest of this post is about.

Trace the decision, not just the action #

With a chatbot, every output has a reader: the person who asked. That's free quality control. With a proactive agent, most runs end in "do nothing" or finish in the background, and nobody looks at them.

So the decision itself needs to be in the trace. For every event that reaches the decision step, log:

  • The event: what woke the agent up.
  • The context: what it retrieved from memory and connected apps.
  • The outcome: act now, wait, or stay silent.
  • How far: notify, suggest, ask for approval, or act.
  • The rationale: why it decided this.

If you split worth_acting_on into a cheap filter (rules, embeddings, or a decision model like Jev) and an expensive judgment call, as Navan recommends, trace both steps. Otherwise you can't tell whether a missed alert was dropped by the filter or judged harmless. Keep the runs that end in "do nothing". A decision not to act produces no output, so it never shows up in a review queue unless you put it there. Navan makes the same point: staying silent is a valid outcome, and often the right one. That only holds if you can check it.

Measure it like a classifier #

worth_acting_on takes an event and returns a decision given the overall context. That makes it a classifier, so evaluate it like one.

Failure What it looks like Signals in production
Low precision: too eager The agent pings you about things you don't care about. Users mute it. Dismissed suggestions, ignored notifications, denied approvals
Low recall: too cautious The agent misses what you actually needed. Nobody notices, because nothing happened. Users doing the task themselves, "why didn't you tell me?"
Wrong depth: went too far The agent acts where it should have asked. Undone actions, complaints, support tickets

Being too eager or going too far is a core risk, it is Dr. Doodle reminding me 5x about a package I already picked up. But it can also have security consequences. One user reported that Instinct hit a login wall while buying something and reset the user's password instead of asking.

The costs are not symmetric. A missed nudge costs little. A wrong purchase costs money and trust. So don't tune one global threshold. Set a stricter bar for actions that are expensive or hard to undo, and a looser one for a suggestion the user can ignore.

To turn dismissals, ignored notifications and undone actions into scores on your traces, see Capturing user signals.

## [Build a dataset of moments and run evals](#build-a-dataset-of-moments)

For a chatbot, a [dataset](https://langfuse.com/docs/evaluation/experiments/datasets) item is a question with an expected answer. For a proactive agent, **it's a moment with an expected decision**.
  • Input: the trigger plus a snapshot of the user's memory and instructions. Without it, you can't judge the decision: the same flight delay means "stay silent" for one user and "rebook" for another.
  • Expected output: act, wait or stay silent, and how far, plus a short note on why.
  • Sources (if based on real failures): flagged traces (user feedback like dismissals, undos, "why didn't you tell me?"), a random sample including runs where nothing happened, and pairs with the same trigger but different users.
  • Experiments:rerun on every change to prompt, model, memory or thresholds, orin CI .Track precision and recall separately , since more helpful often means more annoying. When a run gets it wrong, check whether it's a judgment bug or a memory bug.

It's the same AI engineering loop as for any agent, with decisions as the unit instead of answers.

The decision is the product #

The skill, the tools, the loop, the memory and the trigger are parts every agent has. What makes a proactive agent good or bad is one function: whether this moment is worth acting on, when, and how far.

That function runs mostly where nobody is watching. Make it visible, and it becomes something you can measure and improve.

## [Sources](#sources)

- [Introducing Muse: The World's First Personal AI Agent Built for Everyone](https://about.fb.com/news/2026/09/introducing-muse-personal-ai-agent/amp/) , Meta
- [Introducing Grok Bot 1.0](https://x.ai/news/grok-bot-1-0) , SpaceXAI
- [Profoundly Proactive](https://spyglass.org/proactive-ai-is-the-key/) , Spyglass
- [How to build proactive AI agents that only act when needed](https://navan.com/blog/build-proactive-ai-agents) , Navan
- [What is proactive AI](https://proton.me/blog/proactive-ai) , Proton

## [Further reading](#further-reading)

- [Using TypeSafe's Jev for evals](https://langfuse.com/blog/2026-09-18-using-typesafes-jev-for-evals) : a decision model for cheap classification and eval verdicts.
- [LLM evaluation: methods, best practices, and a practical roadmap](https://langfuse.com/blog/2025-11-12-evals) : the evaluation methods behind this post.
- [Steal our eval setup for chat agents](https://langfuse.com/blog/2026-07-16-steal-our-eval-setup) : datasets, evaluators and experiments we run on our own agent.
- [Systematic evaluation of AI agents](https://langfuse.com/blog/2025-11-06-experiment-interpretation) : how to run and interpret experiments.
- [Comparing open-source AI agent frameworks](https://langfuse.com/blog/2025-03-19-ai-agent-comparison) : the harnesses proactive agents are built on, and how to trace them.

Ready to get started with Langfuse?

Join thousands of teams building better LLM applications with Langfuse's open-source observability platform.

No credit card required, free tier available, self-hosting option

Get started today

Interested in tracing and improving your AI agents? Get started with Langfuse in minutes.

── more in #ai-agents 4 stories · sorted by recency
── more on @meta 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/evaluating-proactive…] indexed:0 read:7min 2026-11-19 · —