# Event-driven AI agents: Build multi-agent workflows that survive production failures > Source: > Published: 2026-08-17 01:44:41+00:00 AI agents become fragile when they are connected as long synchronous chains. An event bus lets them work independently, wait for people and tools, recover after restarts, and place policy between a model's recommendation and a real action. Most AI agent demos fit inside a single request: ``` php User -> Agent -> Tool -> Agent -> Response ``` The agent makes a plan, calls a tool, gets an answer, and returns a response. It is easy to understand and easy to demo. Then the same agent meets a real workflow. It needs data from several systems. One tool takes five minutes. Another agent has to review the result. A production change needs human approval. While the approval is pending, one of the services restarts. The model is no longer the hardest part. Coordination is. At that point, adding a better prompt will not fix the system. The agents need a way to work independently, survive failures, and resume after the original request has ended. That is an event-driven architecture problem. Imagine an operations agent investigating a slow checkout service. It needs to collect metrics, inspect dependencies, run a diagnostic job, propose a recovery action, wait for approval, execute the change, and verify recovery. With direct calls, each component knows what comes next. The first agent waits for the second. The second waits for a tool. The request stays open while a person decides whether to approve the change. This works when every step is fast and available. In production, that assumption does not last long. A timeout may leave a tool running after its caller has given up. A retry may execute the same action twice. A restart can erase the current plan. Adding a security review means changing an integration that was already working. Human approval exposes the problem most clearly. A decision may take minutes or hours. Holding an HTTP request open across that wait is a poor way to preserve a workflow. In an event-driven design, an agent publishes what happened. Other components decide whether that fact matters to them. ``` ServiceLatencyIncreased | v DiagnosisRequested | v DependencyFailureSuspected | v RecoveryProposed | v HumanApprovalRequired | v RecoveryApproved | v RecoveryCompleted ``` The service health agent can publish `DiagnosisRequested` without knowing which diagnostic agent will handle it. The diagnostic agent can publish `DependencyFailureSuspected` without calling the remediation agent directly. An audit service, an observability pipeline, and a security agent can all react to the same event. If one consumer is temporarily unavailable, it can process the event after it recovers, subject to the broker's retention settings. This is the useful part of an event bus. Components participate in the workflow without being wired directly to one another. It is the same producer and consumer separation described in [AWS Prescriptive Guidance for event-driven AI](https://docs.aws.amazon.com/prescriptive-guidance/latest/agentic-ai-serverless/event-driven-architecture.html), but the pattern is not tied to any cloud or broker. Agent systems get dangerous when every message is treated as interchangeable. An event records a fact: ``` { "type": "ServiceLatencyIncreased", "eventId": "evt-204-01", "incidentId": "incident-204", "service": "checkout-api", "p95LatencyMs": 1840 } ``` The exact schema is up to the system. When events cross team or platform boundaries, the vendor-neutral [CloudEvents specification](https://cloudevents.io/) provides a common envelope for event metadata. A command requests an action: ``` { "type": "ScaleService", "commandId": "cmd-204-01", "incidentId": "incident-204", "service": "checkout-api", "targetReplicas": 12, "idempotencyKey": "incident-204:scale:12" } ``` An agent decision is a recommendation. It can include evidence and confidence, but it is still a proposal. These distinctions matter. An LLM saying "scale the service" does not mean the service was scaled. It also does not mean the action was authorized. A safer path is: ``` php Event -> Agent decision -> Proposed command -> Policy check -> Authorized command -> Execution -> Outcome event ``` The agent interprets the situation and proposes an action. A policy layer checks permissions, limits, and approval requirements. Deterministic code performs the change. The executor then publishes what actually happened. That separation remains useful even when the model changes. The model can become more capable without quietly gaining permission to restart production or issue a refund. Agent platforms talk a lot about memory. Conversational memory and execution state solve different problems. Memory may contain a conversation summary, user preferences, or retrieved knowledge. Workflow state answers operational questions: A conversation transcript is a fragile place to store those answers. It may be summarized, truncated, or interpreted differently after a model update. If the only record that a refund was issued is a sentence in a context window, the system may eventually issue it again. Store workflow state explicitly. Keep event IDs, command status, approvals, retry counts, and execution results in durable storage. Give the model only the context it needs for its current decision. Moving work onto an event bus changes the failure modes. It does not remove them. Design consumers so they can see the same message more than once. Track stable event IDs, and require idempotency keys for tools that change state. Restarting a service twice or issuing the same refund twice is not a retry strategy. An approval may arrive after a proposal has expired. A recovery result may appear after a newer action has started. Put versions and timestamps on messages, and reject transitions that no longer match the current workflow state. Agent A publishes an event that wakes Agent B. Agent B responds with an event that wakes Agent A. Both agents can keep generating messages and spending tokens without completing useful work. Track causation depth. Set limits for time, tokens, and workflow steps. Route repeated failures to human review. Two agents may propose opposite actions for the same resource. Do not allow both commands to run. Serialize changes for that resource or use a version check before execution. The consumer itself should be boring. That is a compliment: ``` python def handle(event, store, agent, policy, executor): if store.already_processed(event.id): return workflow = store.load_workflow(event.incident_id) if not workflow.accepts(event.type, event.version): store.mark_processed(event.id, outcome="stale") return decision = agent.decide(workflow.context_for(event)) verdict = policy.evaluate(decision) if verdict.requires_human: store.request_approval(workflow, decision) return if not verdict.allowed: store.record_denied(workflow, decision, verdict.reason) return result = executor.run( decision.command, idempotency_key=decision.command.idempotency_key, ) store.append_event(workflow, result.to_event()) store.mark_processed(event.id, outcome="applied") ``` The model performs the ambiguous reasoning. The surrounding code handles state, policy, deduplication, and execution in ways that can be tested. A document summarizer does not need a distributed control plane. A synchronous request is usually enough when one caller expects an immediate answer, the task finishes quickly, and a retry cannot cause harm. Events begin to earn their cost when work continues past a request timeout, several agents act independently, a human interrupts the workflow, or an action has operational or financial consequences. There is a price. Teams have to manage message schemas, retention, tracing, failed-message handling, and asynchronous debugging. Use an event-driven design when those costs buy reliability, not because the architecture sounds more sophisticated. You do not need to rebuild a synchronous agent system all at once. Find the step that blocks the longest. It is often a human approval or a slow external tool. Give that step a durable state record and an event that can resume the workflow. Then add an idempotency key to every command that changes something outside the agent system. Take one agent workflow you already run and draw the waits, retries, approvals, and side effects. Where those boundaries are hidden inside a single synchronous chain, that is where the first event belongs. Better models will improve agent decisions. They will not recover a lost approval, prevent a duplicate command, or resume a half-finished workflow. Once agents work across services and over time, they inherit distributed-system failures. Failures still happen in an event-driven design. Durable workflow boundaries make it possible to stop safely and resume later.