At 2 a.m., a founder's autonomous agent starts returning 500 errors. No dashboard is open, no notification reaches the on-call channel, and the failure continues long enough to damage an SLA and customer trust. The telemetry exists, but nobody has turned it into a decision that reaches the person or system responsible for acting.
That distinction matters. A missing alert is often a product failure, not a telemetry failure. Modern real time alerts should detect meaningful state changes, decide whether they deserve interruption, and deliver useful context within a budget the recipient understands. This guide builds that mental model from first principles, maps the architecture behind it, compares delivery choices, and ends with a reusable alert policy template for agent monitoring, billing, security, and workflow notifications.
Table of Contents #
The Moment an Alert Would Have Saved the Day #
The founder wakes up to a customer message, not an incident page. The agent has been failing, retries have exhausted themselves, and the team's first task is reconstructing when the problem began. A useful alert could have changed the sequence entirely: detect the repeated failure, identify the affected workflow, send the event to the right owner, and include enough context to support a decision.
That's the true purpose of real time alerts. They don't exist merely to make a bell ring. They turn operational state into an actionable product experience for two audiences: humans who need prioritised context and AI agents that can trigger a controlled response.
Practical rule: If nobody can explain what action follows an alert, the system has produced a notification, not an operational signal.
A strong alert answers four questions without forcing the recipient to search through logs:
- What changed? The event or state transition should be explicit.
- Why does it matter? Include impact, affected resource, and confidence where available.
- Who or what should act? Route by ownership, severity, and workflow.
- What happens next? Link to a runbook, escalation path, or automated action.
The rest of the design follows from those questions. You'll need a definition of “real time” that reflects the receiver's needs, an architecture showing where latency accumulates, delivery choices that account for failure and cost, and a policy format your team can maintain. The best system won't alert on every unusual event. It will protect attention for events where delay or inaction creates meaningful risk.
What Real Time Alerts Actually Mean #
Real time alerts are signal pipelines that detect a meaningful state change, decide whether someone or something should know, and deliver that decision within an agreed latency budget. The phrase doesn't mean every event must arrive in milliseconds. It means the notification arrives quickly enough to support the action it represents.
A fire alarm offers a useful analogy. The detector observes smoke or heat, the decision unit determines whether the condition crosses a meaningful threshold, and the horn delivers the warning to people nearby. A modern alerting system has the same three jobs:
- Detect. Read logs, metrics, traces, webhooks, or domain events and identify a condition.
- Decide. Apply thresholds, windows, correlation, suppression, or confidence rules.
- Deliver. Route the resulting signal to a human, an agent, or an automated workflow.
The common confusion is treating detection time as alert time. A metric may be collected quickly, but the system can still lose time in aggregation, queueing, serialisation, routing, rate limits, or a slow notification provider. The recipient experiences the full path, so the budget must cover the entire path rather than just the rule evaluation step.
Real time depends on the action
A security event that requires immediate containment has a different expectation from a weekly usage summary. One expert framework sets end-to-end targets of under 1 second for critical security events, under 2 seconds for system outages, under 4 seconds for performance degradation, and under 10 seconds for business anomalies. Those targets and the reasons behind them are described in Confluent's real-time alerts architecture guidance.
For marketing teams, the same principle applies to campaign, lead, and customer-behaviour events. A practical introduction to real-time alerts for marketers can help connect operational alerting concepts with audience and engagement workflows.
“Real time” should therefore be written as a contract: event class, maximum useful delay, recipient, delivery channel, and fallback. Without those fields, teams often optimise for speed while ignoring whether the signal is accurate, actionable, or trustworthy.
Inside the Architecture of a Modern Alerting System #
A modern alerting system resembles a production line with three stations. Events enter at the source, decision logic evaluates them, and a delivery fabric sends the result to a person or an automated consumer. Each station can consume the end-to-end budget, so measuring only rule-evaluation time gives an incomplete picture.
Event sources
The first layer produces the raw material. Logs record errors and state changes, metrics expose measurements, traces connect work across services, webhooks carry external events, and domain events represent business actions such as a payment transition or workflow completion.
Source design affects alert quality before any rule runs. A metric without the relevant service or tenant label makes routing harder. A webhook without an event identifier weakens deduplication. A log without structured fields forces the evaluator to parse text when speed and consistent interpretation matter most.
Streaming and decision logic
The middle layer turns events into decisions. A threshold rule evaluates a current value. A windowed aggregator measures behaviour across time. Correlation logic combines signals, such as repeated failed authentication with a new device, or an agent heartbeat gap with an unfinished task.
Historical state belongs here too. Grafana's alert state history documentation describes recorded state changes that can be filtered by labels, current state, and previous state. OpenSearch also preserves completed alerts in dedicated history indices, supporting audit, debugging, and incident review.
This layer should retain enough context to explain why an alert fired. That explanation serves human responders, AI agents, and later policy reviews. Without it, a notification may arrive quickly but still require manual reconstruction before anyone can act.
Delivery fabric
The final layer selects the destination and controls interruption. It handles routing, grouping, throttling, escalation, retries, and channel selection. A critical security signal might page an on-call responder and invoke an automated action. A low-confidence business anomaly might enter a digest or an agent summary instead.
A platform team deploying agents can use the Donely Hermes API as one integration point for connecting alert-triggered workflows to an agent runtime. Keep detection, policy, and delivery separate. A channel outage should not erase the event, and every rule should not need to understand every destination.
| Layer | Sub-second tier | Single-digit seconds | Minutes |
|---|---|---|---|
| Event sources | In-memory events, direct streams, local instrumentation | Structured logs, metrics, traces, and webhooks with short queues | Batch exports, scheduled reports, and delayed third-party feeds |
| Streaming engine | Stateful evaluation close to the source, minimal hops | Windowed aggregation, correlation, and policy evaluation | Scheduled queries, replay, and deeper historical analysis |
| Delivery fabric | Persistent connections, direct push, rapid failover | Queued webhooks, chat, incident tools, and controlled retries | Email digests, tickets, audit views, and human review queues |
The architecture determines practical alert speed. A benchmarked real-time data platform reported alert-notification latency of 6.9 seconds compared with 83.2 seconds for a competitive build in a low-volume scenario. The result shows how queueing and cross-service hops can dominate the budget even when the rule itself is simple.
Design each layer around its users and its noise budget. Humans need clear interruption rules, while agents need structured events, stable identifiers, and enough context to act without guessing.
Delivery Mechanisms and How to Choose Between Them #
Delivery should be selected by latency, reliability, and recovery behaviour, not by whichever SDK a team already knows. Push, pull, and webhook-based delivery solve different problems.
| Mechanism | Typical Latency | Reliability Posture | Cost at 1M events/mo | Best Fit |
|---|---|---|---|---|
| Push | Sub-second to seconds when the connection is healthy | Fast, but requires reconnects, backpressure, and client handling | Provider and connection costs vary, usually efficient for active consumers | Interactive user-facing events and live operational views |
| Pull | Minutes or longer, depending on polling schedule | Easy to replay and inspect, but delayed and wasteful when nothing changes | Repeated requests can create avoidable compute and API costs | Compliance, audit reconstruction, and low-urgency review |
| Webhook with fallback | Seconds when the endpoint responds, longer during retries | Strong when signed, queued, retried, and backed by a dead-letter path | Costs include requests, retries, queueing, and fallback channels | Third-party integrations and automated workflows |
These latency bands are design categories, not universal guarantees. A push connection can fail without notification if the client doesn't reconnect. A polling consumer can miss a state transition if it only reads the current value. A webhook can create duplicate work if the receiver doesn't treat event identifiers as idempotency keys.
Push works when immediacy matters
Webhooks, server-sent events, and push notifications send information as soon as the producer has it. They're a strong fit for user-facing status changes, agent completion events, and incidents where a delay makes the response less useful. Build in reconnect behaviour, queue limits, and explicit acknowledgement rather than assuming an open connection equals reliable delivery.
Pull works when reconstruction matters
Polling and scheduled API reads are slower, but they can be easier to govern. They let a compliance or operations team reconstruct history from a durable source and control consumption rather than accepting every event at arrival speed. Pull is often the right choice for audit views, reporting, and low-priority summaries.
Webhooks need a safety net
For third-party integrations, use signed payloads, retries with backoff, idempotent receivers, and a dead-letter path. A webhook is not complete when the sender makes an HTTP request. It's complete when the receiving workflow records the event and either acknowledges it or places it somewhere operators can recover.
Location-aware workflows show why context matters. Teams designing delivery-zone events can review how to improve logistics with geofencing, then decide whether a driver-facing event needs push delivery or whether an auditable pull model is sufficient. For a broader catalogue of connection options, review Donely integrations.
Choose push for immediate interaction, pull for reconstruction, and signed webhooks with retry for system-to-system action.
How Donely Customers Put Real Time Alerts to Work #
Alerting becomes easier to design when each policy names four things: source, condition, destination, and acknowledgement expectation. That turns an abstract rule into a small product contract that an engineer, operator, or agent can understand.
Agent monitoring
The source is an agent run event, heartbeat, tool result, or error log. The condition might be a stalled autonomous task, repeated tool failure, or a run whose token spend exceeds its approved policy. Route urgent failures to the owner or incident channel, while sending recoverable conditions to an agent supervisor that can , retry, or request review.
The acknowledgement expectation should match the consequence. A customer-facing task failure needs rapid ownership. A background enrichment job may be suitable for a queue that an operator reviews later. Include the run identifier, last successful step, tool involved, and suggested next action.
Billing and spend controls
Billing alerts should help a user intervene before a workflow becomes unavailable or costs exceed an approved boundary. The source can be usage events and invoice state. The condition can be a budget milestone, an unusual spend pattern, or a failed payment event.
Route the first warning to the account owner and the final warning to both the owner and finance or operations. A useful payload includes current usage, the relevant period, the affected instance, and the action required. Avoid sending every usage event to a human. Summarise state changes and retain the underlying records for inspection.
Security events
Security signals may come from authentication events, access-policy changes, or suspicious activity metadata. The policy should separate a single failed sign-in from a coordinated pattern, then route the result according to role and resource ownership.
A high-confidence event can page security operations and invoke a controlled containment workflow. An ambiguous event should preserve context for investigation rather than forcing an irreversible action. Every security alert needs an audit trail, an event identifier, and a clear record of acknowledgement and resolution.
Workflow notifications
Workflow alerts connect Donely actions to Slack, PagerDuty, email, or webhook consumers. A completion event can notify a team channel, while a failure event can create an incident with the failed step and retry state. The destination should come from ownership metadata, not from a hard-coded channel buried inside the workflow.
This pattern keeps notifications useful because the event carries operational meaning. A message that says “workflow failed” is weak. A message that identifies the workflow, affected account, failed action, retry state, and owner gives the recipient a path to resolution.
Here's a short visual walkthrough of how these patterns fit into an automation flow:
Reliability, Latency Budgets, and the Cost of Alert Fatigue #
An alerting system has two budgets. The latency budget limits how long a signal may take to reach the right responder. The noise budget limits how many interruptions that responder can process before attention becomes unreliable. A missed page is visible; a crowded inbox trains people to ignore the next one.
Set a delivery target, define a deduplication window, and choose separate paths for pages, tickets, and digests. Critical pages should remain rare enough to prompt immediate action. Repeated events should become one incident containing the count, timeline, affected resource, and recovery state.
A practical starting policy looks like this:
- Cap critical pages: Keep critical pages below three per on-call shift unless an incident is active.
- Group repetition: Group recurring events by service, resource, and failure signature.
- Require context: Include a runbook link, owner, severity, and affected resource on every page.
- Track delivery health: Measure detection delay, delivery delay, acknowledgement, resolution, retries, and dead-letter outcomes.
- Review suppression: Give every muted rule an owner and an expiry, then revisit it.
Clinical alerting shows why interruption needs restraint. One comparison associated interrupting alerts with 34% inappropriate imaging orders versus 18% for a noninterruptive, dynamically annotated visualisation. The quieter format was preferred across scenarios, according to the Retail Gazette summary of the clinical comparison. The operational lesson is broader: an immediate interruption is not automatically better than a digest, queue, or agent-generated summary.
For hosted agent operations, monitor the alerting path as its own product. Teams using Donely Hermes Agent hosting can separate agent-state monitoring from delivery success and escalation outcomes, then tune each against its own SLA and noise budget. That framing gives both human responders and AI agents clearer signals, fewer duplicate interruptions, and a measurable basis for changing policy.
Sample Alert Policies and a Best-Practice Checklist #
A policy should be readable by the person who owns the response, not only by the engineer who wrote the rule. The examples below use YAML-like syntax to show the fields that matter. The thresholds are illustrative policy choices, not universal defaults.
Stalled agent policy
name: agent-stuck
source: agent_heartbeat
condition:
heartbeat_missing: true
run_status: active
idle_for: 5m
severity: critical
route:
primary: agent_owner
fallback: incident_channel
acknowledgement: immediate
payload:
include:
- run_id
- last_completed_step
- last_tool_error
- retry_state
This policy distinguishes an active run from a completed run and gives the responder enough information to decide whether to retry, , or investigate. The heartbeat is the signal, but the operational context makes the alert useful.
Billing threshold policy
name: monthly-budget
source: usage_and_billing_events
conditions:
- usage_reaches: 80%
severity: warning
- usage_reaches: 100%
severity: critical
routing:
warning: account_owner
critical:
- account_owner
- billing_owner
actions:
warning: notify_and_review
critical: _or_approve_continuation
A billing policy should make the next action explicit. Warning recipients can adjust usage or approve a change, while the critical path should prevent an unexpected continuation unless an authorised owner confirms it.
Security event policy
name: authentication-burst
source: authentication_events
condition:
failed_authentication_rate: ">10/min"
severity: critical
route:
primary: pagerduty_security
fallback: security_on_call
deduplication:
key: account_and_source
action:
create_incident: true
require_human_review: true
The policy needs a verification step because high-volume alerts can include artifacts or benign bursts. Recent vital-sign alert research reported 648 alerts, including 171 artifacts, illustrating why false alarms and missed detections belong in the outcome model, as described in the peer-reviewed alert classification research.
Workflow notification policy
name: workflow-completion
source: donely_workflow_events
condition:
event_type: completion
severity: info
route:
destination: slack
channel: team_from_workflow_owner
payload:
include:
- workflow_id
- status
- completed_steps
- output_reference
fallback:
destination: webhook
Completion notifications shouldn't interrupt an on-call engineer unless the workflow is business-critical or failed. Route normal completions to a team channel, preserve the event for history, and reserve escalation for exceptions.
Runbook checklist
- Define severity by action. Critical means someone or something must act promptly. Warning means an owner should review it. Informational means the event can wait.
- Give every alert an owner. Ownership should resolve to a person, team, or service, with a fallback route for unavailable recipients.
- Make events idempotent. Include a stable event identifier so retries don't create duplicate incidents or duplicate automated actions.
- Group and deduplicate deliberately. Choose grouping keys that represent one underlying problem rather than hiding separate failures under a broad label.
- Protect the dead-letter path. Failed deliveries need durable storage, retry visibility, and an operator workflow for replay.
- Test the alert itself. Exercise source failure, evaluator failure, provider rate limits, network interruption, and recipient unavailability.
- Review actionability quarterly. Remove rules nobody uses, rewrite vague messages, adjust thresholds, and record why each policy still exists.
Alert governance is a product responsibility. Assign an owner to every alert, record its intended action, and review whether it still changes a decision at least quarterly. If a rule repeatedly produces noise, tune or retire it rather than asking people to tolerate a broken experience.
Donely provides a unified platform for hosting, deploying, and managing AI employees, with monitoring, integrations, usage, billing, and workflow controls in one dashboard. Visit Donely to connect agent events to practical real time alerts and build an escalation policy your team can operate with confidence.