Short answer: treat a timeout as a bounded, attributable event, not as a single error string. Record one correlation ID for the agent run, separate the model and tool spans, capture deadline and elapsed time at every network boundary, and emit a failure event that preserves the cost already incurred. Then alert on the combination of timeout rate and attributed spend. This identifies the failing segment while preventing a retry storm from making the bill look like the original incident.
For a B2B SaaS agent loop, the dominant cost is usually work that completed before the final notification failed: model tokens, tool calls, and repeated attempts. The network timeout is the symptom at the edge. The accounting question is harder: which attempt consumed the budget, which attempt was safe to retry, and what evidence can be retained without storing customer content?
The retry is the bill.
Start with an event ledger for the loop. For each run, assign an immutable run ID and an attempt number. A useful record contains the operation name, tenant identifier or an irreversible tenant key, start and end timestamps, configured deadline, outcome, bytes sent and received, model usage if available, and a retry reason. The record should describe the operation, not copy the prompt or response into every log line.
Suppose an agent performs two model calls and one billing-system lookup before posting a failure alert. If the lookup reaches its deadline at 2.0 seconds, the run has still consumed the first model call and the lookup's network work. A later retry may consume another model call. Charging the failed run zero is wrong; charging every retry as a new business action is also wrong. The accounting boundary belongs around the attempt, while the business idempotency key belongs around the intended action.
This distinction makes the alert useful. A spike in elapsed time with stable model usage points toward transport or dependency saturation. A spike in attributed tokens with stable timeout counts points toward retries or a longer agent path. The same top-level failure status can represent very different cost behavior.
A timeout diagnosis needs aligned signals, not a larger pile of messages. Traces show the parent-child relationship between the agent run, model calls, tool calls, and notification request. Metrics show whether the pattern is isolated to a dependency, tenant cohort, region, or deployment. Logs carry the decision record: the selected deadline, retry policy, cancellation cause, and the final attribution. OpenTelemetry describes logs as a signal that can be correlated with traces and metrics; that correlation is the important property here, not the log backend.
The most common trap is recording only the client-side error. A client can report a deadline exceeded after the remote service accepted the request, or after the connection failed before any request bytes arrived. The retry decision is different in those cases. Capture phase information where the runtime exposes it: DNS resolution, connection establishment, transport security, request write, response headers, and response body. Keep those fields as bounded enums. Free-form network diagnostics quickly become expensive and hard to aggregate.
A small Go event type keeps the contract explicit:
type AttemptEvent struct {
RunID string
Attempt int
Operation string
Dependency string
Phase string
Outcome string
DeadlineMS int64
ElapsedMS int64
Retryable bool
AttributedUnits int64
}
The phase is evidence, not a diagnosis. A response-header timeout can indicate a slow dependency, queueing, or a lost response; it does not prove which one. That uncertainty should survive into the event so an operator does not mistake a convenient label for a causal fact.
That is the audit boundary.
The alert path needs its own timeout and its own idempotency key. Reusing the agent run ID as the notification key can suppress a legitimate second notification after a policy change; generating a random key on each retry can duplicate an alert. Define the notification identity from the business event, such as tenant + run + failure-class, and persist the decision before attempting delivery. An exactly-once mindset is useful here: delivery itself may be at-least-once, while the recorded decision and downstream effect must be made idempotent.
In Go, the boundary can be represented without coupling it to a particular provider:
type FailureNotice struct {
Key string
RunID string
Failure string
CostUnits int64
OccurredAt time.Time
}
type Notifier interface {
Send(context.Context, FailureNotice) error
}
The durable record should contain the notice key, first decision time, delivery attempts, last phase, and final state. Do not put secrets, authorization headers, or full customer payloads into that record. Retention also needs a policy: retain identifiers, timings, classifications, and cost dimensions long enough to reconcile an operational period; sample or delete payload-bearing diagnostics sooner, subject to contractual and regulatory requirements. Compliance regimes generally require purpose limitation and controlled access, not indefinite retention of everything that might someday help debugging.
Alert on a joint condition rather than a raw timeout count. One practical rule is to page when the timeout ratio for a dependency exceeds its agreed error budget and the attributable cost per failed run rises above its normal band for a sustained window. A separate lower-severity signal can report timeouts with no cost increase. This avoids paging on harmless client cancellations while still exposing retry amplification.
The exact thresholds belong to the workload. Establish them from a labeled baseline by dependency, operation, and tenant tier; do not copy a universal number into every system. The useful dashboard has at least four views: timeout phase, deadline versus elapsed time, retry count, and cost units by run outcome. A fifth view for missing correlation IDs catches instrumentation regressions before they turn into unexplainable invoices.
I initially treated the final failed notification as the unit of analysis. That was too late in the lifecycle. The better unit is the attempt ledger, joined to the business run and the notification decision. It answers the operational question and the finance question with the same identifiers.
Keep the causal envelope: correlation IDs, timestamps, dependency and phase, configured deadlines, outcome, retry decisions, and aggregated cost dimensions. Discard or aggressively age out raw bodies unless a documented investigation or legal obligation requires them. The trade-off is real. Removing bodies makes some payload-specific debugging slower, but retaining them by default enlarges the breach surface and turns an observability system into an unbounded data store.
A post-incident reconciliation should be able to prove three things: the customer-visible failure, the work already charged to that run, and why each retry was permitted. If those facts cannot be joined, the system may be observable in a dashboard while remaining unauditable. For payment and ledger systems, that distinction is not cosmetic; an operational alert must never become a second source of truth for money movement.
The least complex method that works is therefore a correlated attempt ledger with phase-aware timeout evidence, bounded retention, and an idempotent failure-notification decision. It is enough to locate the failing boundary, quantify retry amplification, and explain the remaining cost without recommending a particular observability product.
context package: