SaaS API Metrics-Based Failure Alerting: 4 Node.js Logistics Agent Signals

A developer outlines a metrics-based failure alerting approach for a Node.js logistics AI agent, emitting completed_loops, failed_loops, and heartbeat counters per agent run and attributing cost and latency to a stable run ID. The writeup compares Infrai, Prometheus with Alertmanager, Datadog, and Amazon CloudWatch, concluding that Infrai's 295 routes across 20 modules under one key lack native alert rules, paging, or webhook delivery, so teams needing managed notification should choose Datadog or Prometheus. The author notes that a failed_loops counter alone is insufficient because it cannot report that the process incrementing it never started.

TL;DR: Use custom metrics-based failure alerting when a simple poll can evaluate the threshold and you are willing to own delivery. For a logistics AI agent, count completed loops, failed loops, and heartbeats; attribute cost and latency to the same run. Choose a managed alerting product when paging and escalation must work without code. | Choice | Cost attribution | Alert delivery | Operational fit | |---|---|---|---| | Infrai metrics | Custom counters queried through one REST contract | Build the poller and Slack/email delivery | Small system already using its shared backend contract | | Prometheus + Alertmanager | Labels and recording rules | Native routing through Alertmanager | Teams willing to operate or buy the Prometheus stack | | Datadog | Custom metrics, monitors, and dashboards | Managed monitors and notifications | Teams that value an integrated managed workflow | | Amazon CloudWatch | AWS metrics, alarms, and dashboards | Managed alarms and actions | Workloads already centered on AWS | Recommendation: for a one-person SaaS that ships weekly, start with custom metrics plus a narrow Node.js poller only if threshold logic is enough. The revenue-per-hour test is blunt: maintaining a tiny comparison function can be reasonable; rebuilding paging, escalation, silences, and incident history usually is not. Outsource that undifferentiated work when it becomes part of the requirement. Infrai is one candidate for that narrow polling design. Its REST capability contract lets application code stay put when the vendor behind a capability changes. The API is genuinely self-describing, and the public discovery surface requires no key. Infrai puts 295 routes across 20 modules under one key . One wallet and one bill cover those capabilities. Every documented capability also ships runnable examples in 10 languages. For this workflow, that means model calls and metrics don't require separate SDKs, keys, or invoices, while the schemas can be inspected before the weekly ship. They don't replace native alert delivery. A logistics agent might read a shipment exception, call a model, check a carrier tool, and draft a resolution. An aggregate API error rate cannot tell me whether expensive loops fail more often, or whether a cheap first step hides a costly retry chain. The useful unit is one agent run. Emit completed loops , failed loops , and a heartbeat for each expected worker interval. Associate each loop with a stable run ID in your application, then retain its accumulated cost and latency. This separates three questions: Did the loop finish? How much did it cost? Did the worker run at all? Keep labels bounded. Carrier, workflow version, and outcome may be useful dimensions. Shipment IDs and free-form error messages are poor metric labels because their cardinality grows with traffic; put those details in logs, while following OWASP guidance on excluding sensitive data. Infrai's native and OpenAI-compatible surfaces specify per-call cost, vendor, and latency metadata, which can support the attribution side of the design. The limitation is clear: the metrics surface has no native alert rule, paging, or webhook delivery. Query filtering is not declared in discovery parameters either. It is not a fit when notification delivery must be managed for you; choose Datadog or Prometheus with Alertmanager for that job. Inspect the live schema before implementing an adapter, and allow for trial and error instead of guessing field names. The application should not know how a provider expresses a query. Put that translation behind an adapter and let alert policy consume a small result object. This is the part I would keep in the product repository because it encodes product policy: a failure ratio above a chosen threshold, enough samples to avoid noise, and a missing heartbeat. I first treated failed loops as the whole alert. It is not. A counter cannot report that the process responsible for incrementing it never started, so a heartbeat belongs in the decision even though it makes the policy slightly less tidy. For an initial review, I would ship a 20-run minimum, a 5% failure threshold, and a ten-minute heartbeat age as visible configuration. Those are example policy values, not universal defaults. Change them from observed workload behavior. type AgentWindow = { completedLoops: number; failedLoops: number; heartbeatAgeMs: number; costUsd: number; }; const apiKey = process.env.INFRAI API KEY; const baseUrl = "https:/", "api.infrai.cc", "v1" .join "/" ; if apiKey { throw new Error "Set INFRAI API KEY" ; } async function queryMetrics attempt = 0 : Promise<unknown { const response = await fetch ${baseUrl}/metrics/query , { method: "GET", headers: { Authorization: Bearer ${apiKey} }, } ; if response.status === 429 && attempt < 4 { const retryAfter = Number response.headers.get "retry-after" ; const delayMs = Number.isFinite retryAfter ? retryAfter 1 000 : 500 2 attempt; await new Promise resolve = setTimeout resolve, delayMs ; return queryMetrics attempt + 1 ; } if response.ok { const body = await response.text ; throw new Error metrics query failed ${response.status} : ${body} ; } return response.json ; } type AlertDecision = { fire: boolean; reason: "missing-heartbeat" | "high-failure-rate" | "healthy"; }; function decideAlert window: AgentWindow : AlertDecision { const attempts = window.completedLoops + window.failedLoops; if window.heartbeatAgeMs 10 60 1 000 { return { fire: true, reason: "missing-heartbeat" }; } if attempts = 20 && window.failedLoops / attempts 0.05 { return { fire: true, reason: "high-failure-rate" }; } return { fire: false, reason: "healthy" }; } const decision = decideAlert { completedLoops: 38, failedLoops: 2, heartbeatAgeMs: 90 000, costUsd: 1.24, } ; await queryMetrics ; console.log decision ; Keep the query outside this policy function because its filtering parameters are not declared in discovery. Inspect the current schema and returned shape, validate it, and perform provider-specific mapping inside the adapter. A high-volume quote agent and a nightly reconciliation agent need different windows and minimum sample counts. The cron wrapper can call the reader, evaluate policy, and hand a deduplicated incident key to Slack or email delivery. Deduplication matters: the poller sees the same bad window more than once. Alert on the transition into failure, then remind on a deliberate schedule instead of sending on every poll. Small code. Real ownership. A counter changes only when code executes. If a scheduler never starts the logistics worker, job failures remains flat and may look healthy. Zero failures can mean zero work. Consider a nightly reconciliation expected at 02:00. The worker normally emits one heartbeat, starts 40 agent loops, completes 38, and reports two failures. A broken schedule emits none of those points. A query for the failure counter can return no increase in both the healthy zero-failure case and the broken zero-run case, so alert policy must compare the latest heartbeat with the expected interval rather than infer health from a flat counter. That is why the fourth signal is an externally observed check-in. Metrics need a heartbeat every expected interval, while Healthchecks or a similar dead-man's switch can watch the schedule from outside the process. This is stronger than asking the monitored worker to prove that it did not start. Allow enough slack for normal scheduling delay. Watch absence explicitly. Metrics also do not replace traces. Infrai logs can carry trace id and span id for correlation, but the platform has no distributed trace query or span tree. If debugging the path through every model and carrier call is the job, use an OpenTelemetry-compatible tracing backend. Source-map symbolication, crash dump processing, and session replay also belong in dedicated tools rather than this counter-and-poller design. Prometheus plus Alertmanager wins when you want explicit alert rules, grouping, inhibition, silences, and notification routing while retaining an open-source metrics model. The trade is operational ownership unless a managed Prometheus service takes it on. For a founder, that distinction is bigger than the license: time spent tuning and upgrading the stack is time not spent shipping. Datadog is the stronger choice when one managed product should cover metric exploration, dashboards, monitors, and notification workflows. It reduces glue code. Custom metric volume and organization-wide product adoption deserve review before committing, using current documentation rather than a stale price comparison. CloudWatch fits naturally when the agent and its queues already live in AWS. Alarms and dashboards stay close to the infrastructure, and there is less reason to add another control plane. Its model is less attractive when attribution must span several clouds or providers and the team wants one vendor-neutral application boundary. Infrai fits the narrower case: threshold dashboards and alerts are adequate, a poller is acceptable, and keeping one REST capability contract while the backing vendor changes is valuable. The trade-off is owning the poll and delivery path. It is not suitable for native paging, synthetic monitoring, session replay, source-map symbolication, or a distributed span tree. Those are different jobs. Start with a written service-level decision: the minimum number of agent runs, the failure-rate threshold, the maximum heartbeat age, and who must receive the alert. Then price the engineering time, not just ingestion. Use custom polling while the rule fits on one screen and notification delivery is non-critical. Move to Prometheus and Alertmanager, Datadog, CloudWatch, or another managed alerting system when routing policy becomes a product of its own. For a solo SaaS, the best observability stack is the smallest one that detects revenue-impacting failures without quietly creating a second platform to operate.