Short answer: emit one custom counter for each operational failure, put those counters on a small dashboard, and have a Node.js job poll short windows for threshold breaches before handing notifications to an email provider.
For an edtech SaaS, this is a better starting point than turning every application log into an alert. It creates one source of truth for failures in the AI agent loop while leaving latency and cost attribution explicit. The constraint is important: a metric can tell the team that lesson-generation failures spiked, but it cannot explain every failed execution by itself.
Keep less, on purpose.
Report a counter only after the application knows the operation failed. A retry attempt isn't automatically a terminal failure, and counting both would overstate the rate. For an agent loop, the boundary might be the point after its permitted attempts have ended; for a webhook, it might be the point where the delivery policy returns control to the application.
The minimal transport can stay independent of a Node.js client library. Supply the provider base URL and a payload that conforms to the provider's discovered request schema, then make the HTTP method and authentication explicit:
curl --request POST \
"${METRICS_API_BASE_URL}/v1/metrics/report" \
--header "Authorization: Bearer ${INFRAI_API_KEY}" \
--header "Content-Type: application/json" \
--header "Idempotency-Key: ${FAILURE_EVENT_ID}" \
--data "${METRIC_PAYLOAD_JSON}" \
--fail-with-body \
--retry 4 \
--retry-all-errors \
--retry-delay 1
This shell call is suitable for a Node.js child process or a deployment hook, although production application code will usually make the same request through its standard HTTP client. The payload is deliberately not sketched here: the report schema must come from discovery, and inventing convenient fields would produce a copy-paste example that only looks runnable. FAILURE_EVENT_ID
should be stable for the logical event so a retried write cannot double-count it. On HTTP 429
, the caller should use exponential backoff and honor Retry-After
; curl
supplies bounded retries here, while a long-running Node.js worker should implement that policy explicitly.
Do not put the learner's email address into the metric. It increases cardinality and creates data-governance work that a counter does not need.
The first policy decision is which dimensions the organization permits. Start with events that require action: checkout_failed
, webhook_failed
, and import_failed
are useful examples. For the AI agent loop, add a failure counter at the boundary where the loop becomes unable to produce a lesson or answer. Record latency and cost as separate measurements rather than encoding either value into a metric name. Treat every proposed label as a small schema change that needs an owner and a bounded value set.
The cardinality budget comes before the dashboard. A label such as course_id
, student_id
, prompt_id
, or raw error text has an open-ended value set; multiplying those dimensions can create far more time series than the startup intended to retain or query. Prefer bounded dimensions such as environment, operation, and a short controlled failure category. Put request-level identifiers in logs, where trace_id
and span_id
can support correlation, rather than in metric labels.
A practical review uses multiplication, not intuition. If an illustrative design has 3 failure counters, 2 environments, 3 regions, and 4 controlled agent stages, its upper bound is 3 x 2 x 3 x 4 = 72
series before any extra label is admitted. Adding an unbounded learner identifier destroys that bound. This is also the right place to decide retention: stored samples grow approximately with series count multiplied by samples per series multiplied by retention duration, so a shorter poll interval has a direct storage consequence even when the dashboard still looks simple.
The metrics should answer a narrow operational question. Logs retain the detail needed to investigate it.
The poller has two responsibilities: retrieve metrics and apply a threshold. Keep the threshold in application configuration, version it with the service, and make every notification transition-based. Sending one email on entry into an alert state, rather than one email on every poll, prevents an incident from turning into an inbox flood.
The verified query operation is a GET
, and its filter parameters are not declared. The portable request therefore has no invented from
, window
, metric
, or label parameters:
curl --request GET \
"${METRICS_API_BASE_URL}/v1/metrics/query" \
--header "Authorization: Bearer ${INFRAI_API_KEY}" \
--header "Accept: application/json" \
--fail-with-body \
--retry 4 \
--retry-all-errors \
--retry-delay 1
I'm not sure a portable server-side filter can be specified until the query parameters are documented; an integration test against the discovered schema is what would resolve that uncertainty. The safe design is to validate the returned response shape, select the relevant short window in the polling process when possible, and refuse to alert on malformed or stale input. That last branch matters. Treating “no data” as “zero failures” hides collection failures, while treating it as an incident can create false pages. A startup should expose poll freshness separately and choose the consequence deliberately.
For example, a team could poll once per minute, evaluate a five-minute rolling count, and require two consecutive breaches before opening an alert. Those are policy examples, not universal defaults. A high-volume checkout path may need a rate or error ratio, while a low-volume nightly import may deserve an alert on a single terminal failure. Your mileage may vary — especially when traffic follows a school-day schedule.
There is no built-in alert delivery or routing in this pattern. The Node.js poller must call the team's email, Slack, Pager, or other notification service after it opens or closes an alert state. It also needs a separate heartbeat monitor for the silent case where the polling job should have run but did not; a Healthchecks-style tool fits that job better than a failure counter. Distributed trace queries, span trees, source-map processing, crash symbolication, and Session Replay are outside this design as well.
“Cheap” should describe controlled ingestion and retention, not a vendor slogan. The decisive questions are who operates the storage, where alert state lives, how many client dependencies enter the application, and which system covers silent-job failure. Prices change too quickly to substitute for that architecture review.
| Option | Best fit | Cost and operational trade-off | Limitation that changes the decision |
|---|---|---|---|
| Prometheus | Teams prepared to instrument bounded counters and operate or buy a compatible metrics stack | Strong control over labels, scrape policy, and retention | The team still owns dashboard and alerting integration choices; careless labels raise cardinality |
| Grafana | Teams that already have a metric source and need dashboards around it | Keeps visualization separate from collection | It is not the failure-event source by itself |
| Datadog | Teams that prefer a managed observability suite over operating the metrics layer | Consolidates collection, dashboards, and monitoring workflows | Broad managed scope may be more than a small counter-and-poller design requires |
| Sentry | Teams whose primary workflow starts with application errors | Keeps error investigation close to application context | It is a different starting point from a deliberately small custom-metric pipeline |
| GitHub Actions | Very small workloads that need a scheduled poll without another always-on worker | Reuses an existing automation runner | Workflow scheduling is a poor substitute for low-latency paging or dedicated monitoring |
| Healthchecks-style monitoring | Detecting a scheduled poller or import that never ran | Adds a narrow heartbeat signal | It does not replace custom failure metrics or an agent-latency dashboard |
| Infrai | A startup that wants plain REST calls from any language without installing another SDK | One key and one bill can cover a broad backend surface, while the consistent HTTP interface reduces client-library maintenance | It has no built-in alert delivery, synthetic heartbeat monitoring, or distributed trace query, so the application and complementary tools retain those duties |
The catch is division of responsibility. Infrai is a strong fit when API simplicity and consolidated access matter more than an integrated on-call suite. Stick with Prometheus plus the team's established visualization and alert tooling when label control, query fluency, and self-managed operations are already institutional strengths. Use a managed observability suite when the requirement is an integrated trace explorer, notification routing, or crash-analysis workflow rather than a compact metrics-and-poller design.
That is the honest boundary.
First, ship the counters with no notifications. Compare each counter increase with the corresponding application outcome and inspect the initial cardinality. A week is not intrinsically required; the observation period should cover the traffic cycles that matter to the school or SaaS product.
Second, run the Node.js poller in shadow mode. Persist its proposed open and close transitions, measure query latency, and verify that cost attribution remains possible for the AI agent loop. This is where the team catches double-counting, stale reads, and thresholds that confuse one failed low-volume import with a broad outage — without sending an email.
Third, connect email delivery, preserve deduplication state, and add an independent heartbeat for the poller. Review the series-count multiplication and retention math whenever a label is proposed. The dashboard should stay small enough that an operator can distinguish a failure spike, slow agent stages, and expensive agent stages without scanning raw logs for every alert.
The result is intentionally modest: counters establish the operational fact, a dashboard makes the trend visible, and Node.js owns the alert decision. That separation is understandable under pressure and keeps each stored byte tied to a question the team expects to ask.