Import Incident Reconstruction: Compare Dashboard Events with a Custom Metrics API

A developer outlined a telemetry design for reconstructing failed scheduled customer-support imports, arguing that a single success/failure counter or result-count time series cannot distinguish an empty source, a skipped schedule, a rejected batch, or a run that never completed. The approach retains one record per scheduled attempt — with run ID, schedule ID, expected and actual timing, terminal state, result count, and a low-cardinality reason category — and derives dashboard metrics from that ordered event set, keeping raw error text out of metric labels to avoid high-cardinality dimensions.

Short answer: choose the approach that preserves enough evidence to reconstruct why a scheduled customer-support import stopped producing results; use event analytics when operators need to explore individual runs, and use a custom metrics API when the business question and aggregation rules are stable. The dashboard is the last step, not the decision. A chart can show results imported = 0 , yet that single number cannot distinguish an empty source, a skipped schedule, a rejected batch, or a run that never completed. For an AI support feature, those paths matter: an empty import may be valid, while a missing import can leave retrieval grounded in stale tickets. My evaluation constraint is simple: given only the retained telemetry, can an engineer explain one alert without reopening application logs or guessing? I first thought a success counter plus a failure counter would be enough. It isn't. The useful unit is an import attempt with identity, timing, outcome, and result count. Retain one record per scheduled attempt before deciding how to display it. At minimum, that record needs a stable run ID, schedule ID, expected time, actual start time, completion time, terminal state, result count, and a low-cardinality reason category. It should also carry the customer-support data source and deployment environment, because two simultaneous zero-result runs mean something different when they share a source than when they share only a deployment. There are two clocks. The scheduler clock answers, "Did a run begin when expected?" The production clock answers, "Did that run produce usable results before the freshness deadline?" Monitoring only completion events loses the first question. Monitoring only starts loses the second. Emit the expected schedule separately or derive it from versioned schedule configuration, then compare it with observed runs. Keep raw error text out of metric labels. Unbounded messages, ticket IDs, and run IDs create high-cardinality dimensions, which make aggregation expensive and dashboards noisy. Put those details in the per-run event or log record; expose a controlled reason such as auth , rate limit , validation , or unknown to the aggregate layer. RFC 5424 is useful here because it separates severity from structured message content and defines ordered severity levels. A business outcome still needs its own field: a technically successful run that imports zero results isn't automatically an operational success. One alert should therefore evaluate absence and outcome together. For example: no completed run within the freshness window, or the latest completed run produced zero results when the source was expected to be nonempty. The expectation must come from business state, not a hard-coded assumption that every import always has data. That's the hinge. The simple approach stores one time-series point per run: timestamp, result count, and perhaps a Boolean success flag. It makes a clean line chart. It also collapses the sequence that explains an incident. If a run starts at 09:00, receives a rate-limit response, retries, and completes at 09:08 with 240 results, the final count looks healthy while the eight-minute freshness gap disappears. If only the retry error is counted, the same run looks failed. Neither projection is a faithful reconstruction. Instead, model the run as an ordered set of facts and derive dashboard metrics from it. The focused Python example below uses local data and deterministic rules, so it works in a notebook and in a production evaluator without an SDK or a vendor-specific endpoint. python from dataclasses import dataclass from datetime import datetime, timedelta, timezone from typing import Literal State = Literal "scheduled", "started", "completed", "failed" @dataclass frozen=True class ImportEvent: run id: str occurred at: datetime state: State result count: int | None = None reason: str | None = None def reconstruct events: list ImportEvent , due at: datetime - dict str, object : ordered = sorted events, key=lambda event: event.occurred at terminal = next event for event in reversed ordered if event.state in {"completed", "failed"} , None, started = next event for event in ordered if event.state == "started" , None return { "run id": ordered 0 .run id, "started": started is not None, "terminal state": terminal.state if terminal else "missing", "result count": terminal.result count if terminal else None, "late": terminal is None or terminal.occurred at due at, "reason": terminal.reason if terminal else "no terminal event", } utc = timezone.utc run = ImportEvent "run-1042", datetime 2026, 8, 15, 9, 0, tzinfo=utc , "scheduled" , ImportEvent "run-1042", datetime 2026, 8, 15, 9, 1, tzinfo=utc , "started" , ImportEvent "run-1042", datetime 2026, 8, 15, 9, 8, tzinfo=utc , "completed", 240 , summary = reconstruct run, due at=datetime 2026, 8, 15, 9, 5, tzinfo=utc assert summary "terminal state" == "completed" assert summary "result count" == 240 assert summary "late" is True The reducer deliberately reports a completed run as late. That distinction gives an alert evaluator a precise input without turning the dashboard into a workflow engine. It also makes replay possible: when the definition of "late" changes from five minutes to ten, recompute the projection from retained events instead of rewriting history. Now trace what a responder would see at 09:06. The schedule says run-1042 was due at 09:00, the start arrived at 09:01, and no completion has arrived, so the evaluator opens a late-run alert with no terminal event ; the dashboard must not call the run failed because the evidence does not yet support that claim. At 09:08 the completion event arrives with 240 results. The reducer closes the missing-terminal condition but preserves late = True , letting the incident view show both the recovered business outcome and the freshness breach. If delivery duplicates that completion at 09:09, event identity and an idempotent projection prevent another 240 results from entering the total. If events arrive out of order, their occurrence timestamps restore the run sequence while ingestion timestamps still reveal pipeline delay. One small timeline answers four questions — whether scheduling happened, whether work began, whether results became usable, and whether the telemetry itself was delayed — that a lone counter cannot answer. Retries need the same discipline. The AWS Builders' Library explains why retries are selfish and can multiply load across service layers, and why backoff with jitter helps spread repeated requests. For monitoring, record attempts under the same logical run ID and count the final business outcome once. Otherwise, a three-attempt success becomes two failures plus one success, inflating both sides of a reliability chart. 429 can be a reason category for an attempt; it should not silently become the terminal state of the logical import if a later attempt completes. The named tools in this question sit in different operating models. Mixpanel and Amplitude are candidates for event-oriented exploration; Metabase and Redash are candidates for querying data already available to a database or warehouse; a custom metrics API is an application-owned projection. That taxonomy is more useful than a feature checklist because incident reconstruction depends on where the durable evidence lives and who owns its schema. | Approach | Best fit for this import monitor | Incident-reconstruction boundary | Main trade-off | |---|---|---|---| | Event analytics | Operators explore run sequences, segments, and changing questions | Retained events can preserve the path from schedule to result | Event governance and cardinality need active control | | Query dashboard | Run facts already land in a queryable store | Reconstruction is as complete as the stored tables and joins | Alert evaluation and freshness semantics may need separate ownership | | Custom metrics API | A small, stable set of business indicators drives automation | The team defines exact projections and can retain raw events elsewhere | The team owns schema evolution, uptime, access control, and dashboard UX | Product choice comes later. Compare Mixpanel and Amplitude by validating how each candidate retains, exports, and governs the specific run-event schema you need. Compare Metabase and Redash by testing the actual incident query against your store, including late-arriving events and access boundaries. I'm not sure a generic feature matrix can settle those questions; a replay of representative runs will. Your mileage may vary when the organization already has one of these data paths under reliable ownership. The catch is that event analytics is not suitable when the telemetry contains sensitive support content that cannot enter that system, or when operators need a tightly controlled operational endpoint rather than exploratory analysis. A custom API is not suitable when questions change weekly and the team cannot staff schema, storage, authorization, and on-call ownership. Stick with a query dashboard when the warehouse is already the timely source of truth and SQL gives responders enough detail; don't add another serving layer merely to draw the same aggregate. Treat the monitor like an eval harness. Build a compact corpus of run histories and assert the alert decision, reconstruction fields, and dashboard projection. Include an on-time nonempty completion, an on-time empty completion that is allowed by business state, a missed schedule, a started run with no terminal event, a late success, duplicate delivery, out-of-order delivery, and a retry that eventually succeeds. These are scenario classes, not invented production frequencies. Then test idempotency. Replaying the same run id and event identity must not double the imported-result total. Test ordering by delivering completion before start. Test the boundary at exactly the freshness deadline. Finally, test configuration changes: the evaluator should retain which schedule version and threshold produced an alert, or a later responder may be unable to reproduce the decision. Prompt cost belongs in this loop if an AI component summarizes incidents. Keep alert detection deterministic; pass the reconstructed, bounded run summary to the model only after the threshold fires. Measure input tokens, output tokens, summary usefulness, and unsupported claims against the eval corpus. Don't ship raw ticket bodies merely to explain a scheduler state. A concise structured summary is easier to redact, cheaper to evaluate repeatedly, and less likely to distract the model with unrelated customer language. Sparse alerts deserve special care. A missing event cannot carry a severity level, so a separate evaluator has to notice the absence and emit an alert record. Map that record into the team's existing severity policy rather than inventing a second scale. RFC 5424 defines severity numerically from Emergency through Debug, but it does not decide whether one stale support import is critical for your business. That policy requires an explicit freshness objective and impact model. Short test suites lie. Before deployment, run the same corpus against the full path: event ingestion, storage, projection, alert evaluation, and the view responders open. A unit test of the reducer cannot reveal a delayed pipeline or a dashboard cache whose refresh interval exceeds the alert window. Roll out with shadow evaluation first, compare decisions with the current monitor, and preserve enough identifiers to explain disagreements without placing those high-cardinality IDs into aggregate labels. Measure reconstruction completeness before dashboard polish: the proportion of alerts for which the run, expected schedule, terminal outcome, result count, reason category, and policy version are all available. Also measure detection delay, late-event rate, duplicate-event rate, query latency during an incident, retained-event volume, and the engineering time spent changing the schema. For AI-generated summaries, add token use and evaluator-scored factual consistency. No single approach wins those dimensions. Choose event analytics when exploration and changing segmentation justify event governance. Choose a query dashboard when durable run facts already exist and the team can express the operational question safely in that data layer. Choose a custom metrics API when the contract is narrow, stable, and important enough to own. The dashboard can stay simple because the evidence underneath it isn't.