{"slug": "the-alarm-wasn-t-silent-it-was-lying", "title": "The alarm wasn't silent. It was lying.", "summary": "A developer's self-hosted observability stack on Railway silently failed for six days because Prometheus stopped writing to disk due to a full volume, yet queries continued from memory, masking the issue. The developer found the failure only by reading container logs manually and discovered that Prometheus was not scraping its own metrics, leaving no alerting series. After fixing that, a deduplication bug in the watchdog suppressed repeated alerts, making the alarm appear transient when it was continuous.", "body_md": "*This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.*\n\n[ agentic-os](https://github.com/MK023/agentic-os) is a small self-hosted observability stack I run on Railway: Prometheus, Grafana, an OpenTelemetry Collector, a cloudflared tunnel, and a public status API that publishes three numbers about my Claude Code usage — requests, tokens, cost.\n\nIt is a personal project, but it is a real deployment with real uptime, and the entire purpose of it is to be the thing that tells me when something is wrong. That framing matters for what follows.\n\nOn 13 August, Prometheus stopped writing blocks to disk.\n\nIts TSDB compaction started failing with `no space left on device`\n\n, once a minute, and never stopped. Sixty failures per hour — which is not a degradation, it is *every single attempt*.\n\nWhat made it interesting is that **nothing looked broken.**\n\nPrometheus answers queries out of the head block, which lives in memory. So from the outside, a database that had not persisted anything in hours was indistinguishable from a healthy one. The failure was invisible to every instrument pointed at it.\n\nTo be fair to Prometheus: a restart is not instant data loss, because the head is reconstructed from the write-ahead log. But the WAL is on the same volume that had no space left, which makes \"it will recover on restart\" a bet on the one resource that had already run out. I did not want to find out which way that bet resolved.\n\nI found it by reading container logs by hand. That was the only way available, and the reason is embarrassing in hindsight: `prometheus_tsdb_*`\n\nmetrics were not being scraped by anything. Prometheus was the one service in the stack that nobody was watching. There was no series to write an alert rule against, because nothing was collecting the series.\n\nSo I fixed it: scrape Prometheus with Prometheus, cap retention by size and not only by time, and add a watchdog that reports the condition to Sentry.\n\n**And then the real bug showed up.**\n\nSentry received one event. Then silence for six days.\n\nI checked the issue on 19 August and it read *last seen five days ago*. That is exactly what a fault that has cleared looks like. I very nearly closed it.\n\nThe dedup in my watchdog was a module-level set — one notification per process lifetime. The watchdog fired once, added the key, and never spoke again for as long as that process lived. The four events I did receive were not four detections. They were four **restarts**. Between them, compaction was failing sixty times an hour, continuously, and the alerting path had nothing left to say.\n\nThe alarm was not silent. Silence would have been honest. It was actively reporting a *shape* — one event, then nothing — that means \"transient, resolved\" to every human being who reads an issue tracker.\n\nBeing precise about the failure mode: the bug is not that I deduplicated. Deduplication is correct — a compaction failing every minute must not produce 1,440 Sentry events a day. The bug is that I picked a deduplication **window** of \"forever,\" and forever cannot distinguish \"happened once\" from \"still happening.\"\n\nAlongside the correctness fix there was a cost defect in the same code path, covered below.\n\n`fix(prometheus): cap retention by size, not just time — the volume is full`\n\n`fix(sonda): l'allarme taceva da sei giorni, la retention era tarata sul volume vecchio, e il percorso di errore costava più di quello di successo`\n\nHeads up for anyone clicking through: this repo's commit messages and code comments are in Italian. PR #86 reads\n\n\"the alarm had been silent for six days, retention was sized for the old volume, and the error path cost more than the success path.\"The code and the diffs speak for themselves; the prose around them does not, and I would rather say so than quietly present a translation as the original.\n\nThe watchdog query:\n\n```\nPERSISTENCE_QUERY = \"sum(increase(prometheus_tsdb_compactions_failed_total[1h]))\"\n```\n\nThe dedup, before — one notification per process lifetime:\n\n```\n_INFRA_ALERTS_SENT: set[str] = set()\n```\n\nAnd after, a per-key timestamp that expires (comments stripped here for length — they are in Italian in the source, and I translate the one that matters below):\n\n```\nINFRA_ALERT_INTERVAL_S = 3600.0\n\n_INFRA_ALERTS_SENT: dict[str, float] = {}\n\nasync def _report_infra_throttled(key: str, exc: Exception) -> None:\n    last_sent = _INFRA_ALERTS_SENT.get(key)\n    now = time.monotonic()\n    if last_sent is not None and now - last_sent < INFRA_ALERT_INTERVAL_S:\n        return\n    _INFRA_ALERTS_SENT[key] = now\n    await capture_exception(exc, tags={\"endpoint\": \"status\"})\n```\n\nThe line that took the longest to decide is the placement of `_INFRA_ALERTS_SENT[key] = now`\n\n, and the comment above it in the source reads, translated:\n\nMarked BEFORE sending, deliberately:`capture_exception`\n\nswallows every delivery error, so an unreachable Sentry would cost the hour of silence either way — but marking after would turn a 5xx on their side into a burst of retries on every public request. Losing an event is preferable to amplifying someone else's outage.\n\n**The dedup window gets a TTL.** One hour is not an arbitrary number. It is the coarsest interval at which a Sentry alert rule based on *event frequency* — rather than on issue creation — still has something to count. A once-per-process alarm gives a frequency rule nothing to work with, which was the other half of the six-day silence.\n\n**The order of operations inside the throttle is a deliberate trade.** The timestamp is written *before* the send, not after. `capture_exception`\n\nswallows delivery errors, so if Sentry itself is unreachable the hour of silence is paid either way; but marking after the send would turn a 5xx on their side into a retry burst on every public request. Losing one event is better than amplifying someone else's outage. Same fail-open posture the rest of the project uses.\n\n** time.monotonic(), not time.time().** This measures a duration, not a moment. A system clock stepped backwards would freeze the alarm until the gap was made up — an ugly, seasonal, nearly untestable bug to inherit, for no benefit.\n\n**The watchdog left the hot path.** It now runs at most once a minute instead of on every request. This is a cost bug as much as a correctness one: the project moved to a usage-based plan on 19 August and the public status endpoint is not rate-limited, so four Prometheus queries per request instead of three is a spend multiplier available to anyone who knows the URL.\n\n**Retention was retuned to the disk that actually exists.** 7d/300MB became 30d/3GB. The old numbers were sized for a 500 MB volume; the volume is 5 GB now. Left alone they would have truncated history to about 6% of a disk already being paid for. Time is now what expires first, with size as the backstop that prevents the original failure from recurring.\n\n**A watchdog for the watchdog.** A fourth exception type came out of this — `PrometheusWatchdogBlind`\n\n— which fires when the watchdog's *own input* is missing. A watchdog whose query returns no series reports \"everything is fine\" forever, and I had just spent a week learning what that costs.\n\nThe takeaway I actually carry from this: every layer was blind in a *different* way, which is why it survived so long. It was not one mistake repeated three times.\n\nGreen dashboards are not evidence. They are the *absence* of evidence, and the two are only the same thing once you have proven your instruments can go red.\n\n**Error Monitoring** is not decoration on this one. To be exact about what it did and did not do: the *first* detection was mine, reading logs by hand, because nothing was scraping Prometheus yet. What Sentry made visible was everything after that — the recurrence, the persistence, and ultimately the second bug, which only exists as a pattern across events and could never have been read off a single log line.\n\nThe condition is modelled as a typed exception that is *captured, never raised* — the status endpoint has to keep answering while it complains:\n\n```\nclass PrometheusNotPersisting(Exception):\n    \"\"\"Prometheus answers queries but cannot write blocks. Reported, never raised.\"\"\"\n\nclass PrometheusWatchdogBlind(Exception):\n    \"\"\"The watchdog's own input is missing. Reported, never raised.\"\"\"\n```\n\nEvents are tagged by `endpoint`\n\nand `environment`\n\nso an infrastructure fault never lands in the same bucket as an upstream API error.\n\n**The event list is what solved the second bug.** Four events, their timestamps, and the release SHA attached to each:\n\n| Event | Release | Failures/hour |\n|---|---|---|\n| 13 Aug 07:42 | `289fa20` |\n59 |\n| 13 Aug 23:08 | `14a1141` |\n60 |\n| 13 Aug 23:40 | `f491baa` |\n60 |\n(six days of silence) |\n||\n| 19 Aug 13:24 | `757758c` |\n60 |\n\nCorrelating those four timestamps against my deploy history is what proved the events were **restarts, not detections**. I could not have reached that from logs — they had already rotated. Release tagging turned four data points into a diagnosis.\n\n**Seer**, run against the same issue after the fact, returned:\n\nPrometheus cannot write TSDB blocks due to a full or unavailable storage volume, causing all compaction attempts to fail.\n\nThat is correct, and it took one command against a single event. It is also — precisely — the *first* bug, the boring one. Seer answered the question an issue can ask on its own: **why is this event happening.** It could not answer the question that cost me six days, because that question is *why am I only seeing four of these* — a question about events that were never sent. No single event contains its own absence.\n\nI find that a fair division of labour rather than a limitation. The automated analysis collapsed the mechanical cause to one line and freed me to look at the part that needed a human holding a deploy history next to a timestamp column.\n\n**What Sentry taught me about alerting.** An issue page communicates through shape, not only content. \"Last seen five days ago\" is a claim about the world. If the thing producing your events cannot repeat itself, your issue tracker will make that claim on your behalf — and it will be wrong. The one-hour throttle exists so that a Sentry frequency rule has a heartbeat to measure, which is the difference between an issue tracker that records faults and one that can alert on them.\n\n*Written with Claude Code as a pair. The investigation and the reasoning behind each decision are in the PR descriptions and the code comments; I have left the AI collaboration visible in the commit trail rather than tidied out of it.*", "url": "https://wpnews.pro/news/the-alarm-wasn-t-silent-it-was-lying", "canonical_source": "https://dev.to/mk023/the-alarm-wasnt-silent-it-was-lying-iam", "published_at": "2026-08-20 02:12:27+00:00", "updated_at": "2026-08-20 02:43:12.070469+00:00", "lang": "en", "topics": ["developer-tools", "mlops", "ai-infrastructure"], "entities": ["Prometheus", "Grafana", "OpenTelemetry Collector", "cloudflared", "Railway", "Sentry", "Claude Code", "MK023"], "alternates": {"html": "https://wpnews.pro/news/the-alarm-wasn-t-silent-it-was-lying", "markdown": "https://wpnews.pro/news/the-alarm-wasn-t-silent-it-was-lying.md", "text": "https://wpnews.pro/news/the-alarm-wasn-t-silent-it-was-lying.txt", "jsonld": "https://wpnews.pro/news/the-alarm-wasn-t-silent-it-was-lying.jsonld"}}