# The alarm wasn't silent. It was lying.

> Source: <https://dev.to/mk023/the-alarm-wasnt-silent-it-was-lying-iam>
> Published: 2026-08-20 02:12:27+00:00

*This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.*

[ agentic-os](https://github.com/MK023/agentic-os) is a small self-hosted observability stack I run on Railway: Prometheus, Grafana, an OpenTelemetry Collector, a cloudflared tunnel, and a public status API that publishes three numbers about my Claude Code usage — requests, tokens, cost.

It is a personal project, but it is a real deployment with real uptime, and the entire purpose of it is to be the thing that tells me when something is wrong. That framing matters for what follows.

On 13 August, Prometheus stopped writing blocks to disk.

Its TSDB compaction started failing with `no space left on device`

, once a minute, and never stopped. Sixty failures per hour — which is not a degradation, it is *every single attempt*.

What made it interesting is that **nothing looked broken.**

Prometheus answers queries out of the head block, which lives in memory. So from the outside, a database that had not persisted anything in hours was indistinguishable from a healthy one. The failure was invisible to every instrument pointed at it.

To be fair to Prometheus: a restart is not instant data loss, because the head is reconstructed from the write-ahead log. But the WAL is on the same volume that had no space left, which makes "it will recover on restart" a bet on the one resource that had already run out. I did not want to find out which way that bet resolved.

I found it by reading container logs by hand. That was the only way available, and the reason is embarrassing in hindsight: `prometheus_tsdb_*`

metrics were not being scraped by anything. Prometheus was the one service in the stack that nobody was watching. There was no series to write an alert rule against, because nothing was collecting the series.

So I fixed it: scrape Prometheus with Prometheus, cap retention by size and not only by time, and add a watchdog that reports the condition to Sentry.

**And then the real bug showed up.**

Sentry received one event. Then silence for six days.

I checked the issue on 19 August and it read *last seen five days ago*. That is exactly what a fault that has cleared looks like. I very nearly closed it.

The dedup in my watchdog was a module-level set — one notification per process lifetime. The watchdog fired once, added the key, and never spoke again for as long as that process lived. The four events I did receive were not four detections. They were four **restarts**. Between them, compaction was failing sixty times an hour, continuously, and the alerting path had nothing left to say.

The alarm was not silent. Silence would have been honest. It was actively reporting a *shape* — one event, then nothing — that means "transient, resolved" to every human being who reads an issue tracker.

Being precise about the failure mode: the bug is not that I deduplicated. Deduplication is correct — a compaction failing every minute must not produce 1,440 Sentry events a day. The bug is that I picked a deduplication **window** of "forever," and forever cannot distinguish "happened once" from "still happening."

Alongside the correctness fix there was a cost defect in the same code path, covered below.

`fix(prometheus): cap retention by size, not just time — the volume is full`

`fix(sonda): l'allarme taceva da sei giorni, la retention era tarata sul volume vecchio, e il percorso di errore costava più di quello di successo`

Heads up for anyone clicking through: this repo's commit messages and code comments are in Italian. PR #86 reads

"the alarm had been silent for six days, retention was sized for the old volume, and the error path cost more than the success path."The code and the diffs speak for themselves; the prose around them does not, and I would rather say so than quietly present a translation as the original.

The watchdog query:

```
PERSISTENCE_QUERY = "sum(increase(prometheus_tsdb_compactions_failed_total[1h]))"
```

The dedup, before — one notification per process lifetime:

```
_INFRA_ALERTS_SENT: set[str] = set()
```

And after, a per-key timestamp that expires (comments stripped here for length — they are in Italian in the source, and I translate the one that matters below):

```
INFRA_ALERT_INTERVAL_S = 3600.0

_INFRA_ALERTS_SENT: dict[str, float] = {}

async def _report_infra_throttled(key: str, exc: Exception) -> None:
    last_sent = _INFRA_ALERTS_SENT.get(key)
    now = time.monotonic()
    if last_sent is not None and now - last_sent < INFRA_ALERT_INTERVAL_S:
        return
    _INFRA_ALERTS_SENT[key] = now
    await capture_exception(exc, tags={"endpoint": "status"})
```

The line that took the longest to decide is the placement of `_INFRA_ALERTS_SENT[key] = now`

, and the comment above it in the source reads, translated:

Marked BEFORE sending, deliberately:`capture_exception`

swallows every delivery error, so an unreachable Sentry would cost the hour of silence either way — but marking after would turn a 5xx on their side into a burst of retries on every public request. Losing an event is preferable to amplifying someone else's outage.

**The dedup window gets a TTL.** One hour is not an arbitrary number. It is the coarsest interval at which a Sentry alert rule based on *event frequency* — rather than on issue creation — still has something to count. A once-per-process alarm gives a frequency rule nothing to work with, which was the other half of the six-day silence.

**The order of operations inside the throttle is a deliberate trade.** The timestamp is written *before* the send, not after. `capture_exception`

swallows delivery errors, so if Sentry itself is unreachable the hour of silence is paid either way; but marking after the send would turn a 5xx on their side into a retry burst on every public request. Losing one event is better than amplifying someone else's outage. Same fail-open posture the rest of the project uses.

** time.monotonic(), not time.time().** This measures a duration, not a moment. A system clock stepped backwards would freeze the alarm until the gap was made up — an ugly, seasonal, nearly untestable bug to inherit, for no benefit.

**The watchdog left the hot path.** It now runs at most once a minute instead of on every request. This is a cost bug as much as a correctness one: the project moved to a usage-based plan on 19 August and the public status endpoint is not rate-limited, so four Prometheus queries per request instead of three is a spend multiplier available to anyone who knows the URL.

**Retention was retuned to the disk that actually exists.** 7d/300MB became 30d/3GB. The old numbers were sized for a 500 MB volume; the volume is 5 GB now. Left alone they would have truncated history to about 6% of a disk already being paid for. Time is now what expires first, with size as the backstop that prevents the original failure from recurring.

**A watchdog for the watchdog.** A fourth exception type came out of this — `PrometheusWatchdogBlind`

— which fires when the watchdog's *own input* is missing. A watchdog whose query returns no series reports "everything is fine" forever, and I had just spent a week learning what that costs.

The takeaway I actually carry from this: every layer was blind in a *different* way, which is why it survived so long. It was not one mistake repeated three times.

Green dashboards are not evidence. They are the *absence* of evidence, and the two are only the same thing once you have proven your instruments can go red.

**Error Monitoring** is not decoration on this one. To be exact about what it did and did not do: the *first* detection was mine, reading logs by hand, because nothing was scraping Prometheus yet. What Sentry made visible was everything after that — the recurrence, the persistence, and ultimately the second bug, which only exists as a pattern across events and could never have been read off a single log line.

The condition is modelled as a typed exception that is *captured, never raised* — the status endpoint has to keep answering while it complains:

```
class PrometheusNotPersisting(Exception):
    """Prometheus answers queries but cannot write blocks. Reported, never raised."""

class PrometheusWatchdogBlind(Exception):
    """The watchdog's own input is missing. Reported, never raised."""
```

Events are tagged by `endpoint`

and `environment`

so an infrastructure fault never lands in the same bucket as an upstream API error.

**The event list is what solved the second bug.** Four events, their timestamps, and the release SHA attached to each:

| Event | Release | Failures/hour |
|---|---|---|
| 13 Aug 07:42 | `289fa20` |
59 |
| 13 Aug 23:08 | `14a1141` |
60 |
| 13 Aug 23:40 | `f491baa` |
60 |
(six days of silence) |
||
| 19 Aug 13:24 | `757758c` |
60 |

Correlating those four timestamps against my deploy history is what proved the events were **restarts, not detections**. I could not have reached that from logs — they had already rotated. Release tagging turned four data points into a diagnosis.

**Seer**, run against the same issue after the fact, returned:

Prometheus cannot write TSDB blocks due to a full or unavailable storage volume, causing all compaction attempts to fail.

That is correct, and it took one command against a single event. It is also — precisely — the *first* bug, the boring one. Seer answered the question an issue can ask on its own: **why is this event happening.** It could not answer the question that cost me six days, because that question is *why am I only seeing four of these* — a question about events that were never sent. No single event contains its own absence.

I find that a fair division of labour rather than a limitation. The automated analysis collapsed the mechanical cause to one line and freed me to look at the part that needed a human holding a deploy history next to a timestamp column.

**What Sentry taught me about alerting.** An issue page communicates through shape, not only content. "Last seen five days ago" is a claim about the world. If the thing producing your events cannot repeat itself, your issue tracker will make that claim on your behalf — and it will be wrong. The one-hour throttle exists so that a Sentry frequency rule has a heartbeat to measure, which is the difference between an issue tracker that records faults and one that can alert on them.

*Written with Claude Code as a pair. The investigation and the reasoning behind each decision are in the PR descriptions and the code comments; I have left the AI collaboration visible in the commit trail rather than tidied out of it.*