{"slug": "the-cluster-you-do-not-watch", "title": "The Cluster You Do Not Watch", "summary": "A site reliability engineer at an unnamed company describes an operations architecture that uses an AI agent to read metrics, logs, and Kubernetes state and propose changes via pull requests, with no direct write access, running on under US$10 a month of infrastructure. The system, which relies on VictoriaMetrics, Loki, and Grafana, has reduced the need for human dashboard monitoring, and the author notes that a check that cannot fail is not a check.", "body_md": "# The Cluster You Do Not Watch\n\nAugust 14, 2026\n\nI almost never open Grafana any more. This is the operations architecture that made that true, on under US$10 a month of infrastructure — an agent that reads everything and writes nothing directly, alerts precise enough to act on, repairs that must prove they are safe before running, and one rule that turns out to be load-bearing: a check that cannot fail is not a check. None of it depends on which Kubernetes you run.\n\nAn operations loop drawn as a ring around a central AI operator: alerting into chat, into an agent, into a pull request, through CI, into a reconciler, and back around.\n\nThe best compliment I can pay our monitoring is that I have stopped looking at it.\n\nThat sounds like neglect, and for most of my career it would have been. A dashboard is a confession of uncertainty: you build one because you do not know in advance which number will matter, so you put forty of them on a wall and train your eyes to notice when the shape changes. It works, in the sense that a smoke detector made of a person sniffing the air works. It also means the quality of your operations is bounded by how much attention you are willing to spend, and attention is the one resource that does not scale with the fleet.\n\nThe alternative is not fewer signals. It is signals precise enough that a machine can act on them, arranged so that a human is interrupted only when the machine should not act alone.\n\nThis post is the architecture of that arrangement. **Nothing in it depends on which Kubernetes you run.** The examples come from our cluster, but the\nloop, the constraints, and the failure modes are the same whether your control plane is managed by Google, Amazon, Microsoft, or by you. The parts\nthat differ are at the bottom of the post, deliberately, because they are the least interesting part.\n\n## The loop\n\nThe operational path is a ring ([Fig. 1](#figure-1)):\n\n**alerting → chat → agent → pull request → CI → reconciler → cluster**, and the cluster is what the alerting watches.\n\nAn alert fires. It routes into a channel. An agent with live read access to your metrics, your logs, and the Kubernetes API picks it up, investigates,\nand — this is the part that took discipline — **proposes a change instead of making one.**\n\nThe agent reads everything and writes nothing directly. It holds no production credentials for mutation. Every change it wants lands as a pull request\nagainst the repository the reconciler already syncs from, which means review and CI sit in the path *by construction* rather than by policy. There is\nno mode where the agent is trusted enough to skip them, because there is no credential with which it could.\n\nThat constraint is doing more work than it appears. An agent that can edit production is only as safe as its best moment; an agent that can only open\npull requests is as safe as your merge process, which you already trust with everything else. The GitOps repository was not adopted for the agent — it\nwas already the source of truth — but it turned out to be the thing that makes an autonomous operator tractable. The agent inherits a review culture\nit did not have to be taught, and you keep every control you already had. [Fig. 2](#figure-2) is one of those proposals, verbatim.\n\nConcretely, that read plane is two datasources and an API: **VictoriaMetrics** for metrics, **Loki** for logs, and the platform’s own API for object\nstate. Grafana sits on top of the first two, and it is there for me rather than for the agent — the agent queries the datasources directly, because a\ndashboard is a rendering decision, and anything reading renderings inherits every choice a human already made about what to display.\n\nThe log half is what separates an operator from a restart loop. Metrics say that something is wrong and when; they rarely say why. Almost every diagnosis here that ended in a durable fix ended in a log line — the error immediately before a flatline, the retry that never succeeded, the entry proving a counter had been reset by a restart rather than by the thing under investigation. An agent holding only alerts and metrics can report that a component stopped making progress. An agent that can also read that component’s logs from the minute before it stopped, and its peers’ across the same window, can say what stopped it — which is the difference between a pull request that restarts something and a pull request that changes a number.\n\nLoki’s shape helps more than I expected it to. It indexes labels rather than log content, so a query has to name a stream — namespace, pod, container — before it can search inside one. For a human that is a mild annoyance. For an agent it is a guardrail: there is no cheap way to grep the entire fleet, so an investigation stays scoped to whatever alerted.\n\nAny reconciler will do. Any chat surface will do. Either datasource is replaceable too, though having both is not — an operator that can see metrics\nand not logs is limited to noticing. The properties that matter are that the agent’s proposals are **legible** (a diff a human can read), **gated**\n(CI runs before merge), and **reversible** (git history) — and those are properties of the workflow rather than of any product in it.\n\n## Observability for humans\n\nDashboards and alert rules live in version control and are applied by the same reconciler as everything else. Dashboards as code, reviewed like code, arriving through the same pull request path.\n\nThe discipline that matters is not the storage format, though. It is **curation over accumulation**, and it is unnatural to sustain, because every\nindividual panel is easy to justify and no individual panel is ever the problem. Vendor-bundled boards are the pathological case: they render a panel\nper label value, because they cannot know which values you have. A board meant to show notification health draws two panels for every integration the\nalerting system *could* have — most of them permanently empty on any given installation. Nothing is wrong with any panel. The board is still useless,\nbecause the ratio of signal to furniture has crossed the point where a human stops reading it.\n\nWe replaced ours with a handful of panels that filter to the series with actual traffic ([Fig. 3](#figure-3)). The rule we settled on: **a panel earns\nits place by changing somebody’s next action.** Everything else is furniture, and furniture on a dashboard is worse than absence, because absence is\nhonest.\n\nThresholds get the same treatment. Every threshold on our boards and in our alert rules comes from a **measured distribution** — the actual multi-day\nmaximum or percentile of the thing being watched — never a vendor default and never a number that sounded round. This is not fussiness. Grafana’s\nstock stat-panel threshold is red above 80, applied to the raw value regardless of unit, which paints any panel measuring bytes or seconds permanently\nred the moment it exceeds eighty of them. A dashboard that is always red is a dashboard nobody reads, and you have then achieved negative\nobservability: the display actively trains its audience to ignore it.\n\nWhat survives that discipline is small ([Fig. 4](#figure-4)): a shelf of boards, one per concern, and panels that are only there because somebody\nreads them.\n\n## Nothing is trusted until it has failed\n\nThis section is one rule applied twice. A signal you have never seen fire and a check you have never seen fail are the same object: an assertion of optimism. The first half is about the metrics that lie; the second is the drill that makes every check prove it can go red.\n\n### Detection, and the metrics that lie\n\nAlerts split by severity into two channels: one that wakes a human, one the agent watches continuously. That split is the entire reason a human can\nstop watching. It is not that fewer things go wrong; it is that the class of things which merely *need doing* has a competent recipient that is not a\nperson. ([Fig. 8](#figure-8), below, is five minutes of that second channel at work.)\n\nEvery rule carries a runbook URL, so detection and response ship together. An alert that tells you something is broken without telling you what to do about it has externalized its hardest part onto whoever is holding the pager at 3am.\n\nThe harder lesson is about which metrics to trust. **Health flags lie under exactly the conditions you built them for.** A component that has become\ndetached from its peers will frequently continue to report that it has a leader, that its internal indices agree, and that its process is up — every\none of those true in isolation, and the conjunction of them still describing something doing nothing at all. The self-consistency is the trap: the\ncomponent is internally coherent and externally dead.\n\nThe signal that exposes it is comparative rather than local: **progress measured against a peer’s** ([Fig. 5](#figure-5)). When you write detection\nfor anything with consensus or replication in it, assume every self-reported health field is a claim by an unreliable narrator, and find the quantity\nthat requires two parties to agree. Replication lag, applied-index deltas, and consumer-group offsets are all this shape. A single number reported by\nthe sick component about itself is not.\n\n### Verification, and making a check fail on purpose\n\nAutomation is trustworthy in proportion to the honesty of its verification, and verification fails in ways that are almost invisible, because a broken check and a passing check produce identical output: silence, and a green mark.\n\nThree ways I have watched a check quietly lie, none of them exotic:\n\n**Privilege mismatch.** A test that runs with different credentials than the operation it guards is answering a different question. A directory\nexistence test executed as an unprivileged user against a root-only path returns *false* whether or not the directory exists — so a script gated on it\nskips its own work and reports there was nothing to do. The fix is trivial: run the check at the privilege of the action. The failure mode is not,\nbecause it is silent and it reads as success.\n\n**Operator precedence.** In PromQL, comparison binds more tightly than set operations, so a threshold written after a join filters the *join key*\nrather than the value you meant. Both forms parse. Both look correct in review. One of them fires forever.\n\n**Reading the wrong instance.** Log queries return oldest-first by default, so a naive “first match” reads the *earliest* record rather than the\nlatest, and cheerfully reports a value from a previous boot. This one is especially cruel because it stays invisible while the values agree — which is\nexactly until you change something and need the check to notice.\n\nThe unifying property is that none of these could have failed ([Fig. 6](#figure-6)). Each returns the same comfortable answer under every condition,\nincluding the conditions it existed to detect. So the practice we adopted is to **make the check fail on purpose before trusting it**: mutate the\ninput, move the threshold, point it at a known-bad state, and confirm it goes red. A guard never observed refusing is not a guard, it is decoration.\n\nAnd confirm the *effect*, not the action. Having run a command is not evidence the command worked — the two diverge precisely when something else has\ngone wrong, which is the only time it matters.\n\n## Automatic fix, in three layers\n\n“Self-healing” is usually a marketing word. Concretely it has three tiers, in descending order of how much they are worth ([Fig. 7](#figure-7)).\n\n**Configuration that makes the failure survivable.** The highest-value fix removes a class of incident from the pager entirely, and it is almost\nalways a parameter rather than a mechanism. Distributed systems ship defaults tuned for the environments their authors had, and a default that is\ngenerous in one environment can be marginal in yours. The gap between “recovers by itself” and “requires a human to intervene” frequently sits on one\nside or the other of a single number. Finding those numbers is unglamorous: you measure your actual throughput, compute what the default tolerates in\nunits of *your* system’s time, and discover the margin is minutes when you assumed it was hours. Nothing about that work looks like automation. It\neliminates more pages than any automation we have written.\n\n**Reconciliation.** Continuous restoration of declared state converts an entire category of drift into a non-event. This is the layer people usually\nmean by GitOps, and it is genuinely load-bearing, but it only repairs divergence from a spec — it cannot help a component that is unhealthy while\nmatching its spec perfectly. It also only covers what it continuously watches, which is less than the repository contains; the next section is about\nthe rest.\n\n**Gated repair, for everything destructive.** When a fix requires deleting data, the agent does not improvise against production. It generates a\nscript whose every destructive step is preceded by a live proof that the system survives it — not a health flag, but an actual write executed through\na *different* replica, demonstrating that the cluster still accepts work without the component about to be removed. The script refuses to continue if\nthat write fails. It refuses to act on something still making progress. And it will not report success until it has re-read the effect it was supposed\nto produce.\n\nThat last property is the one I would keep if I could keep only one.\n\n[Fig. 8](#figure-8) is the middle tier caught on the record, which is rare, because its successes are non-events. A node dropped out and took its\nmetrics collector with it. The platform restarted the collector the moment the node returned, so the warning that landed in the agent’s channel at\n18:20 resolved beside itself at 18:25 — repaired before anyone could have acted on it. A worse arrangement pages a person at 18:20 so they can watch\nthe system fix itself by 18:25.\n\nTo be precise about who repaired that: Kubernetes’s own reconciliation — a kubelet re-running a container the node’s spec says should exist — not the\nGitOps repository’s. The tier is bigger than the buzzword. The GitOps flavor is quieter still, so to photograph it I had to stage its trigger — the\nsame drill this post applies to every mechanism: make it show its behavior on purpose. I scaled an exporter from one replica to two by hand, exactly\nthe out-of-band edit the loop forbids, against a manifest that pins one. [Fig. 9](#figure-9) is the transcript: at the first poll that saw my edit,\nthe reconciler was already syncing, and by the next poll — three seconds later — the count was back where the repository said it should be. No alert,\nno message, and an extra pod that was gone before it ever became ready. That is the middle tier’s entire working day, compressed to where you can\nwatch it.\n\n## The layer no reconciler watches\n\nA reconciler watches what lives inside the cluster. The DNS records, the repositories these changes travel through, the object storage the backups\nland in, the cloud projects underneath them — that layer is **Terraform**, and Terraform is not a reconciler. It converges when you run it, and\nbetween runs it holds no opinion at all. So drift here is *silent* ([Fig. 10](#figure-10)): somebody edits something in a console, a provider moves a\ndefault, and the repository is fiction while every dashboard stays green — until the next apply arrives, carrying a change nobody wrote alongside the\none they did.\n\nThe treatment is the same as everywhere else in this post: **the absence of drift has to be established rather than assumed.** A plan runs against\nevery root module on a schedule, and a non-empty result raises an alert like any other signal. The absence of any result at all raises another,\nbecause a quietly disabled workflow and a fleet with nothing wrong otherwise produce identical output — silence — and silence reads as health.\n\nThe drift alert itself was first written the obvious way — fire when the metric reports drift — and it could never fire: a result lands every six hours, the metric store only looks back five minutes, so the rule spent the gap between results evaluating against nothing, while looking exactly like coverage. What caught it was forcing it to go red before trusting it — the drill from the verification section above.\n\nEverything else follows the loop unchanged. Changes arrive as pull requests, CI posts the plan back onto the pull request — the diff a human reads is\nthe plan, which says what will happen rather than what you wrote — and apply runs from CI on merge, with credentials the agent does not hold. Two\nguardrails are specific to this layer: `prevent_destroy`\n\non anything stateful, because a plan that replaces a database is the same category of event\nas a script that deletes a data directory, and the state file in a versioned remote store, because losing it leaves the infrastructure standing while\nnothing remains that can describe it.\n\n## The system writes its own reports\n\nOnce a day, a read-only routine produces a posture digest and lands it as a pull request: tier by tier, each claim carrying the metric behind it, with\nCI posting the summary into the incident channel ([Fig. 11](#figure-11)). A human merges it to keep the history, or closes it.\n\nTwo properties make it worth reading rather than skimming.\n\nIt **validates its own pipeline before reporting** ([Fig. 12](#figure-12)). A monitoring report drawn from a dead collector looks identical to one\ndrawn from a healthy system — both say “no problems found.” So the digest first establishes that its data source is live and states that it did so.\nAbsence of evidence is only evidence of absence once you have shown you would have seen it.\n\nAnd it **names what it could not see**. One read-only service account is short a permission for one resource type; rather than silently omitting that\nsection, the digest reports the gap every single day, alongside the proxy signals it used instead. A report that quietly drops the checks it could not\nperform is worse than no report, because it manufactures confidence out of a blind spot.\n\nIncidents become postmortems in the same repository, as runbooks the alerts link to. When later evidence contradicts one — and it does, because early\ndiagnoses are frequently wrong — it gets a **correction banner** rather than a quiet edit. The wrong conclusion stays visible beside the right one.\nThat is not sentimentality about history: the reasoning that produced a confident wrong answer is the most reusable artifact an incident generates,\nand deleting it guarantees a rediscovery.\n\n## The substrate is interchangeable\n\nEverything above is portable. This section is the part that is not, and it is short on purpose.\n\nOur cluster is self-managed on hardware we own, joined by a zero-trust mesh rather than a cloud network, with the only public entry point an outbound tunnel — so there are no inbound ports and no public address to scan. Internal surfaces are reachable only over the mesh, through tag-scoped authorization rather than network membership. Backups leave the same way, off-host to object storage.\n\nIf you run GKE, EKS, or AKS you get most of those properties differently and with less work: private clusters, managed ingress, IAM, service controls.\nThe properties are what matter — **no listening surface, authorization that is not membership, state that survives the cluster** — not the components\nthat provide them ([Fig. 13](#figure-13)).\n\nWhat genuinely changes is the *failure distribution*, and therefore how much repair machinery you need. A managed control plane makes an entire class\nof incident somebody else’s pager. Ours does not, which is why the consensus-member failure below was mine to diagnose. That difference decides how\nmuch of the gated-repair layer you build — not whether the loop, the curation discipline, or the verification rule apply. Those hold anywhere; the\nmanaged case simply gets to skip a chapter.\n\n## One incident, end to end\n\nAbstractions are cheap. Here is one failure that exercised every layer above, including the part where my own tooling lied to me.\n\n**The symptom.** Over two weeks, a replicated stateful component stopped participating five times. The process stayed up, the metrics endpoint kept\nanswering, and the member did nothing. Each time, the fix was a hand rebuild — turning it off and on again, with extra steps and no explanation.\n\n**The detection.** Every health flag was green: the member had a leader, its internal indices agreed, it was up and being scraped. All true, and\ntogether they described a corpse. What finally exposed it was the one signal that requires two parties: **progress compared against the leader’s.**\nSampled thirty seconds apart, the leader advanced about 150 entries; the member advanced zero. That comparison is now the alert — *has a leader,\ncommitting nothing* — exactly the pair a flag-based check reads as healthy.\n\n**The cause.** A default, not a bug. The system keeps 5,000 log entries so a lagging member can catch up incrementally; fall further behind and it\nneeds a full snapshot instead, and snapshot transfers frequently failed on our network. At our measured **543 entries per minute**, 5,000 entries is\n**nine minutes**. Any stall longer than that killed the member, deterministically.\n\n**The fix.** Raising retention to 50,000 moved the cliff from nine minutes to roughly ninety ([Fig. 14](#figure-14)), at a cost of a few hundred\nmegabytes. The same class of event has resolved itself ever since. The best automation we wrote that fortnight was a number — and it is in no\ndocumentation, because the default is only wrong *here*, and only measurement makes that visible.\n\nThe shape outlives the component. On a managed control plane you will never touch this particular system, but every replicated stateful thing you run has the same failure mode — a database replica, a broker, a queue consumer. Internally consistent, externally dead, and reporting healthy is a category, not an incident.\n\n**The repair, gated.** A rebuild, where one is still needed, deletes state on a consensus member — irreversible. So the script proves safety before\neach destructive step: it executes a **real write through a healthy peer** and refuses to continue unless it succeeds. It also refuses to touch a\nmember whose progress is still advancing. That second gate has stopped me twice: I was sure the member was dead, the script declined, and the member\nrecovered on its own a few minutes later.\n\n**The lie.** The first version of that script reported `data directory removed`\n\nwithout removing anything. Its guard was an existence test run as the\nlogin user against a root-only path — which returns *false* whether or not the path is there — and its “verification” reused the same test. Green\ncheckmarks for twenty-five minutes, on a member that still held every byte. The script was not wrong about what to do; it was wrong about whether it\nhad done it, and nothing in its output distinguished the two. That is the verification section in one paragraph.\n\n## What it costs\n\nThe metered services are very nearly free, and I mean that arithmetically rather than rhetorically. All backups together — database base backups and\nwrite-ahead log, control-plane snapshots, secrets, object and vector store dumps — total about 28 GiB, which after the free tier bills at **roughly\nUS$0.30 a month**. The tunnel is free. The mesh is free at our size: twenty-five devices, but only two human accounts, and the free tier covers\nthree. CI runs on our own hardware, so the minutes cost nothing, and the metric and log stores sit on the same machines — retention for both is a disk\ndecision rather than an invoice. Call the cloud bill **US$1 a month**, most of which is rounding.\n\nStated alone that would be dishonest, because one cost sits outside it and exceeds everything above combined. **The operator is not free**: the agent\ndoing this work is a paid subscription and is comfortably the largest single line item. There is a version of this post that omits that to make the\neconomics look sharper. It would be the wrong post — the agent is not an accessory to the stack, it is what makes an unmanaged fleet tractable for two\npeople, and its cost belongs in the total.\n\nWhat none of these figures include is the cluster itself, or the domains that point at it. The machines would be drawing power and the names would\nrenew whether or not any of this ran, so both are the cost of *having* the thing rather than of running it like this, and they sit outside every\nnumber here.\n\nItemized at list price, so the headline number has nowhere to hide:\n\n| Line item | Per month | How it is known |\n|---|---|---|\n| Backups, 28 GiB | ~$0.30 | billed, after the free tier |\n| Mesh | $8 | list price; free at our size, so we pay nothing |\n| Metrics, logs, dashboards | $0 | self-hosted; retention is a disk decision |\n| Tunnel, chat, CI, alerting, reconciler | $0 | free tiers and self-hosted open source |\nInfrastructure tooling | ~$8 | what the description calls under US$10 |\n| The operator, agent subscription | $20–200 | published tiers — the same ladder for Claude Code and Codex |\nAll in | $28–208 | dominated by the last row |\n\nTwo things that table makes obvious ([Fig. 15](#figure-15)). The cheap number is real but partial: US$8 a month buys the tooling and nothing else,\nand our own bill is nearer US$1 because the mesh’s free tier covers a two-person team. And the headline sits well below the total — the agent reading\nthe alerts still bills monthly. Anyone reproducing this should budget for the bottom row, not the top.\n\nOn a managed cluster the arithmetic inverts: the control plane, load balancers, and egress become the dominant line, and electricity disappears into somebody else’s bill. The agent-ops layer costs the same either way, because it is the same layer.\n\n## What it adds up to\n\nNone of these pieces is exotic. An alert manager, a time-series database, a log store, a GitOps reconciler, Terraform, a chat surface, an agent with API access — all available to anyone, most of it free, none of it specific to a distribution.\n\nWhat makes it work as a system is a set of constraints that each look like a limitation:\n\n- The agent\n**cannot** write to production, so every change is reviewed. - Dashboards\n**cannot** bypass code review, so nobody accumulates furniture unnoticed. - Thresholds\n**cannot** be chosen by taste, so alerts mean something when they fire. - A component\n**cannot** vouch for itself, so detection keys on quantities that take two parties to agree. - Checks\n**cannot** be trusted until they have been made to fail. - Destructive repairs\n**cannot** proceed without a live proof of safety. - Infrastructure\n**cannot** change without a plan somebody read, so drift is found rather than discovered. - Reports\n**cannot** omit the checks they failed to run, so silence never reads as health.\n\nEvery one of those is something the system refuses to do. It turns out that is what buys the thing it does: I do not have to watch it. Not because it never breaks — it breaks about as often as anything assembled from software — but because the parts that break announce themselves precisely, the recipient of the announcement is frequently not a person, and when it is a person, the message includes what to do.\n\nA dashboard you have to monitor is an alert you have not written yet.", "url": "https://wpnews.pro/news/the-cluster-you-do-not-watch", "canonical_source": "https://pinglin.tw/blog/the-cluster-you-do-not-watch/", "published_at": "2026-08-14 00:00:00+00:00", "updated_at": "2026-08-19 16:43:31.481458+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-infrastructure", "mlops"], "entities": ["VictoriaMetrics", "Loki", "Grafana", "Kubernetes"], "alternates": {"html": "https://wpnews.pro/news/the-cluster-you-do-not-watch", "markdown": "https://wpnews.pro/news/the-cluster-you-do-not-watch.md", "text": "https://wpnews.pro/news/the-cluster-you-do-not-watch.txt", "jsonld": "https://wpnews.pro/news/the-cluster-you-do-not-watch.jsonld"}}