I run a home lab: a couple of always-on machines, a GPU node, some AI jobs that are supposed to chew through work while I'm asleep or at my day job. The first fear anyone has with unattended infrastructure is simple — if it breaks, will I even notice? So you add notifications. You put up a dashboard. You get an alert when a process dies. That's a real and necessary step.
But here's the thing I keep relearning: monitoring can be green while nothing is getting done.
Last season on the blog I wrote up the incidents from actually running this stuff unattended — the silent stops, the healthy-looking numbers that were lying, the resource contention nobody flagged. The one that reframed everything for me: a GPU node sat idle for 16 hours. The process was alive. The network was fine. Every check I had said "OK." The only signal that didn't lie was the one I wasn't watching — the output count wasn't going up.
This post is the design I landed on, and the checklist I now use before I trust any monitoring setup, mine included.
Most home-lab monitoring collapses into one undifferentiated blob of "checks." It's much easier to reason about — and much easier to find the holes — if you split it into three layers and ask what each one can and cannot tell you.
Layer 1 — Liveness. Is the process alive? This tells you it crashed, or it restarted. What it misses: the process is running but the work has stalled.
Layer 2 — Connectivity. Does the port open? Does the API respond? This tells you the thing is reachable from outside. What it misses: it responds, but it isn't doing the job you think it's doing.
Layer 3 — Progress. Did the amount of finished work increase over the last unit of time? This tells you the job is actually moving forward. What it misses: your definition of "output" could be wrong, or the measurement itself could be broken. (That failure mode is real, and I'll come back to it.)
Liveness is the "notice when it dies" layer. It's the cheapest to add and the one most people already have. Connectivity is the next step up — proving the port is open and the API answers, not just that a PID exists. Both of those are worth having. Neither of them answers the question I actually care about at 2am.
Because for AI agent workloads and GPU nodes, the interesting failures aren't crashes. They're waits. Queues that stop draining. Work that blocks on something and never times out. Silent stops where every component is technically healthy and collectively they've achieved nothing.
Measuring "is it alive" does not tell you "is it working." That gap is the whole reason to bring observability — not just monitoring — into a home lab.
Two incidents changed how much I trust a displayed value.
Capacity showed 99% free while real errors were happening. The monitored value said there was plenty of room. Actual operations were failing. The number wasn't wrong when it was written — it was stale, and stale values get rendered as "normal" by default. Nothing in the display distinguishes "I checked this a second ago and it's fine" from "I last checked this hours ago and it was fine then."
The rule I took from that: when a monitored value and a real error contradict each other, believe the real error. And structurally, the fix isn't a better threshold — it's a freshness gate. A metric that hasn't been updated recently should not be allowed to count as healthy.
A 10G link that only delivered about 3G effective. Bandwidth alone looked survivable. What was actually going on showed up somewhere else: the outward-facing port count climbed to roughly 240. That's the pattern worth internalizing — resources usually have a leading indicator that saturates before the headline number does. If you only watch the headline metric, you find out late.
So the honest summary is: monitoring starts lying the moment you install it. The value might be old. The notification might not get delivered. The word "OK" on a dashboard might be actively concealing a stalled job.
I want to be clear that none of this means throwing out your liveness checks or your telemetry pipeline. Both are necessary. The mistake is letting their roles blur together.
If you already have the first two, the realistic move is to add one progress signal on top. Not to rebuild everything. Pick the workload that would hurt most if it silently stalled, define what "one unit of progress" means for it, and alert when that number stops increasing for longer than it should. The same skepticism generalizes. Backups are the classic case: the existence of a backup tells you nothing about whether you can restore. Only an actual restore test does. I found broken restores and prune candidates precisely because I stopped trusting the "backup completed" line and went and checked. Monitoring is not a substitute for operating the system — it's an instrument for finding the holes in how you operate it.
Depends on what you're running. Roughly how I'd advise choosing:
| Setup | Good fit for | Bad fit for | Upside | Downside |
|---|---|---|---|---|
| Liveness only | Anyone with no down-alerts at all today | Anyone depending on unattended jobs finishing | Fast to start | Silent stalls slip through |
| Liveness + connectivity | Services consumed from outside | Anyone who needs to see output increase | Easy to isolate unreachability | Confuses "responds" with "produces" |
| Liveness + connectivity + progress | Unattended AI work, backups, scheduled jobs | Anyone who can't yet define "progress" | You can answer "is it working?" | Needs a per-workload definition of output |
| Add nothing yet | Still deciding what you run and how you'd respond | Anyone already trusting unattended work | Avoids alert noise | The blind spot stays |
If you can't define progress for a workload yet, don't rush to add more metrics. Write down the answer to this instead: what is the moment at which I can say this job succeeded? Is it that a file appeared? That N items completed? That a restore verification passed? The answer is different for every workload, and metrics you add before you can answer it are decoration. Go count what you have right now.
A missing layer isn't a failure. It's just the next thing to build. No liveness alerting? Start there — it's the cheapest reduction in "didn't notice for 16 hours" risk you can buy. Telemetry that you suspect isn't flowing? Verify the pipeline before you build on it. Liveness and connectivity both green but work still stalling? That's exactly where progress monitoring earns its keep.
What I want to be left with — and what I'd want for anyone running a lab like mine — isn't having notifications. It's being able to confirm, at the moment it matters, that things are moving forward.
The full Japanese article, with the incident details, the layer-by-layer breakdown, and the reasoning behind the freshness gate and the connectivity leading indicators, is on the blog: