A watchdog is not a health check. A health check observes. A watchdog enforces.
In Linux, systemd has a built-in watchdog: you tell it "if this process doesn't prove it's alive every N seconds, kill it." No cron job. No sidecar. No if pgrep script. The kernel-facing supervisor does it.
WITHOUT WATCHDOG WITH WATCHDOGProcess: [alive] [alive] [HUNG...] Process: [alive] [alive] [HUNG...]systemd: (watching) (watching) systemd: ping ✓ ping ✓ ping ✗Result: zombie forever Result: SIGKILL at T+10s Restart=always → fresh process
That’s the whole idea. The process must periodically send a heartbeat. Miss the deadline, systemd assumes it’s deadlocked, wedged on a network call, or stuck in a loop it can’t escape — and terminates it. Restart=always then brings it back clean.
For AI-generated code, this matters. Generated code can call external APIs, load models, run loops, or spawn concurrent work. A hung generated function can stall forever. A watchdog turns that silent failure into a loud, logged, auto-recovered one.
I built a minimal Go program that pings systemd every 2 seconds, with a flag to simulate a hang. Then I ran it two ways:
Test 1 — Normal: ping every 2s, exit cleanly after 3s Expected: exit 0, result: successTest 2 — Hang: ping every 2s, stop pinging after 2s, deadlock Expected: SIGKILL at ~10s, result: watchdogsystemd (PID 1) │ ├─ watchdog-test.service Type=notify WatchdogSec=10s │ └─ go-program -ping-interval=2s -normal-exit=3s │ └─ watchdog-hang.service Type=notify WatchdogSec=10s └─ go-program -ping-interval=2s -hang-after=2s
The Go program is just a means to demonstrate the utility. The point is the mechanism. Code: https://github.com/cheikh2shift/go-snippets/tree/main/watchdog-sec
The full program is ~90 lines. Four parts matter.
watchdog, err := daemon.SdWatchdogEnabled(false)if err != nil { log.Fatalf("Failed to check watchdog: %v", err)}if watchdog == 0 { log.Fatal("Watchdog not enabled — run under systemd with WatchdogSec set")}log.Printf("Watchdog enabled — must ping every %s", watchdog)
SdWatchdogEnabled reads WATCHDOG_USEC from the environment. systemd only sets that variable for Type=notify services with WatchdogSec= configured. If it's 0, you're not under a watchdog — fail fast instead of pretending.
_, err = daemon.SdNotify(false, daemon.SdNotifyReady)if err != nil { log.Fatalf("SdNotify READY failed: %v", err)}log.Printf("Sent READY=1 to systemd")
This is the step everyone misses. With Type=notify, systemd waits for READY=1 to consider the service started. Exit before sending it and systemd reports:
Finished with result: protocolMain processes terminated with: code=exited, status=0/SUCCESS
Exit code 0, but systemd-run --wait returns 1. The process succeeded; the protocol failed.
ctx, cancel := context.WithCancel(context.Background())defer cancel()go func() { ticker := time.NewTicker(*pingInterval) defer ticker.Stop() for { select { case <-ctx.Done(): return case <-ticker.C: _, err := daemon.SdNotify(false, daemon.SdNotifyWatchdog) if err != nil { log.Printf("Watchdog ping failed: %v", err) } else { log.Printf("Watchdog ping sent") } } }}()
daemon.SdNotify(false, daemon.SdNotifyWatchdog) sends WATCHDOG=1 over the NOTIFY_SOCKET datagram socket systemd gave us.
if *hangAfter > 0 { go func() { time.Sleep(*hangAfter) log.Printf("SIMULATING HANG — stopping pings, deadlocking") cancel() // stop watchdog pings select {} // deadlock forever }()}
cancel() stops the ping loop. select {} with no cases blocks forever. That's what a real hang looks like from systemd's perspective.
T+0s Program starts, sends READY=1 T+0s Ping loop starts (every 2s) T+2s Last ping sent T+2s cancel() → ping loop exits T+2s select{} → main goroutine deadlocks T+10s systemd: no ping for 10s → SIGKILL T+10s Result: watchdog
go: go-program imports github.com/coreos/go-systemd/v22/sdnotify:module ... found (v22.7.0), but does not contain package .../sdnotify
There is no sdnotify package. The correct path is github.com/coreos/go-systemd/v22/daemon, with functions daemon.SdNotify, daemon.SdNotifyWatchdog, daemon.SdNotifyReady, and daemon.SdWatchdogEnabled.
Unknown assignment: WatchdogSec=30s
WatchdogSec is a service property, not a scope property. Use a transient service:
systemd-run --wait --unit=watchdog-test \ --property=Type=notify \ --property=WatchdogSec=10s \ --property=NotifyAccess=main \ ./go-program -ping-interval=2s -hang-after=2s
Always send READY=1 at startup for Type=notify services.
A busy loop in one goroutine doesn’t stop the ping goroutine. Use context cancellation + select {}.
Clear them:
systemctl stop watchdog-test.service watchdog-hang.service 2>/dev/null || truesystemctl reset-failed watchdog-test.service watchdog-hang.service 2>/dev/null || truesystemctl daemon-reload
Or use a unique name per run: --unit="watchdog-test-$$".
=== systemd WatchdogSec — Quick Test ===[2/3] Test 1: Normal run (watchdog active, no hang, exits 0) Finished with result: successMain processes terminated with: code=exited, status=0/SUCCESS Service runtime: 3.032s exit code: 0 (expected 0) ✓ PASS[3/3] Test 2: Hang simulation (hangs after 2s, watchdog kills at ~10s) Finished with result: watchdog Service runtime: 12.140s exit code: 1 (expected non-zero) ✓ PASS - watchdog killed hung programNov 05 12:16:31 mainnet systemd[1]: watchdog-hang.service: Failed with result 'watchdog'.
Test 1: clean exit in 3 seconds, result: success. Test 2: pings stop at 2s, systemd kills at ~10s, result: watchdog.
systemd’s watchdog is the simplest liveness enforcement available to a Linux process. No sidecar, no external monitor, no polling loop. Three lines of setup (Type=notify, WatchdogSec=, SdNotifyReady) and a periodic SdNotifyWatchdog call, and a hung process becomes a logged, auto-restarted one.
The pitfalls are all in the setup, not the concept: wrong import path, --scope instead of a service, missing READY=1, and a hang simulation that doesn't actually stop the heartbeat. Get those four right and it works exactly as documented.
For AI-generated code, the watchdog interval is a design decision, not a constant. Model needs a longer window than token generation. Set WatchdogSec to 2–3× your longest legitimate blocking operation, and ping from a goroutine that isn't blocked by that operation.