systemd WatchdogSec: The Kill Switch for Hung AI-Generated Code

A developer built a roughly 90-line Go program demonstrating how systemd's WatchdogSec mechanism can detect and kill hung processes, including AI-generated code that stalls on external API calls or runaway loops. The program pings systemd every 2 seconds via SdNotify and, in a simulated hang test, stops pinging and deadlocks, prompting systemd to SIGKILL the process at the 10-second deadline and restart it cleanly. The writeup stresses that Type=notify services must send READY=1 or systemd reports a protocol failure even when the process exits with status 0.

A watchdog is not a health check. A health check observes . A watchdog enforces . In Linux, systemd has a built-in watchdog: you tell it "if this process doesn't prove it's alive every N seconds, kill it." No cron job. No sidecar. No if pgrep script. The kernel-facing supervisor does it. WITHOUT WATCHDOG WITH WATCHDOGProcess: alive alive HUNG... Process: alive alive HUNG... systemd: watching watching systemd: ping ✓ ping ✓ ping ✗Result: zombie forever Result: SIGKILL at T+10s Restart=always → fresh process That’s the whole idea. The process must periodically send a heartbeat. Miss the deadline, systemd assumes it’s deadlocked, wedged on a network call, or stuck in a loop it can’t escape — and terminates it. Restart=always then brings it back clean. For AI-generated code, this matters. Generated code can call external APIs, load models, run loops, or spawn concurrent work. A hung generated function can stall forever. A watchdog turns that silent failure into a loud, logged, auto-recovered one. I built a minimal Go program that pings systemd every 2 seconds, with a flag to simulate a hang. Then I ran it two ways: Test 1 — Normal: ping every 2s, exit cleanly after 3s Expected: exit 0, result: successTest 2 — Hang: ping every 2s, stop pinging after 2s, deadlock Expected: SIGKILL at ~10s, result: watchdogsystemd PID 1 │ ├─ watchdog-test.service Type=notify WatchdogSec=10s │ └─ go-program -ping-interval=2s -normal-exit=3s │ └─ watchdog-hang.service Type=notify WatchdogSec=10s └─ go-program -ping-interval=2s -hang-after=2s The Go program is just a means to demonstrate the utility. The point is the mechanism. Code: https://github.com/cheikh2shift/go-snippets/tree/main/watchdog-sec https://github.com/cheikh2shift/go-snippets/tree/main/watchdog-sec The full program is ~90 lines. Four parts matter. watchdog, err := daemon.SdWatchdogEnabled false if err = nil { log.Fatalf "Failed to check watchdog: %v", err }if watchdog == 0 { log.Fatal "Watchdog not enabled — run under systemd with WatchdogSec set" }log.Printf "Watchdog enabled — must ping every %s", watchdog SdWatchdogEnabled reads WATCHDOG USEC from the environment. systemd only sets that variable for Type=notify services with WatchdogSec= configured. If it's 0, you're not under a watchdog — fail fast instead of pretending. , err = daemon.SdNotify false, daemon.SdNotifyReady if err = nil { log.Fatalf "SdNotify READY failed: %v", err }log.Printf "Sent READY=1 to systemd" This is the step everyone misses. With Type=notify, systemd waits for READY=1 to consider the service started. Exit before sending it and systemd reports: Finished with result: protocolMain processes terminated with: code=exited, status=0/SUCCESS Exit code 0, but systemd-run --wait returns 1. The process succeeded; the protocol failed. ctx, cancel := context.WithCancel context.Background defer cancel go func { ticker := time.NewTicker pingInterval defer ticker.Stop for { select { case <-ctx.Done : return case <-ticker.C: , err := daemon.SdNotify false, daemon.SdNotifyWatchdog if err = nil { log.Printf "Watchdog ping failed: %v", err } else { log.Printf "Watchdog ping sent" } } }} daemon.SdNotify false, daemon.SdNotifyWatchdog sends WATCHDOG=1 over the NOTIFY SOCKET datagram socket systemd gave us. if hangAfter 0 { go func { time.Sleep hangAfter log.Printf "SIMULATING HANG — stopping pings, deadlocking" cancel // stop watchdog pings select {} // deadlock forever } } cancel stops the ping loop. select {} with no cases blocks forever. That's what a real hang looks like from systemd's perspective. T+0s Program starts, sends READY=1 T+0s Ping loop starts every 2s T+2s Last ping sent T+2s cancel → ping loop exits T+2s select{} → main goroutine deadlocks T+10s systemd: no ping for 10s → SIGKILL T+10s Result: watchdog go: go-program imports github.com/coreos/go-systemd/v22/sdnotify:module ... found v22.7.0 , but does not contain package .../sdnotify There is no sdnotify package. The correct path is github.com/coreos/go-systemd/v22/daemon, with functions daemon.SdNotify, daemon.SdNotifyWatchdog, daemon.SdNotifyReady, and daemon.SdWatchdogEnabled. Unknown assignment: WatchdogSec=30s WatchdogSec is a service property, not a scope property. Use a transient service: systemd-run --wait --unit=watchdog-test \ --property=Type=notify \ --property=WatchdogSec=10s \ --property=NotifyAccess=main \ ./go-program -ping-interval=2s -hang-after=2s Always send READY=1 at startup for Type=notify services. A busy loop in one goroutine doesn’t stop the ping goroutine. Use context cancellation + select {}. Clear them: systemctl stop watchdog-test.service watchdog-hang.service 2 /dev/null || truesystemctl reset-failed watchdog-test.service watchdog-hang.service 2 /dev/null || truesystemctl daemon-reload Or use a unique name per run: --unit="watchdog-test-$$". === systemd WatchdogSec — Quick Test === 2/3 Test 1: Normal run watchdog active, no hang, exits 0 Finished with result: successMain processes terminated with: code=exited, status=0/SUCCESS Service runtime: 3.032s exit code: 0 expected 0 ✓ PASS 3/3 Test 2: Hang simulation hangs after 2s, watchdog kills at ~10s Finished with result: watchdog Service runtime: 12.140s exit code: 1 expected non-zero ✓ PASS - watchdog killed hung programNov 05 12:16:31 mainnet systemd 1 : watchdog-hang.service: Failed with result 'watchdog'. Test 1: clean exit in 3 seconds, result: success. Test 2: pings stop at 2s, systemd kills at ~10s, result: watchdog. systemd’s watchdog is the simplest liveness enforcement available to a Linux process. No sidecar, no external monitor, no polling loop. Three lines of setup Type=notify, WatchdogSec=, SdNotifyReady and a periodic SdNotifyWatchdog call, and a hung process becomes a logged, auto-restarted one. The pitfalls are all in the setup, not the concept: wrong import path, --scope instead of a service, missing READY=1, and a hang simulation that doesn't actually stop the heartbeat. Get those four right and it works exactly as documented. For AI-generated code, the watchdog interval is a design decision, not a constant. Model loading needs a longer window than token generation. Set WatchdogSec to 2–3× your longest legitimate blocking operation, and ping from a goroutine that isn't blocked by that operation.