{"slug": "systemd-watchdogsec-the-kill-switch-for-hung-ai-generated-code", "title": "systemd WatchdogSec: The Kill Switch for Hung AI-Generated Code", "summary": "A developer built a roughly 90-line Go program demonstrating how systemd's WatchdogSec mechanism can detect and kill hung processes, including AI-generated code that stalls on external API calls or runaway loops. The program pings systemd every 2 seconds via SdNotify and, in a simulated hang test, stops pinging and deadlocks, prompting systemd to SIGKILL the process at the 10-second deadline and restart it cleanly. The writeup stresses that Type=notify services must send READY=1 or systemd reports a protocol failure even when the process exits with status 0.", "body_md": "A watchdog is not a health check. A health check *observes*. A watchdog *enforces*.\n\nIn Linux, systemd has a built-in watchdog: you tell it \"if this process doesn't prove it's alive every N seconds, kill it.\" No cron job. No sidecar. No if pgrep script. The kernel-facing supervisor does it.\n\n```\nWITHOUT WATCHDOG                    WITH WATCHDOGProcess:  [alive] [alive] [HUNG...]   Process:  [alive] [alive] [HUNG...]systemd:  (watching) (watching)       systemd:  ping ✓   ping ✓   ping ✗Result:   zombie forever              Result:   SIGKILL at T+10s                                        Restart=always → fresh process\n```\n\nThat’s the whole idea. The process must periodically send a heartbeat. Miss the deadline, systemd assumes it’s deadlocked, wedged on a network call, or stuck in a loop it can’t escape — and terminates it. Restart=always then brings it back clean.\n\nFor AI-generated code, this matters. Generated code can call external APIs, load models, run loops, or spawn concurrent work. A hung generated function can stall forever. A watchdog turns that silent failure into a loud, logged, auto-recovered one.\n\nI built a minimal Go program that pings systemd every 2 seconds, with a flag to simulate a hang. Then I ran it two ways:\n\n```\nTest 1 — Normal:    ping every 2s, exit cleanly after 3s                      Expected: exit 0, result: successTest 2 — Hang:      ping every 2s, stop pinging after 2s, deadlock                      Expected: SIGKILL at ~10s, result: watchdogsystemd (PID 1)  │  ├─ watchdog-test.service   Type=notify  WatchdogSec=10s  │    └─ go-program -ping-interval=2s -normal-exit=3s  │  └─ watchdog-hang.service   Type=notify  WatchdogSec=10s       └─ go-program -ping-interval=2s -hang-after=2s\n```\n\nThe Go program is just a means to demonstrate the utility. The point is the mechanism. Code: [https://github.com/cheikh2shift/go-snippets/tree/main/watchdog-sec](https://github.com/cheikh2shift/go-snippets/tree/main/watchdog-sec)\n\nThe full program is ~90 lines. Four parts matter.\n\n```\nwatchdog, err := daemon.SdWatchdogEnabled(false)if err != nil {    log.Fatalf(\"Failed to check watchdog: %v\", err)}if watchdog == 0 {    log.Fatal(\"Watchdog not enabled — run under systemd with WatchdogSec set\")}log.Printf(\"Watchdog enabled — must ping every %s\", watchdog)\n```\n\nSdWatchdogEnabled reads WATCHDOG_USEC from the environment. systemd only sets that variable for Type=notify services with WatchdogSec= configured. If it's 0, you're not under a watchdog — fail fast instead of pretending.\n\n```\n_, err = daemon.SdNotify(false, daemon.SdNotifyReady)if err != nil {    log.Fatalf(\"SdNotify READY failed: %v\", err)}log.Printf(\"Sent READY=1 to systemd\")\n```\n\nThis is the step everyone misses. With Type=notify, systemd waits for READY=1 to consider the service started. Exit before sending it and systemd reports:\n\n```\nFinished with result: protocolMain processes terminated with: code=exited, status=0/SUCCESS\n```\n\nExit code 0, but systemd-run --wait returns 1. The process succeeded; the *protocol* failed.\n\n```\nctx, cancel := context.WithCancel(context.Background())defer cancel()go func() {    ticker := time.NewTicker(*pingInterval)    defer ticker.Stop()    for {        select {        case <-ctx.Done():            return        case <-ticker.C:            _, err := daemon.SdNotify(false, daemon.SdNotifyWatchdog)            if err != nil {                log.Printf(\"Watchdog ping failed: %v\", err)            } else {                log.Printf(\"Watchdog ping sent\")            }        }    }}()\n```\n\ndaemon.SdNotify(false, daemon.SdNotifyWatchdog) sends WATCHDOG=1 over the NOTIFY_SOCKET datagram socket systemd gave us.\n\n```\nif *hangAfter > 0 {    go func() {        time.Sleep(*hangAfter)        log.Printf(\"SIMULATING HANG — stopping pings, deadlocking\")        cancel()   // stop watchdog pings        select {}  // deadlock forever    }()}\n```\n\ncancel() stops the ping loop. select {} with no cases blocks forever. That's what a real hang looks like from systemd's perspective.\n\n```\n  T+0s    Program starts, sends READY=1  T+0s    Ping loop starts (every 2s)  T+2s    Last ping sent  T+2s    cancel() → ping loop exits  T+2s    select{} → main goroutine deadlocks  T+10s   systemd: no ping for 10s → SIGKILL  T+10s   Result: watchdog\ngo: go-program imports github.com/coreos/go-systemd/v22/sdnotify:module ... found (v22.7.0), but does not contain package .../sdnotify\n```\n\nThere is no sdnotify package. The correct path is github.com/coreos/go-systemd/v22/daemon, with functions daemon.SdNotify, daemon.SdNotifyWatchdog, daemon.SdNotifyReady, and daemon.SdWatchdogEnabled.\n\n```\nUnknown assignment: WatchdogSec=30s\n```\n\nWatchdogSec is a *service* property, not a *scope* property. Use a transient service:\n\n```\nsystemd-run --wait --unit=watchdog-test \\  --property=Type=notify \\  --property=WatchdogSec=10s \\  --property=NotifyAccess=main \\  ./go-program -ping-interval=2s -hang-after=2s\n```\n\nAlways send READY=1 at startup for Type=notify services.\n\nA busy loop in one goroutine doesn’t stop the ping goroutine. Use context cancellation + select {}.\n\nClear them:\n\n```\nsystemctl stop watchdog-test.service watchdog-hang.service 2>/dev/null || truesystemctl reset-failed watchdog-test.service watchdog-hang.service 2>/dev/null || truesystemctl daemon-reload\n```\n\nOr use a unique name per run: --unit=\"watchdog-test-$$\".\n\n```\n=== systemd WatchdogSec — Quick Test ===[2/3] Test 1: Normal run (watchdog active, no hang, exits 0)          Finished with result: successMain processes terminated with: code=exited, status=0/SUCCESS               Service runtime: 3.032s  exit code: 0 (expected 0)  ✓ PASS[3/3] Test 2: Hang simulation (hangs after 2s, watchdog kills at ~10s)          Finished with result: watchdog               Service runtime: 12.140s  exit code: 1 (expected non-zero)  ✓ PASS - watchdog killed hung programNov 05 12:16:31 mainnet systemd[1]: watchdog-hang.service: Failed with result 'watchdog'.\n```\n\nTest 1: clean exit in 3 seconds, result: success. Test 2: pings stop at 2s, systemd kills at ~10s, result: watchdog.\n\nsystemd’s watchdog is the simplest liveness enforcement available to a Linux process. No sidecar, no external monitor, no polling loop. Three lines of setup (Type=notify, WatchdogSec=, SdNotifyReady) and a periodic SdNotifyWatchdog call, and a hung process becomes a logged, auto-restarted one.\n\nThe pitfalls are all in the setup, not the concept: wrong import path, --scope instead of a service, missing READY=1, and a hang simulation that doesn't actually stop the heartbeat. Get those four right and it works exactly as documented.\n\nFor AI-generated code, the watchdog interval is a design decision, not a constant. Model loading needs a longer window than token generation. Set WatchdogSec to 2–3× your longest legitimate blocking operation, and ping from a goroutine that *isn't* blocked by that operation.", "url": "https://wpnews.pro/news/systemd-watchdogsec-the-kill-switch-for-hung-ai-generated-code", "canonical_source": "https://cheikhhseck.medium.com/systemd-watchdogsec-the-kill-switch-for-hung-ai-generated-code-ce221e603e69?source=rss-81ed16b24275------2", "published_at": "2026-11-05 12:59:24+00:00", "updated_at": "2026-11-05 13:28:26.847017+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-agents"], "entities": ["systemd", "Go", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/systemd-watchdogsec-the-kill-switch-for-hung-ai-generated-code", "markdown": "https://wpnews.pro/news/systemd-watchdogsec-the-kill-switch-for-hung-ai-generated-code.md", "text": "https://wpnews.pro/news/systemd-watchdogsec-the-kill-switch-for-hung-ai-generated-code.txt", "jsonld": "https://wpnews.pro/news/systemd-watchdogsec-the-kill-switch-for-hung-ai-generated-code.jsonld"}}