# Scheduled helpers rot long after their job is done

> Source: <https://dev.to/prodbymarcu/scheduled-helpers-rot-long-after-their-job-is-done-3fn0>
> Published: 2026-10-31 22:13:35+00:00

A watchdog fixes the outage and nobody turns it off. Months later that watchdog is a spam fountain or a wrecking ball aimed at a healthy system, and nothing announces the decay because success gets reported right beside it. Five incidents logged on this machine stayed invisible until an audit or a human caught them mid-damage, each one timestamped and counted. What holds against all of them is a receipt plus an actual delete.

An hourly launchd job existed to bring the shared messaging gateway back after idle stretches. On October 28 it met a slower boot than its own check could wait for. Connected at 11:55:56, SIGUSR1 arrived at 11:56:08.958, twelve seconds after readiness both adapters disconnected and the gateway exited 75, then a new startup control socket appeared at 11:56:15.251 while platforms finished ready at 12:00:27. The MCP-heavy start took about 250 seconds and the helper verified replacement inside a 180 second window, so verification failed, exit 1 came back, and launchd relaunches failed submitted jobs. The cadence value had already been saved and activated before any of this started, nobody needed a standing helper by then. The loop wasn't fixing anything, it was re-breaking what it broke first, every hour of that day until I deleted the job at 20:04.

The same shape appears under any supervisor that relaunches failed jobs. Every layer behaved as designed, together they produced an hourly self-inflicted wound on a system that booted fine when nothing poked it.

The price monitor ran as a full agent cron every 15 minutes for a two-call check, burned 5.36M input tokens across 7 days at roughly 39k per fire, went down once token forensics flagged it. A flight watcher kept firing every 30 minutes after its subject resolved, found the `[SILENT]` suppression token mangled to `[ SILENT ]` above 50 times, raw brackets landing in my iMessage chat before someone deleted that job. The nightly backup went quiet for four straight days when a global model change drift-skipped it, `failure_streak: 4`, `drift_alerted: true`, visible only when a cron audit pulled it; the scheduler raises `drift_skip` once then keeps skipping with nothing user-visible. Cron sessions accumulated on every tick even when the agent returned an empty response, one observed pile-up reached 857 entries over 6 weeks.

Four shapes, same disease. Each job did its work or lost the ability to do it, then kept ticking beside a green report. Four of five were caught by audits that ask what a scheduled thing is still doing, the restart loop got caught the hard way, at 20:04, by a human reading gateway logs wondering why a healthy system kept getting stabbed.

The repair that stopped the loop landed ahead of everything else. Prior done or retired receipts return success before any Hermes config or restart subprocess call runs, marked retired with a reason and backed up alongside their state. Tests in an isolated temporary HOME prove both states exit 0 without the Hermes executable or gateway state present, receipts intact either way, calling the live retired helper exits 0 with no restart. Then `launchctl remove` took the one obsolete job while the shared gateway finished its own startup at 12:00:27 and stayed connected that same morning.

A receipt checked first makes relaunching a finished job free. A guard bolted onto the restart path still owns the damage on the first pass, so idempotency wraps the action end to end, ahead of everything else the job runs.

Two of the five failed quietly and silence is where the audit budget belongs. The drift-skipped backup surfaced as `drift_skip` in a last_error field nobody reads until `hermes cron doctor` pulls it, the fix chain running doctor output plus the run logs under the job's output directory. Engine updates retire pinned model slugs and pinned jobs then die with `RuntimeError: HTTP 404: no router for requested model`, around ten straight failures escalating to a single one-time notice before the repin cleared it. The sunrise ramp missed its own fire by 55 minutes because the cron sat at `55 5,6 * * 1-5` while the script gated on hour equality alone, an hour check can't tell the pre-warm tick apart from the intended delivery.

Every one of those had a fix that fits in one line, pin the model and provider in the job spec so a later default change can't break it, fire the cron at the gate time and never minutes before it, repin to a served slug when an update retires one, remove pending one-shot jobs once their subject resolves. The fixes are cheap. Finding them means assuming something scheduled is still doing something, which costs a terminal a few seconds and costs your eyeballs nothing at all.

The token case carries the practice rule. A job that fetches a number and compares it needs no model at all, `no_agent: true` skips SessionDB and the agent import entirely, zero tokens per tick, confirmed in the scheduler source itself. Needs research or judgment stays an agent job, one HTTP call plus a comparison stays a script. The stock monitor ran seven days under the wrong half of that sentence. Same logic killed the session pollution once the hourly sweeper went `no_agent`, two-stage cleanup with prune first and a per-id delete pass catching archived or orphaned stragglers.

Quiet ticks have their own contract, each one emits exactly `[SILENT]` character-for-character, whitespace variants defeat the suppressor's exact match and mail the raw token to your phone. That spacing bug turned the flight watcher into 50 small insults. Briefings omit sections with nothing to report, announcing absence is noise, and if everything is empty one fallback line does. An honest digest prints each item verbatim so a fresh session can re-derive what "these" meant when the reply lands hours later.

Since October 28 the deletes stuck, helper gone with its receipts kept, price monitor deleted, flight watcher removed, backup repinned, sweeper running hourly, delivery going to explicit phone numbers since bare channel resolution reads env vars that go stale. The remove-on-resolution rule fired twice this month, both jobs went down the day their damage was confirmed. The bb-watchdog runs every minute and stays, it has never been the thing past its purpose on this machine.

The claim for readers, scheduled helpers fail decomposable once their subject goes healthy, receipts plus deletion close them, nobody should ever be guessing whether they're still alive. So what does your cron list say fourteen days in? Mine matched nothing it was actually supposed to need until an audit made the mismatch visible, sometimes four days of silence deep. Tell me about your resident job and where it bit you.
