{"slug": "my-repair-loop-restarted-the-same-wrong-thing-16-times", "title": "My repair loop restarted the same wrong thing 16 times", "summary": "A developer traced 16+ failed restarts of a primary inference lane to an orphaned Maya engine process (PID 130993) holding 23,516 MiB on GPU0 and 22,860 MiB on GPU1, launched from a systemd session scope with no service unit or autostart entry, which sat outside the watchdog's visibility. A single SIGTERM to the Maya parent released both GPUs and restored production, after which the developer patched run_server() to refuse stray launches without a MAINTENANCE marker (exit 75) and added a 180-second grace window to the watchdog to prevent repairs from interrupting cold model loads.", "body_md": "A repair loop bets that the cause lives inside the thing it restarts. On 2026-10-21 at 20:07 that bet had cost 16+ restarts on my primary inference lane, both backends sat with no listener, their llama-swap routers were still up, and the cause stood nowhere in the loop's field of view. A process held 23,516 MiB on one GPU and 22,860 MiB on the other, and a backend start that can't allocate memory doesn't finish, no matter how many times the loop asks.\n\nThe watchdog here polls the backends, counts two failed observations, then repairs, with a 240 second cooldown per lane so a cold model load can't chase itself into a second repair. Good idea, useless here, because every restart failed in the same place: startup died in Vision._start_reader, the GPU allocation could not complete, and around it went again. Restart one and restart sixteen carried the same promise, this time the memory will be there, and the process on both cards gave the same answer every time.\n\nThat's the shape of churn that fools you. Each individual repair looks reasonable in its log line, the symptom it aims at is genuine, the listener really is gone, and the count only reads as churn when you step back and see that nothing moved. The loop's inventory of causes was one item long, the lane is down, so sixteen identical repairs is what that inventory produces. Sixteen restarts bought nothing. Which does your loop budget for, the restart it knows or the cause it can't see?\n\nnvidia-smi answered it. A Maya engine, PID 130993, held 23,516 MiB on GPU0 and 22,860 MiB on GPU1. Its parent, PID 130988, served the project-maya config on :8095, and above that sat a launcher, PID 129105, running maya.py inside a systemd session scope, with no service unit behind it. No persistent Maya unit, no user unit, no autostart entry anywhere on the box. I dug for what had launched it and why, and the record doesn't establish what did.\n\nThe nastiest part of that read is what it implies about the loop. The watchdog can see units, health endpoints, listeners. A bare process inside a session scope sits outside all three, so the loop repaired the only world it could see, the lane that wouldn't start, and the squatter stayed in a world the loop had no eyes on.\n\nStopping the churn came before anything else, so the watchdog timer and both production launch chains went down first. Then one SIGTERM to the Maya parent. Its parent-pipe supervision shut down the child engine and released both cards, all three Maya PIDs exited, and the project, weights, packs, logs and configuration stayed on disk untouched. Production came back up in its own order, strata, then llama-swap, then llama-swap-fe, and both backend /health reports retained 524288 context. One signal to the process nobody owns beat sixteen restarts of the processes I own.\n\nA clean recovery doesn't stop the next stray launch, and this launcher had no unit and no autostart to disable, there was nothing to switch off. I backed up the project's maya.py and patched run_server() instead: on this host its normal launcher now refuses to start unless an ops marker named MAINTENANCE exists. The refusal returns exit 75 with a clear message, before any child startup, so a stray Maya launch costs one line and leaves both GPUs alone. The guard covers the inspected launch paths, maya.py and run-maya-ds4.sh, direct engine binaries sit outside it, and a future upstream replacement could remove the guard entirely.\n\nI extracted run_server into a regression test with a subprocess sentinel before I trusted any of it: run without the marker, the sentinel spawns nothing, and with it the engine starts exactly as before. That branch is the one a review never runs, so it got the test first.\n\nStopping the churn exposed a second defect, this one in the watchdog. Its failure state could fire another repair immediately after a manual cold start, straight into model loading, interrupting it mid-load. The fix is a 180 second grace window, computed from systemd's ActiveEnterTimestampMonotonic and the live MainPID, layered on top of the existing 240 second per-lane repair cooldown. Both branches got tested, a recent startup suppresses repair and an old unhealthy service stays eligible. I reenabled the watchdog only after both backends were loaded, and the counters read zero everywhere, failure counts reset to 0, production NRestarts reset to 0.\n\nEvery check ran against the actual Hermes AIAgent on each lane, explicit model and endpoint, no fallback configured. Custom provider names normalize to provider=custom inside Hermes, so the endpoint and model identity in each result establishes which lane answered. Both engines retained 524288 total tokens and no cloud or Ollama fallback got introduced anywhere. One repair I deliberately didn't make: the long desktop turn that failed during the outage stays retained on disk, and it doesn't get replayed automatically, because a turn like that can contain external actions and replaying it is how an outage gets a second life.\n\nThe loop itself is still the same bet, that the cause lives inside what it restarts. The guard and the grace window don't change that bet, they shrink the set of causes the loop can meet by surprise, one at launch time and one at cold-start time.\n\nWhat's the highest restart count your repair loop has rung up on a cause it never looked at? Drop the number and what was actually holding the resource. I want the ugly one in the comments.", "url": "https://wpnews.pro/news/my-repair-loop-restarted-the-same-wrong-thing-16-times", "canonical_source": "https://dev.to/prodbymarcu/my-repair-loop-restarted-the-same-wrong-thing-16-times-39fh", "published_at": "2026-10-24 22:30:16+00:00", "updated_at": "2026-10-24 22:41:17.532533+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "ai-tools"], "entities": ["Maya", "llama-swap", "NVIDIA", "systemd"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/my-repair-loop-restarted-the-same-wrong-thing-16-times", "markdown": "https://wpnews.pro/news/my-repair-loop-restarted-the-same-wrong-thing-16-times.md", "text": "https://wpnews.pro/news/my-repair-loop-restarted-the-same-wrong-thing-16-times.txt", "jsonld": "https://wpnews.pro/news/my-repair-loop-restarted-the-same-wrong-thing-16-times.jsonld"}}