This week the open source trend radar showed hermes-agent picking up more than eight thousand GitHub stars in seven days. A self-hosted, customizable AI assistant. The kind of thing you clone on a Sunday, wire into your own tools, and leave running on a small server because it is useful.
I am happy about that number. I also know what most of those installs will look like in three weeks. A process on a box that nobody watches, doing real work, and failing in a way that nobody notices until a customer or a cron job does.
I run 84 containers on two servers as a one-person company. My agents write code, migrate databases and answer support tickets at night. The guard layer that stops them from doing something stupid gets most of the attention when I write about this setup. But guards only cover one failure class. They stop an agent from causing damage. They do nothing about the damage that arises on its own, at 3 a.m., when a container exits, a connection pool is exhausted or a load spike freezes the machine.
This article is about the second class. What happens after the thing you self-hosted breaks while you sleep.
At 00:30 my monitoring loop logged a load average of 33.59 on a machine that normally sits around four. One of the 84 containers had gone into a tight loop. The loop escalated the finding as red, the repair routine for that pattern ran, and at the 01:00 scan every category was green again. I read about it at breakfast. The entry sits in my operations journal for today, timestamped, with the load numbers and the action taken.
Nothing about this is impressive. That is the point. A load spike at half past midnight used to mean a dead app in the morning and a customer email before my first coffee. Now it means a log line.
The difference is three scripts. Together they are just over a thousand lines of bash, and every line exists because of a real incident.
The obvious way to build self-healing is to write a restart loop. Something fails, restart it. Still failing, restart again. That is also the obvious way to make a bad night worse. A restart loop that never succeeds burns CPU, floods the log, and hides the real problem under noise. I have watched a well-meant loop restart the same container forty times while the actual cause was a full disk.
So the first rule is a hard limit on retries. Every repair attempt gets a small budget. When the budget is spent, the script stops, marks the incident open, and wakes a human. Not after the hundredth try. After the third.
The second rule follows from the first. An emergency fix is always explicitly temporary. It buys time. It does not replace the real fix. If a script starts a bridge container from a locally cached image because the deployed one vanished, that bridge is a bandage, and the incident stays open until a proper deployment replaces it. Treating an emergency container as a permanent solution is exactly the shortcut that turns one bad night into a recurring one.
With those two rules in place, the scripts become small and boring, which is what you want from something that runs unattended.
live-app-watchdog.sh is 289 lines and does one thing. It checks whether every customer-facing application that should have a running container actually has one. It compares container state, domain health and declared target state, every few minutes, around the clock.
It is allowed to start an emergency container only under narrow conditions. No healthy backend is serving the domain. No deployment or maintenance window is active. The locally retained image has been verified by digest and configuration. An exclusive lock prevents two watchdog runs from starting two bridges at once.
That last condition took me an incident to learn. Two overlapping runs, each convinced the app was down, each starting a container, and a proxy that suddenly had two backends with different code. The lock is one line. It is the most important line in the file.
post-deploy-repair-loop.sh is the tightest of the three at 187 lines, because it fires immediately after every deployment, and a bad pattern here would spread across thirteen apps before lunch.
The loop is four steps. An HTTP check, does the app answer correctly. A visual check, does the rendered page look right, not just return 200. A fix attempt from a short list of known remediations. Then the checks again. Three iterations at most.
attempt=0
max_attempts=3
while [ "$attempt" -lt "$max_attempts" ]; do
if http_check_ok && visual_check_ok; then
log_success "Deploy verified healthy after $attempt repair attempt(s)."
exit 0
fi
attempt=$((attempt + 1))
log_info "Health check failed. Repair attempt $attempt/$max_attempts."
run_known_remediation
done
escalate_to_human "Deploy did not recover after $max_attempts attempts."
exit 1
The phrase "known remediation" is doing real work. It does not mean "let the agent improvise". It means a small, preapproved vocabulary. Restart the container. Restart the proxy so it picks up the new container IP. Clear a specific cache. Rerun one migration step. Each action has been verified as safe to run automatically. If a deploy fails for a reason none of them addresses, the loop fails fast and hands the problem over. Nobody should get creative under pressure at 2 a.m., and that includes scripts.
smart-auto-heal.sh is the big one at 712 lines, and its size embarrasses me a little until I remember why. "Check health and repair it" is not one problem. It is dozens of small, different problems, each with its own signature and its own correct answer. A stuck queue needs a different fix than an exhausted connection pool, which needs a different fix than a container with a non-zero exit code, which needs a different fix than a search index that has not been rebuilt in seventeen hours.
The script accumulates one fix per pattern it has seen before. Its length is a protocol of real incidents, not a design decision made in one afternoon. Last night's load spike matched a pattern that was added in July after a very similar night.
Not every "restart" is a repair. On July 29 an MCP server process was killed mid-session to force a restart after a code change. The instinct was normal. This process has to restart, so kill it and it comes back. That is correct for most long-running services. It was wrong here.
MCP servers talk to the session through standard input and output pipes. Kill the server and the pipe is gone. There is no load balancer, no reconnect, no retry. Every tool that server provided stayed dead for the rest of the session. The fix was not technical. It was a rule. Restart this class of process only through its proper reconnect mechanism or a new session, never by killing it. Before any repair action, the script must know which process class it is looking at, because for one class "kill and restart" is restoration and for another it is destruction.
That rule now lives in the remediation vocabulary. It is one of the reasons the vocabulary is a list and not a prompt.
A script that can restart containers, clear caches and start bridge images has destructive potential by definition. Agent hooks do not automatically apply inside an independent cron job. So the repair scripts carry their own narrow allowlist, locking, bounded attempts, state checks and immutable logs. Where the central policy layer can be placed in front of them, it is. Where it cannot, the script gets a smaller, explicitly approved operating space.
And there is a firm line around what they never touch. No customer data. No schema changes. No message to a real user. The remediation vocabulary is deliberately infrastructural and reversible. That line is what lets me sleep while they run.
The retry limit counts for little if "escalate to a human" means a log line that nobody reads until morning. In practice it means a push notification on my phone within seconds of the budget being spent, carrying enough context to act. Which app, which check failed, how many attempts, which fix was tried last. Most nights the phone stays quiet. The nights it does not, I know exactly where to look.
Across the last months my operations cockpit counts 1,354 tasks completed without a human touching them, out of 1,873 attempted, a 93 percent rate. The remaining seven percent are the ones that escalated, and every one of them arrived with context instead of a vague "something is wrong".
If you just starred a self-hosted agent and are about to leave it running, do not start with a 712-line repair loop. Start with the simplest possible version of the watchdog. A cron job that checks whether your one most important process is running and, if not, restarts it once, logs the result, and stops. No endless retry. No silent failure. Write the retry limit into the very first version, because raising a limit later is easy and finding out the hard way why you needed one is not.
That single script, running every five minutes against your most critical service, is the seed everything else in this article grew from. Add the next fix the next time something breaks, and only then. In a year you will have your own thousand lines, and every one will have a story.
The good news is that the stories get shorter. Mine used to be "the app was down all night". Last night's was one line, with a timestamp, and I was asleep for it.
This is an adapted chapter from my book "Runs Without Me. You Can Too.", the story of how one founder runs thirteen applications with AI agents, including the guard layer, the self-healing scripts above, and a thirty-day blueprint to build your own. It is on Amazon as Kindle and paperback. A German edition, "Läuft ohne mich. Du kannst das auch.", is available on Amazon.de.