I run a small trading bot. Every morning before the market opens it pulls signals, weighs them, and decides whether to act. It had been humming along for weeks, so I'd mostly stopped watching it.
Then one morning I opened the dashboard out of idle curiosity and something was off. No trades. Not "it decided not to trade", just crickets. I scrolled back. No trades the day before either. Or the day before that.
Three days. The bot had silently stopped running and I had no idea.
Here's the part that stuck with me: there was no error to find. No red text, no stack trace, no alert in my inbox. The overnight process that was supposed to kick everything off had just… not kicked off. A scheduling hiccup on my machine, most likely. The code was fine. The code simply never ran.
And every single tool I had for catching problems was completely blind to it.
Think about how we normally catch failures. We wrap risky things in try/except
. We log errors. We wire up something to shout when an exception gets thrown:
try:
run_morning_routine()
except Exception as e:
logging.error("Morning routine failed: %s", e)
send_alert(e)
This is good practice and you should do it. But notice the assumption baked into every line: it assumes the code ran. The try
block has to execute for the except
to ever fire. The logger has to be reached for it to log. The alert has to be triggered by a process that is, by definition, alive.
None of that helps you when the process never starts.
In every one of these cases there is no exception, because there is no running code to throw one. There's no log line, because nothing got far enough to write it. Your monitoring is sitting there patiently waiting for a signal that will never come, and interpreting the silence as "all good."
That's the trap. Absence of an error is not the same as success. For unattended jobs like cron tasks, scheduled scripts, overnight pipelines, AI agents running on a timer etc. the most dangerous failure is the one that produces no output at all.
The fix is to stop waiting for bad news and start expecting good news.
A dead-man's switch (the name comes from the pedal that stops a train if the driver goes unresponsive) flips the logic around. Instead of your code reporting when it fails, it reports when it succeeds and something external watches for that report to arrive. If the expected check-in doesn't show up on time, the watcher raises the alarm.
The crucial word is external. The thing doing the watching has to live somewhere your job doesn't. If the watchdog runs on the same machine as the job, then when that machine dies, the watchdog dies with it and a dead watchdog can't tell you anything is wrong. You need something running elsewhere whose entire purpose is to notice an absence.
Concretely, three pieces:
The silence becomes the signal. You just need something that's actually listening for it.
Let me show the shape of it. This is deliberately minimal. The point is the pattern, not a framework.
The job needs to announce itself at the start and confirm at the end. Pure standard library, no new dependencies:
import urllib.request
PING = "https://your-watchdog.example/ping/<your-token>"
def ping(event):
try:
urllib.request.urlopen(f"{PING}/{event}", timeout=10)
except Exception:
pass
ping("start")
try:
run_morning_routine()
ping("success")
except Exception as e:
ping("fail")
raise
Three things worth calling out:
start
and success
bracket the work, so the watcher can tell "never started" apart from "started but never finished" (a hang) apart from "finished cleanly."curl
:
curl -fsS https://your-watchdog.example/ping/<token>/start
./run_backup.sh && curl -fsS https://your-watchdog.example/ping/<token>/success \
|| curl -fsS https://your-watchdog.example/ping/<token>/fail
On the server, receiving a ping is just recording that it happened and when:
from datetime import datetime, timezone
from flask import Flask
app = Flask(__name__)
@app.route("/ping/<token>/<event>")
def receive_ping(token, event):
monitor = Monitor.query.filter_by(token=token).first_or_404()
monitor.last_event = event
monitor.last_ping_at = datetime.now(timezone.utc)
if event in ("success", "fail"):
monitor.last_finished_at = monitor.last_ping_at
db.session.commit()
return "", 200
Nothing clever here, and that's the point. The receiver's only job is to remember the last time it heard from you.
The interesting logic is the part that runs on its own schedule, independent of any job, and asks: is anyone overdue?
from datetime import datetime, timezone, timedelta
def check_all():
now = datetime.now(timezone.utc)
for monitor in Monitor.query.all():
deadline = monitor.last_ping_at + timedelta(
minutes=monitor.period_minutes + monitor.grace_minutes
)
is_late = now > deadline
new_status = "DOWN" if is_late else "UP"
if new_status != monitor.status:
monitor.status = new_status
db.session.commit()
if new_status == "DOWN":
alert(monitor, "Your job hasn't checked in — it may have stopped.")
else:
alert(monitor, "Your job is reporting again — recovered.")
Two design decisions here earned their keep, and both are about not being annoying:
1. State-transition alerting. You alert on the change, not on the condition. A job that's been down for six hours should generate exactly one "it's down" email, not one every time the watchdog loops. And exactly one "it's back" email when it recovers. Alert on the edge, not the level. This is the single biggest thing standing between "useful monitoring" and "inbox noise you'll mute within a day."
2. A grace window. Jobs are never perfectly punctual; a run that usually takes ten minutes occasionally takes twenty. The grace_minutes
buffer stops those normal wobbles from firing false alarms. A decent rule of thumb is to set the grace to roughly 20–30% of the expected run time, with a sensible floor of a few minutes.
Run check_all()
on its own timer - its own cron entry, a scheduled worker, whatever just somewhere separate from the jobs it's watching. That separation is the whole game. The watchdog has to outlive the thing it watches.
I built the version above for exactly one job: my trading bot. It worked. The next morning the bot failed to start again, and this time my phone buzzed instead of me finding out three days later by accident.
But once I had it, I started pointing it at other things almost reflexively. A nightly database backup which, it turned out, had quietly failed a week earlier and I'd never noticed. A scraper on a schedule. A content pipeline. Every unattended job I owned had the same blind spot, and I'd just been getting lucky that none of them had bitten me yet.
That's when it stopped feeling like a personal script and started feeling like something other solo devs probably needed too. Especially the growing pile of us running AI agents and LLM pipelines on timers, where "did the agent actually run this morning?" is a real and slightly unnerving question. So I cleaned it up, put a proper dashboard on it, and turned it into a small product called PulseWatch.
The whole idea is what's above: your job sends a start and a finish ping, a server watches for them, and you get one email when something goes quiet and one when it comes back. There's a free tier with no card required if you want to point it at a job and see it work.
Even if you never use a tool for this, even if you go build the twenty lines yourself, internalise the core idea, because it changes how you think about reliability:
Your monitoring is only watching for the failures it can see. The failure that produces no output is invisible to everything that waits for output.
Go look at your most important unattended job right now and ask one question: if this silently stopped running tonight, what would tell me?
If the honest answer is "I'd notice eventually, when something downstream breaks" then that's the gap. And the fix is genuinely about twenty lines of code away.