How I Built a Dead-Man's Switch for My AI Trading Pipeline in Python

A developer built a dead-man's switch in Python to detect when their AI trading pipeline silently stops running. The switch flips the monitoring logic: instead of waiting for failure reports, it expects a success signal from the job, and an external watcher raises an alert if the signal doesn't arrive on time. This approach catches failures that produce no errors, such as a scheduling hiccup that prevents the code from ever executing.

I run a small trading bot. Every morning before the market opens it pulls signals, weighs them, and decides whether to act. It had been humming along for weeks, so I'd mostly stopped watching it. Then one morning I opened the dashboard out of idle curiosity and something was off. No trades. Not "it decided not to trade", just crickets . I scrolled back. No trades the day before either. Or the day before that. Three days. The bot had silently stopped running and I had no idea. Here's the part that stuck with me: there was no error to find. No red text, no stack trace, no alert in my inbox. The overnight process that was supposed to kick everything off had just… not kicked off. A scheduling hiccup on my machine, most likely. The code was fine. The code simply never ran. And every single tool I had for catching problems was completely blind to it. Think about how we normally catch failures. We wrap risky things in try/except . We log errors. We wire up something to shout when an exception gets thrown: try: run morning routine except Exception as e: logging.error "Morning routine failed: %s", e send alert e This is good practice and you should do it. But notice the assumption baked into every line: it assumes the code ran. The try block has to execute for the except to ever fire. The logger has to be reached for it to log. The alert has to be triggered by a process that is, by definition, alive. None of that helps you when the process never starts. In every one of these cases there is no exception, because there is no running code to throw one. There's no log line, because nothing got far enough to write it. Your monitoring is sitting there patiently waiting for a signal that will never come, and interpreting the silence as "all good." That's the trap. Absence of an error is not the same as success. For unattended jobs like cron tasks, scheduled scripts, overnight pipelines, AI agents running on a timer etc. the most dangerous failure is the one that produces no output at all. The fix is to stop waiting for bad news and start expecting good news . A dead-man's switch the name comes from the pedal that stops a train if the driver goes unresponsive flips the logic around. Instead of your code reporting when it fails , it reports when it succeeds and something external watches for that report to arrive. If the expected check-in doesn't show up on time, the watcher raises the alarm. The crucial word is external . The thing doing the watching has to live somewhere your job doesn't. If the watchdog runs on the same machine as the job, then when that machine dies, the watchdog dies with it and a dead watchdog can't tell you anything is wrong. You need something running elsewhere whose entire purpose is to notice an absence. Concretely, three pieces: The silence becomes the signal. You just need something that's actually listening for it. Let me show the shape of it. This is deliberately minimal. The point is the pattern, not a framework. The job needs to announce itself at the start and confirm at the end. Pure standard library, no new dependencies: python import urllib.request PING = "https://your-watchdog.example/ping/<your-token " def ping event : try: urllib.request.urlopen f"{PING}/{event}", timeout=10 except Exception: Monitoring must never crash the job it monitors. pass ping "start" try: run morning routine ping "success" except Exception as e: ping "fail" raise Three things worth calling out: start and success bracket the work, so the watcher can tell "never started" apart from "started but never finished" a hang apart from "finished cleanly." curl : curl -fsS https://your-watchdog.example/ping/<token /start ./run backup.sh && curl -fsS https://your-watchdog.example/ping/<token /success \ || curl -fsS https://your-watchdog.example/ping/<token /fail On the server, receiving a ping is just recording that it happened and when: python from datetime import datetime, timezone from flask import Flask app = Flask name @app.route "/ping/<token /<event " def receive ping token, event : monitor = Monitor.query.filter by token=token .first or 404 monitor.last event = event monitor.last ping at = datetime.now timezone.utc if event in "success", "fail" : monitor.last finished at = monitor.last ping at db.session.commit return "", 200 Nothing clever here, and that's the point. The receiver's only job is to remember the last time it heard from you. The interesting logic is the part that runs on its own schedule , independent of any job, and asks: is anyone overdue? python from datetime import datetime, timezone, timedelta def check all : now = datetime.now timezone.utc for monitor in Monitor.query.all : deadline = monitor.last ping at + timedelta minutes=monitor.period minutes + monitor.grace minutes is late = now deadline new status = "DOWN" if is late else "UP" Only act on a change of state. if new status = monitor.status: monitor.status = new status db.session.commit if new status == "DOWN": alert monitor, "Your job hasn't checked in — it may have stopped." else: alert monitor, "Your job is reporting again — recovered." Two design decisions here earned their keep, and both are about not being annoying : 1. State-transition alerting. You alert on the change , not on the condition. A job that's been down for six hours should generate exactly one "it's down" email, not one every time the watchdog loops. And exactly one "it's back" email when it recovers. Alert on the edge, not the level. This is the single biggest thing standing between "useful monitoring" and "inbox noise you'll mute within a day." 2. A grace window. Jobs are never perfectly punctual; a run that usually takes ten minutes occasionally takes twenty. The grace minutes buffer stops those normal wobbles from firing false alarms. A decent rule of thumb is to set the grace to roughly 20–30% of the expected run time, with a sensible floor of a few minutes. Run check all on its own timer - its own cron entry, a scheduled worker, whatever just somewhere separate from the jobs it's watching. That separation is the whole game. The watchdog has to outlive the thing it watches. I built the version above for exactly one job: my trading bot. It worked. The next morning the bot failed to start again, and this time my phone buzzed instead of me finding out three days later by accident. But once I had it, I started pointing it at other things almost reflexively. A nightly database backup which, it turned out, had quietly failed a week earlier and I'd never noticed. A scraper on a schedule. A content pipeline. Every unattended job I owned had the same blind spot, and I'd just been getting lucky that none of them had bitten me yet. That's when it stopped feeling like a personal script and started feeling like something other solo devs probably needed too. Especially the growing pile of us running AI agents and LLM pipelines on timers, where "did the agent actually run this morning?" is a real and slightly unnerving question. So I cleaned it up, put a proper dashboard on it, and turned it into a small product called PulseWatch https://pulsewatch.ai . The whole idea is what's above: your job sends a start and a finish ping, a server watches for them, and you get one email when something goes quiet and one when it comes back. There's a free tier with no card required if you want to point it at a job and see it work. Even if you never use a tool for this, even if you go build the twenty lines yourself, internalise the core idea, because it changes how you think about reliability: Your monitoring is only watching for the failures it can see. The failure that produces no output is invisible to everything that waits for output. Go look at your most important unattended job right now and ask one question: if this silently stopped running tonight, what would tell me? If the honest answer is "I'd notice eventually, when something downstream breaks" then that's the gap. And the fix is genuinely about twenty lines of code away.