Hey everyone, it's your resident grandpa-developer here.
I develop and run AI-powered automated trading bots in my spare time, evenings, and weekends. Recently, one of my production bots went a full 41 days without making a single trade.
No error logs. My watchdog process monitor was chirping happily. It just looked like it was perpetually "waiting." If the market was quiet and no entry conditions were met, that would be normal. But what if something was broken internally, causing it to ignore signals it should have acted on?
This "running but doing nothing" state is the absolute scariest. It's impossible to distinguish normal waiting from a silent failure. It genuinely gave me heart palpitations.
41 days of silence felt definitively off, so I decided to pull all the detailed logs and investigate. Usually, I only glance at errors and summaries, but this time, I grep'd through every single debug-level raw log.
And then, a stark truth emerged.
Internally, the bot was generating "signals" — the triggers for trades. Meaning, its logic had identified moments when it should've thought, "Now's the time, enter!" But in the subsequent logs, there was no record of an order being placed.
In essence, after generating a signal, the bot was silently suppressing it.
The reason this went undetected for 41 days was a flaw in my log design. I wasn't recording the fact that "a signal occurred" in an easily monitorable way. I was logging "actions" like orders and errors, but not the "opportunities" where an action should have been taken. This made it impossible to tell if the continuous inaction was normal or abnormal.
To prevent these kinds of silent failures, the only way is to log both what the bot intended to do and what it actually did, then monitor the discrepancy. I've dubbed this an "Implementation Fidelity Check."
Specifically, I modified the log output to count the progression from signal generation to execution across different stages.
With this in place, I can immediately spot anomalies like "qualified
is increasing but taken
remains zero" or "the count for fired
and opened
doesn't match."
I rolled out this system to all my bots and set up daily aggregations. And lo and behold, a host of problems became visible.
| Bot | Opportunities | Executions | Fulfillment Rate | Notes |
|---|---|---|---|---|
| trend_exec | 1 | 0 | 0% | 🔴 1 opportunity unfulfilled |
| gotobi_paper | 6 | 4 | 67% | ⚠️ Missed end-of-month signals |
For trend_exec
, an API spec change had added a new mandatory parameter to requests, which I hadn't followed up on. Order requests were silently failing every time. 0% fulfillment rate. This was terrible.
For gotobi_paper
, I had a gap in my calendar definition for specific trading days (like 'Gotobi' days at month-end), leading to signals being generated but not executed. 67% fulfillment rate. Another missed opportunity.
What I learned from this incident is that "no errors ≠ normal."
Failures where a process dies or throws an exception and stops are detectable, so they're manageable. The truly terrifying ones are silent failures that don't throw errors, simply bleeding opportunity in the background.
To prevent this, a system that records the bot's "intent (opportunities)" separately from its "actions (executions)" and monitors the fulfillment rate is indispensable. Without it, you can't tell the difference between normal waiting and a critical bug.
In personal dev environments, it's easy to push off setting up monitoring like this. But if you're running bots in production, I've painfully realized it's one of the first features you absolutely must implement. 41 days of lost opportunities was, frankly, quite painful.
From now on, when building a new bot, I'll prioritize integrating this "Implementation Fidelity Check" system. If you're building similar bots, I hope this experience proves useful.
I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.
If a provider-agnostic RAG Q&A API is useful to you, mine is MIT-licensed on GitHub: rag-faq-api. It runs and passes its full test suite with no API key (offline stub LLM + hashing embedder), swaps to Claude / Gemini / OpenAI via one env var, and ships a retrieval-quality harness (Hit@k / MRR / Recall@k) with a chunking sweep.