[Jul 10] Caught Three Things That Were Going Wrong Quietly

A developer's algorithmic trading system devlog details three silent failures that were caught before causing major issues. The developer capped an LLM's response length to prevent runaway generation, implemented a progress-monitoring mechanism based on silence, and discovered that a wider input window caused GPU memory spillover, slowing throughput by 2.7x in the evening. The developer also found that an external library for market holidays was outdated, and emphasized the need to measure performance after any setting change.

From a generation that spiraled out of control forever, to a morning fix that quietly halved throughput by evening This is the English version of a post originally written in Korean for my algorithmic trading system devlog new tab . Spanning the weekend, today was about wrapping up things found over the past few days. They had one thing in common: all three had been going wrong quietly, with no visible sign. I noticed one of the overnight analysis jobs oddly stuck. It turned out the local LLM server had no upper limit set on response length. Normally a single response ends quickly, but if the model ever got stuck repeating itself without stopping, it could spiral out to tens of thousands of tokens and tens of minutes. What made it worse was that the monitoring setup couldn't catch this case at all — it only got noticed because someone happened to ask "why is this so slow?" I capped the response length to stop runaway generation, and while I was at it, designed the progress-monitoring mechanism that was actually needed. One interesting design choice came out of this: rather than tracking "how many minutes has this been running," it tracks "how many minutes since anything was last logged." The original idea was to flag anything running some multiple longer than a ticker's average time, but that risked wrongly killing large tickers that normally take longer, or missing a small ticker that got stuck. Using "silence" itself as the signal works consistently regardless of ticker size, with no per-ticker tuning needed. For now it just alerts in observation mode; once a few days pass with no false positives, I'll turn on the actual kill-switch. A few days ago I widened how much input the model could read at once, to fix a problem where some inputs were too long and getting truncated. That had an unexpected side effect — with the exact same settings, throughput was fast at dawn and nearly 2.7x slower by evening. The cause: a wider input window uses more GPU memory, and during the day and evening other programs browser, remote access, other analysis jobs were already using some GPU memory, leaving little headroom. Once the model no longer fit entirely on the GPU and part of it spilled over to CPU, speed dropped sharply from that point on. At dawn, everything else was idle and the GPU had headroom, so it stayed fast. I got this wrong twice along the way. I judged one report as "looks duplicated," when it was actually a counting mistake that merged two entries for the same ticker into one. I also brushed off "this ticker is just big, so it's naturally slow" — turns out the same ticker had finished much faster on other days. Relearned the lesson: "whenever you change a setting, you have to actually measure speed before and after." Once again, if nobody had actually measured it, this would have slipped by unnoticed. I found that the external library used to determine market holidays was still treating a day newly designated as a legal holiday this year as a "trading day." This library is widely used and well-tested, but it hadn't caught up in real time with a domestic holiday designation change made earlier this year. Left as-is, that entire day would have gone wrong — the automated analysis running normally, paper-trading verification running on a day with no data, the morning report going out — a chain of errors from multiple places wrongly concluding "yes, that's a trading day." Fortunately I caught it before that day arrived, and worked around it by manually registering the exception date. This confirmed again that neither an external library nor my own knowledge should be assumed "up to date." Things like domestic rules and schedules that rarely change but occasionally do are especially easy for everyone to miss for a while after they change. The trap in all three was that "nothing looks wrong right now." The runaway only happens once in a while, the slowdown comes and goes with time of day, and the holiday problem shows no sign until the day itself arrives. A day spent catching things that could have quietly slipped by.