Tracked down and fixed the real cause of a memory problem, and accidentally killed production while doing it
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
Today I finally traced a problem that had been a mystery for weeks all the way down to its root cause and fixed it. Along the way I also made two mistakes.
For a while now, the dev environment had been dying at a specific hour every early morning (I'd already moved core processes into separate services independent of the dev environment because of this, but that only masked the symptom rather than fixing the cause). Today, digging deeper into the logs with another AI, I finally found the real cause. A library used for heavy data computation was, at the Python code level, clearly releasing memory it was done with — but at the OS level, that memory wasn't actually being reclaimed (a phenomenon called memory fragmentation). This built up a little every day until the system eventually ran out of memory and died overnight. On top of that, digging further turned up a side issue: a calculation that only needed to run once a day was being repeated multiple times a day for no reason.
I fixed three things — never recompute if a cached result already exists, explicitly force-release memory after heavy computation finishes, and skip the computation entirely if the date hasn't changed. Measuring memory usage before and after, one process dropped by more than 20x. This properly fixed something that had been papered over for weeks with "just restart it, it'll be fine."
I restarted the related service to deploy the fix above, and it happened to land right in the middle of that night's core analysis job, which was actively running. I'd even written "this job is top priority, never interrupt it for any reason" directly in the code — and then broke that exact rule while deploying. There was a safeguard that auto-resumed it a few minutes later, so there was no major loss, but it was a reminder to be more careful about deploy timing.
I d the analysis job for a few minutes to fix something else, and in that window a separate piece of automation decided "nothing is running right now, so this is my chance" and kicked off a different GPU-heavy job. Shortly after, the analysis job resumed and the two collided over the same GPU resource, and one of them had to be cleaned up. The cause was missing the distinction between "nothing running right now" and "today's work is actually done."
Neither mistake today led to serious damage, but it felt ironic to fix a weeks-long mystery and immediately make two new mistakes the same day. The lesson to always check "what's currently running" before touching deploys or automation got relearned twice over today.