Development covered 5 Aug 2026 to 7 Aug 2026 (commit dates).
A lot of the work on this project runs overnight with nobody watching. The job plays the game, or edits the book, or sweeps the design documents, for ten hours or so, and in the morning there is a report and a stack of commits. The game tracks dozens of player attributes, and a ten-hour playthrough is the only way to watch them all move.
Getting the work right turned out to be the easy part. Almost everything that went wrong went wrong in the surrounding machinery, and here is the list of it.
The first rule and the one with the largest effect. A run that stops to ask whether it should do something has not saved anybody any effort. It has converted eight hours of unattended work into a question waiting on a desk, and the answer arrives too late to be useful.
So the job decides. It argues both sides against a fixed set of criteria, picks one, records the reasoning and the alternative it rejected, and continues. If a decision turns out wrong, that is one thing to revisit in the morning instead of ten hours of nothing.
The version of this that took longest to accept is that stopping is almost never right. Every case where I wanted a human was a case where an acceptable answer existed and I was reluctant to own it. The book it edits is the companion novel, and a chapter such as the Body Integration act shows the scale of what it keeps consistent.
Long runs leak. The browser they drive grows, the worker processes accumulate, and eventually the machine is swapping and everything takes ten times as long while appearing to still be working.
Three attempts to bound that, and only the third was right. Measuring total memory across the machine tells you there is a problem and not what to do about it. Measuring per process is worse, because one browser is several processes and the biggest one was not the one I was looking at: most of the growth was in a graphics helper process that never appeared in my accounting at all.
The unit that works is the unit you can act on. You restart a browser, not a process, so the measurement has to be the sum of everything that browser owns, compared against a limit, with the action being to recycle that browser.
Two follow-ups, both found only because the thing finally fired. A run that started before the watchdog existed never got one, because the watchdog was created during setup and setup had already happened. And the first recycle handed back a fresh instance that was missing part of its configuration, so the replacement leaked differently rather than not at all.
The command to halt the pipeline did not halt anything, for four separate reasons in sequence, each of which was hidden by the one in front of it.
Worth noting because the shape recurs. The process you started is not the process doing the work, so a stored identifier points at a parent that has already exited. Matching processes by name finds your own matching command as well. And the tool that lists processes truncates its output to the terminal width, so a long command line looks like no match at all, and the count of running jobs reads as zero while the machine is clearly busy.
Every one of those makes a monitoring command lie in the safe-sounding direction, telling you nothing is running.
Small and entirely my fault, kept because it is such a common trap.
A setting bounded how much work a stage would do. I wanted to run the earlier stages and skip that one, so I set it to zero. Zero meant no limit, so it did all of it.
Anything where zero could plausibly mean either none or unlimited should not use zero for either. Name the states.
The last one is about budgets. The run has a ceiling, and it measures its own consumption against it.
The ceiling was reading a figure that only counted the main process, while the run had spawned a set of helpers doing most of the work. So the meter showed a fraction of the true usage, and the ceiling that was supposed to stop things early never came close to firing.
Related, and the thing that actually caused it: a limit on how many helpers may run at once bound me and not them. Each helper was free to spawn its own, so four became twenty one. If you cap concurrency, cap it somewhere every participant has to pass through, not in the instructions given to the first one.
Start playing in the browser for free if you want to see what the overnight runs see; more devlogs are at Xianxia idle incremental game.