Nobody’s agent fleet fails the way the vendors say it will

A practitioner running 49 scheduled AI agents on one laptop reports that no serious failure in the fleet has started with a bad model answer; instead, failures originate lower in the stack, such as a job never loaded, a shell command existing only in the terminal, an expired credential, or a timeout guard built on a utility the operating system does not ship. The author, drawing on Richard Cook's paper on how complex systems fail, describes three incidents from one week where the agent did not go rogue but Unix happened, and emphasizes that scheduler definitions are claims about behavior, not evidence of it.

I run 49 scheduled AI agents on one laptop. Another 24 sit beside them, deliberately switched off. That distinction matters more than it sounds. After eighteen months, I have stopped being surprised by model errors. Models misunderstand requests and occasionally produce confident nonsense. I expect that. I test for it. What still catches me is how rarely those errors cause an outage. No serious failure in my fleet has started with a bad model answer. They start lower in the stack: a job never loaded, a shell command that existed only in my terminal, a credential expired between runs, a timeout guard built on a utility the operating system does not ship. The agent did not go rogue. Unix happened. Most agent-fleet guidance is written forward from a proposed architecture: a router, specialist agents, tools and memory, clean arrows between boxes. I have drawn those diagrams myself. But every design rule I actually use came backward from an incident, after the arrows stopped moving and the dashboard still looked green. Three failures from one week explain the gap. The first failure looked correct everywhere I checked. The job existed on disk, its configuration was valid, a symlink pointed into the expected directory, and the script ran when I invoked it by hand. It had never been registered with the scheduler. I had written the definition, moved it into place, confirmed the file was where it belonged and never once asked the scheduler whether it had picked it up. For several days it did nothing, and produced no error, because no process existed to fail. I had inspected the desired state and mistaken it for the actual one. That sounds embarrassingly basic. It is also a small version of a familiar systems problem. Richard Cook’s paper on how complex systems fail https://www.adaptivecapacitylabs.com/HowComplexSystemsFail.pdf argues that incidents are usually attributed to a nearby, obvious cause even though failure emerges from several normally tolerated conditions. My nearby cause was an unloaded job. The conditions around it mattered more: manual execution passed, the file tree looked right, no freshness alarm existed and my review stopped at configuration. I now treat a scheduler definition as a claim about behavior, not evidence of it. The evidence is the live registry, the last start time, the last successful completion and a fresh artifact produced through the scheduled path. A file can tell me what should run. It cannot tell me what is running. The second failure began with an agent command I use dozens of times a day. I had shortened the executable and its arguments to an alias. In my terminal, the short name worked. Inside a non-interactive script, it did not exist. The failed command should have made this obvious. Instead the wrapper continued, ran a harmless bookkeeping line and returned that line’s status. The scheduler recorded exit code 0. The agent had not run, and the wrapper announced success. There were two bugs, not one. I had relied on an interactive-shell convenience in production code, and I had allowed a later command to erase the failure signal. The first stopped the work. The second stopped me from knowing, and that one cost far more. A job that dies loudly gets fixed the same morning. A job that dies quietly gets fixed whenever somebody happens to go looking for output that was never there. In my case that was four days, and only because I wanted to read the report, not because anything told me to. Shell portability is old work. The current POSIX operating-system specification https://pubs.opengroup.org/onlinepubs/9799919799/ still has to spell out command execution and environment behavior in exhaustive detail, because those assumptions really do change between an interactive shell and a script. This is settled ground. Agent projects just make it easy to walk back onto. The model call feels like the sophisticated part, so the three-line wrapper around it escapes the scrutiny I would give ordinary production code. My fix was not clever. Scheduled scripts now call absolute paths, or resolve and validate them at startup, and run with a deliberately small environment. They fail on an unset variable. Most importantly they verify the intended effect before reporting success. Exit code 0 is necessary. It is not proof that work happened. The third failure was a timeout guard. An agent could occasionally stall on a tool call, so I wrapped it with a 40-minute limit and moved on. The guard depended on a command common on Linux but not installed by default on macOS. My wrapper treated the missing utility like a non-event and continued through its fallback path. For a week, I believed every run had a hard ceiling. None did. Nothing hung during that week, which is the part I keep turning over. The protection was absent and the system looked fine, because the condition it existed for simply did not arise. I got no signal at all — I got lucky, and luck reads exactly like working code right up until it doesn’t. This bothered me more than a stalled job would have. A broken feature reveals itself when someone uses it. A broken safety mechanism stays invisible until the moment the system is already in trouble. So I added a rule: every guardrail has to fail its own startup test. The timeout now proves itself against a process I deliberately hang, which is how I found the missing utility in the first place – it took one afternoon and I should have done it a year earlier. Locks, credential checks and circuit breakers get the same treatment. If I cannot force a protection to fire on demand, I do not get to count it as protection. That instinct is not just mine. A 2025 USENIX OSDI paper on silent semantic failures https://www.usenix.org/system/files/osdi25-lou.pdf argues the same thing at far larger scale: process survival and surface-level success are weak substitutes for checking the behavior a system promised. My fleet is tiny next to the systems in that paper. The lesson transfers anyway. I need an oracle for the guardrail, not just an absence of errors. The most important failure was quieter. One agent summarizes a small set of email inputs. During a scheduled run its token expired. It could have returned “zero new items,” which would have matched the empty response in front of it and been, operationally, a lie. It had not seen an empty inbox. It had failed to see the inbox. Instead, the run reported unknown , named the inaccessible source and refused to produce a summary. It then carried on with the parts of the job it could still do, and said which those were. The report that morning was shorter than usual and honest about why. That took me ten seconds to read. Reconstructing the same fact from a confident zero would have taken a week. That is what I now want from every agent in the fleet. A good agent reports unknown. A bad one reports zero. The difference is not philosophical. Zero is a measurement. Unknown is a statement about whether measurement was possible. Collapse them and the fleet looks healthiest when it is blind. Observability guidance usually begins with logs, metrics and traces. The OpenTelemetry observability primer https://opentelemetry.io/docs/concepts/observability-primer/ gives the conventional framing: a properly instrumented application emits enough signals to answer questions about its behavior. That is necessary, but agent fleets need another layer. I also need evidence about the boundaries of observation: which sources were reachable, which credentials were valid, how fresh each input was and which expected jobs emitted nothing at all. A trace explains what a process did. It cannot explain a process that never ran, unless something outside it was watching for the absence. I now model each scheduled run with more than success and failure. A run can have succeeded, failed, arrived late, been skipped by policy or be unknown because an input could not be observed. That last state is the awkward one, and it is the one that earns its keep. Every expected run has a freshness deadline. Every important data source has an availability result separate from its item count. A heartbeat is accepted only after the job produces the artifact the schedule exists to produce. This creates more yellow on the dashboard. Good. I would rather investigate an honest yellow than trust a fabricated green. The fleet is not less reliable than it was before I started tracking this. It is exactly as reliable as it always was, and I can finally see it. None of this is new computer science, which is exactly why it is easy to miss. Agent systems arrive wrapped in new language — planners, tool use, memory, reflection — but they still depend on schedulers, shells, paths, credentials, clocks and exit codes. The new layer does not repeal the old ones. It gives the old ones more ways to fail quietly. I still evaluate model quality. I still test prompts, tool selection and output grounding. But the questions I ask first are less fashionable. Did the scheduler load the job? Did it start when expected? Did the wrapper invoke the binary I thought it did? Did the safety check prove itself? The rule I wrote down, and then broke anyway, is simple: Never infer a system’s current behavior from a configuration file. Read the running state. That is also why 24 agents are switched off rather than deleted. Each one was useful enough to build and not useful enough to keep watching. Every agent I add is another scheduled job that can quietly stop, another credential that can lapse, another silence I have to notice. The cost of a fleet is not the tokens. It is the number of things that can fail without telling you. The architecture diagram tells me what I intended to build. The running state tells me what I actually have. Only one of them gets to wake me up at 3 a.m.