A 20-year shortcut to build reliable agents

Maxim Fateev, CTO and co-founder of Temporal, warns that agentic systems face the same distributed systems failures—state loss and retry storms—that he solved over 20 years at Amazon and Uber, and that Temporal's durable execution platform now runs production workloads for OpenAI, GitLab, Lovable, Docker, and Cloudflare with more than 2,500 customers globally. He argues that developers building agents today are rediscovering the need for flow control, queues, and rate limiting to prevent self-inflicted outages.

When Maxim Fateev, CTO and co-founder of Temporal, joined Zero-Shot Learning, he brought a historical perspective to the challenges developers face when building agentic systems today. From vanishing state to retry storms, Fateev saw that the failures of deploying long-running agents have parallels to the problems he’s been working on for decades. Maxim joined Amazon in 2002, where he co-created Simple Workflow Service, the internal orchestration platform that became one of the most widely used services at Amazon. At Uber, he built Cadence, the open-source predecessor to Temporal https://temporal.io/ , the durable execution platform, which he co-founded in 2019. Temporal now runs production workloads for OpenAI, GitLab, Lovable, Docker, and Cloudflare, and has more than 2,500 customers globally. As the industry builds agentic systems, Fateev is watching it rediscover exactly what his infrastructure was built to solve. Agents become distributed systems the moment they cross a network. Every call to an LLM, every tool invocation, every write to a downstream service crosses a process boundary, and a process boundary is where distributed systems failures begin. Two common ways agents fail in production https://1password.com/blog/agent-evals-production are state loss and retry storms. While working, an agent builds state, e.g. a record of which tools it called, the results it received, and how far it progressed in a task. When the process crashes, that record is gone. There is no checkpoint to resume from, no record of what was completed, no way to distinguish completed work from incomplete work. The next run starts from scratch, leaving the operator unsure which actions can be safely repeated. When an agent calls an external service and gets no response, it retries. If it does not succeed, that retry turns a short synchronous call into a long-running operation. Multiply that across thousands of agents http://1password.com/blog/how-to-build-secure-agent-swarms-that-power-autonomous-systems hitting the same service simultaneously, and the load compounds, the service stalls, and the retry pressure forces an outage. It ends up becoming a self-inflicted Distributed Denial of Service DDoS . To survive this, a system needs flow control to pace requests, queues to absorb spikes, and rate limiting to protect downstream services. An agentic system built without any of that suddenly needs all of it. And when agents crash under the strain, there is no built-in way to tell which ones were mid-task and need recovery. "All these things stack up, and I think people who started from 'synchronous Python in-memory program' are learning the hard way that these are not simple problems,” Maxim said. Engineers first encountered these challenges in the 2000s while software was becoming the primary way people do business and manage our personal lives. As everything from booking travel to routing deliveries and managing health records became digital processes, the systems supporting it had to grow from single machines into networks of interdependent services. Serving millions of users led to process crashes and lost work, retries amplified into outages, and tasks were repeated because callers retried after partial success without confirmation. Buried in software, and software problems, engineers built their way out. Over time, they developed heartbeats to detect process crashes, queues to absorb retry storms, and architected execution guarantees that preserve workflows even if the process running it fails mid-task. These solutions eventually matured into infrastructure so foundational that most developers building agents today never had to think about it. Until now, when agentic developers have to decide which parts of that history belong in their own systems. When state loss and retry storms appear, developers reach for patterns they already know. For many, event-driven architecture is a natural first choice. Teams will wire services to communicate through shared channels, reducing direct dependencies and making it easier to add or remove components. One part of the system posts a message when something happens, and others listen and react. This approach is loosely coupled by design, so it’s flexible, and it’s a family way to coordinate work that most teams have built before. But that flexibility has a cost that becomes due when the messaging format changes. Update one, and you may not know which services depend on it or what state they have already accumulated. There is no single, visible contract between components. Instead, the contract is partly embedded in the assumptions each service makes about every other service. Maxim describes this dependency by saying, “Events are the global variables of distributed systems.” In an agent system, a change can surface later as an incorrect result, an inconsistent downstream action, or a task that inaccurately appears incomplete. Workflow diagrams solve a different part of the problem. BPMN, Step Functions, and similar tools show the intended sequence. But agent tasks change as tools return data, state accumulates, and new decisions follow. The sequence is visible but the reasoning and state that shape it aren’t. Both approaches coordinate work, but neither event channels nor workflow diagrams can guarantee that work will survive a failure. When building the infrastructure today’s software depends on, distributed systems engineers developed a process called durable execution to ensure every external operation a process performs is recorded, so if the process crashes, the platform replays those results and picks up where it left off. In the episode https://www.youtube.com/watch?v=N1wvZP41H78 , Maxim demonstrated durable execution using the OpenAI Agents SDK. For builders wondering what agent recovery looks like without writing recovery logic, Maxim kills the worker process mid-task and resumes the agent from exactly where it left off. He then restarts it with an invalid API key and immediately shows the stack trace, retry status, and exponential backoff surface in the Temporal UI, with the agent recovering automatically once the credentials are fixed. None of it requires a single line of recovery code in the application. The failures showing up in agentic deployments right now are the same problems distributed systems engineers spent two decades solving. While state and retries are only one aspect of developing and securing agentic workflows, the patterns they express in production mirror those documented by Fateev and his work bringing Temporal to market. The engineers who built it learned the hard way so you don't have to. The shortcut to building reliable agents is standing on the shoulders of those who built reliable infrastructure before you. Building reliable agents means handling their credentials reliably too.