July 23, 2026, (Inside AI) — A team at Planck discovered that adding more AI agents to their production system didn't just slow things down—it caused timeouts and mysterious latency spikes. The culprit wasn't the large language models themselves, but tiny CPU tasks that snowballed under massive concurrency, exposing a design flaw in how asynchronous I/O handles real-world workloads.
When Planck first deployed a handful of LLM agents, performance was acceptable. But as they scaled to hundreds of agents, each triggering dozens of sub-agent calls, latency grew non-linearly. Engineers initially blamed external model providers, but logs revealed that requests were completing quickly on the provider side. The real issue was closer to home: their own event loop was overwhelmed by a deluge of small CPU-bound operations.
The team's Python-based service used asyncio
and aiohttp
to fan out requests concurrently. In theory, since most work was I/O-bound, latency should have been dominated by the slowest LLM call, not the total number of agents. But profiling told a different story. Serialization and deserialization overhead, connection pool limits, and the Global Interpreter Lock (GIL) all contributed, but the root cause was more fundamental.
"Every time an LLM response arrives, the event loop has to execute a small amount of CPU work before it can move on to the next coroutine," the Planck team noted. "Individually, these operations are almost free. Collectively, across hundreds or thousands of concurrent calls, they become the bottleneck."
Even after optimizing JSON parsing with orjson
and increasing the aiohttp
connection limit, latency still climbed with agent count. A simplified benchmark using asyncio.sleep
for I/O and `time.sleep`
for CPU work confirmed the pattern: pure I/O tasks scaled effortlessly, but mixing in minuscule CPU work caused linear latency growth.
The insight led to a redesign. Instead of a single process handling all agents, the team introduced a router that distributed work across multiple workers, each with its own event loop and connection pool. This didn't eliminate CPU work—every response still needed deserialization and validation—but it spread the load across cores and machines, preventing any single process from becoming a bottleneck.
"We did not eliminate the bottleneck, we divided it into smaller units that could be scaled horizontally," the team explained. The new architecture allowed them to add agents without degrading existing performance and freed engineers from constant micro-optimizations.
Why Fan-Out Architectures Hit a CPU Wall #
The problem Planck faced is not unique to Python or AI agents. It's a classic fan-out challenge: a single request spawns many parallel tasks, each requiring a sliver of CPU after I/O completes. In any language, there's a finite number of cores, and when thousands of callbacks compete for them, latency spikes. The GIL exacerbates this in Python by serializing bytecode execution, but even Go or Java would eventually hit limits.
Research on scalable microservice architectures highlights that fan-out patterns demand careful resource management to avoid thundering herd problems. Planck's solution—splitting work across multiple processes—aligns with best practices for CPU-bound task distribution. However, it introduces complexity in state management and load balancing.
Some engineers might argue for switching to a language without a GIL, but Planck's experience suggests that architectural choices matter more than runtime. "Good engineering isn't just about writing efficient code, it's about designing systems that continue to scale as they grow," the team wrote. "Well-written code cannot compensate for a poorly designed system."
The team also hinted that their RAG pipeline required a shift from in-memory to shared storage, a topic they covered in a separate post on when not to use vector databases. This underscores how scaling agents often forces a reevaluation of the entire data infrastructure.
Planck's story is a cautionary tale for anyone deploying LLM agents at scale. The initial simplicity of async I/O can mask hidden CPU costs that emerge only under load. Profiling and incremental optimizations help, but the ultimate fix is to design for horizontal scalability from the start. As the team put it: "Sometimes, you have to step back and ask yourself: does this approach really scale, or should we consider redesigning our system?"