Every agent framework you use right now holds memory for agents that aren’t doing anything. An agent waiting on a database query or a web search isn’t reasoning — it’s just parked in RAM, keeping a process alive because that’s how frameworks are built. AWS and UC Berkeley’s Hydro project released Infinity on October 23 to fix this. It’s a Rust runtime that suspends agents between tool calls, drops idle memory to roughly 103 KB per agent, and fits 80,000 concurrent agents in 8.4 GB.
The Problem Is Structural #
Most agent frameworks were designed for one agent at a time. When your agent calls a tool — a database, a web search, an external API — the framework keeps the full agent alive in memory while it waits. That means a running process, an open connection, the conversation history held in-heap, and all the framework state layered on top. Agents typically idle between 16% and 60% of their wall-clock time waiting on tools. For a small deployment that’s fine. For a system running thousands of concurrent agents, you hit memory walls fast.
The insight behind Infinity is simple: between tool calls, an agent is just data. It’s not thinking. There’s no reason to keep it resident.
How Infinity Works: Slices and Suspension #
Infinity reimplements the agent loop as a series of bounded “slices.” Each slice does exactly one thing:
- Load conversation history from a pluggable store
- Run one model completion
- Dispatch at most one tool call
- Serialize the turn and yield
When the tool result arrives, a new slice loads the history and continues. Between slices, the agent doesn’t exist as a running process — it’s a record in a store. The runtime treats tool calls as suspend points, the same way programming languages treat async/await, except the process actually exits. There is no call stack held in memory. There is no open connection. Just bytes.
This is closer to how Erlang actors work than how Python coroutines work. An Erlang process has near-zero overhead at rest because its state is immutable data. Infinity applies that model to LLM agents.
The Numbers #
AWS benchmarked 80,000 agents running 3.2 million completions on a single AMD EPYC thread using an in-memory store and a simulated model. Idle memory per agent after 20 tool-calling turns: approximately 103 KB. Total memory for 80,000 idle agents: 8.4 GB.
AWS extrapolates this to a Raspberry Pi 5 (8 GB RAM) — a useful illustration of the scale, even if running 75,000 production agents on a Pi is not a real recommendation. One important caveat: the benchmark uses a simulated model, not real LLM API calls. Production performance will depend on actual model latency, tool latency, and store I/O. AWS is asking for feedback before making any production claims.
The Serverless Unlock #
Because agents are fully suspended between slices, the serverless function hosting the agent can terminate while the tool runs. If your agent fires a tool call that kicks off an async job and waits for a callback that takes hours — or days — nothing runs in between. No idle Lambda. No idle container accruing cost. This isn’t just a cost optimization; it makes long-horizon agent tasks viable on serverless platforms that have execution time limits.
Where This Fits in the Agent Infrastructure Stack #
It’s been a busy week for agent infrastructure. LayerFS 1.0 brought isolated workspaces per agent, NVIDIA OpenShell added hardware-sandboxed execution, and Microsoft’s Execution Containers went GA for agent sandboxing. Those releases solve isolation and security. Infinity solves scale. They’re not competing — a production system running thousands of agents would ideally want both.
Infinity doesn’t replace LangGraph, CrewAI, or any existing framework. It’s a runtime layer underneath them — or alongside them as an alternative programming model for workloads where concurrency is the primary constraint.
Get It #
Infinity is open source under Apache 2.0, written in Rust, and available on GitHub. The official documentation is at infinity.hydro.run. The Hydro project — the UC Berkeley and AWS distributed systems research effort behind Infinity — is at hydro.run. The AWS announcement post covers the full technical design and benchmark methodology.
This is a research release. AWS says so clearly in the announcement. It is not production-ready, the programming model is different from what you’re used to, and the benchmark numbers come from a simulated workload. But the architecture is sound, the code is available, and this is the kind of foundational work that tends to show up in production frameworks two years later. Worth understanding now.