{"slug": "aws-infinity-run-80000-ai-agents-in-8-gb-ram", "title": "AWS Infinity: Run 80,000 AI Agents in 8 GB RAM", "summary": "AWS and UC Berkeley's Hydro project released Infinity on October 23, a Rust runtime that suspends AI agents between tool calls and cuts idle memory to roughly 103 KB per agent, fitting 80,000 concurrent agents in 8.4 GB of RAM. AWS benchmarked 80,000 agents running 3.2 million completions on a single AMD EPYC thread using an in-memory store and a simulated model, and the company is asking for feedback before making production claims. Infinity reimplements the agent loop as bounded slices that load history, run one model completion, dispatch at most one tool call and serialize the turn, letting the hosting serverless function terminate while tools run.", "body_md": "Every agent framework you use right now holds memory for agents that aren’t doing anything. An agent waiting on a database query or a web search isn’t reasoning — it’s just parked in RAM, keeping a process alive because that’s how frameworks are built. AWS and UC Berkeley’s Hydro project released **Infinity** on October 23 to fix this. It’s a Rust runtime that suspends agents between tool calls, drops idle memory to roughly 103 KB per agent, and fits 80,000 concurrent agents in 8.4 GB.\n\n## The Problem Is Structural\n\nMost agent frameworks were designed for one agent at a time. When your agent calls a tool — a database, a web search, an external API — the framework keeps the full agent alive in memory while it waits. That means a running process, an open connection, the conversation history held in-heap, and all the framework state layered on top. Agents typically idle between 16% and 60% of their wall-clock time waiting on tools. For a small deployment that’s fine. For a system running thousands of concurrent agents, you hit memory walls fast.\n\nThe insight behind Infinity is simple: between tool calls, an agent is just data. It’s not thinking. There’s no reason to keep it resident.\n\n## How Infinity Works: Slices and Suspension\n\nInfinity reimplements the agent loop as a series of bounded “slices.” Each slice does exactly one thing:\n\n1. Load conversation history from a pluggable store\n2. Run one model completion\n3. Dispatch at most one tool call\n4. Serialize the turn and yield\n\nWhen the tool result arrives, a new slice loads the history and continues. Between slices, the agent doesn’t exist as a running process — it’s a record in a store. The runtime treats tool calls as suspend points, the same way programming languages treat `async/await`, except the process actually exits. There is no call stack held in memory. There is no open connection. Just bytes.\n\nThis is closer to how Erlang actors work than how Python coroutines work. An Erlang process has near-zero overhead at rest because its state is immutable data. Infinity applies that model to LLM agents.\n\n## The Numbers\n\nAWS benchmarked 80,000 agents running 3.2 million completions on a single AMD EPYC thread using an in-memory store and a simulated model. Idle memory per agent after 20 tool-calling turns: approximately 103 KB. Total memory for 80,000 idle agents: 8.4 GB.\n\nAWS extrapolates this to a Raspberry Pi 5 (8 GB RAM) — a useful illustration of the scale, even if running 75,000 production agents on a Pi is not a real recommendation. One important caveat: the benchmark uses a simulated model, not real LLM API calls. Production performance will depend on actual model latency, tool latency, and store I/O. AWS is asking for feedback before making any production claims.\n\n## The Serverless Unlock\n\nBecause agents are fully suspended between slices, the serverless function hosting the agent can terminate while the tool runs. If your agent fires a tool call that kicks off an async job and waits for a callback that takes hours — or days — nothing runs in between. No idle Lambda. No idle container accruing cost. This isn’t just a cost optimization; it makes long-horizon agent tasks viable on serverless platforms that have execution time limits.\n\n## Where This Fits in the Agent Infrastructure Stack\n\nIt’s been a busy week for agent infrastructure. [LayerFS 1.0](https://byteiota.com/layerfs-1-0-give-every-ai-agent-its-own-workspace-in-20ms/) brought isolated workspaces per agent, NVIDIA OpenShell added hardware-sandboxed execution, and Microsoft’s Execution Containers went GA for agent sandboxing. Those releases solve isolation and security. Infinity solves scale. They’re not competing — a production system running thousands of agents would ideally want both.\n\nInfinity doesn’t replace LangGraph, CrewAI, or any existing framework. It’s a runtime layer underneath them — or alongside them as an alternative programming model for workloads where concurrency is the primary constraint.\n\n## Get It\n\nInfinity is open source under Apache 2.0, written in Rust, and available on [GitHub](https://github.com/hydro-project/infinity). The official documentation is at [infinity.hydro.run](https://infinity.hydro.run/). The Hydro project — the UC Berkeley and AWS distributed systems research effort behind Infinity — is at [hydro.run](https://hydro.run/). The [AWS announcement post](https://aws.amazon.com/blogs/opensource/introducing-infinity-a-runtime-for-extreme-agent-concurrency/) covers the full technical design and benchmark methodology.\n\nThis is a research release. AWS says so clearly in the announcement. It is not production-ready, the programming model is different from what you’re used to, and the benchmark numbers come from a simulated workload. But the architecture is sound, the code is available, and this is the kind of foundational work that tends to show up in production frameworks two years later. Worth understanding now.", "url": "https://wpnews.pro/news/aws-infinity-run-80000-ai-agents-in-8-gb-ram", "canonical_source": "https://byteiota.com/aws-infinity-run-80000-ai-agents-in-8-gb-ram/", "published_at": "2026-10-26 03:16:16+00:00", "updated_at": "2026-10-26 04:15:31.745453+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["AWS", "UC Berkeley", "Hydro", "Infinity", "AMD EPYC", "Raspberry Pi 5", "LangGraph", "CrewAI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/aws-infinity-run-80000-ai-agents-in-8-gb-ram", "markdown": "https://wpnews.pro/news/aws-infinity-run-80000-ai-agents-in-8-gb-ram.md", "text": "https://wpnews.pro/news/aws-infinity-run-80000-ai-agents-in-8-gb-ram.txt", "jsonld": "https://wpnews.pro/news/aws-infinity-run-80000-ai-agents-in-8-gb-ram.jsonld"}}