{"slug": "agent-execution-runtime-engineered-to-run-millions-of-sandboxes", "title": "Agent execution runtime engineered to run millions of sandboxes", "summary": "Google has introduced Agent Substrate, a secure-by-default agent execution runtime that runs millions of sandboxes at 10x higher density than standard container runtimes, delivering sub-500ms resume operations at over 500 suspend/resume activations per second. The runtime maps a larger set of \"actors\" onto a smaller pool of ready \"workers\" and supports microVMs and gVisor with native zero-trust kernel and network isolation, with a demo multiplexing roughly 250 stateful actors across 8 physical pods for 30x+ oversubscription. Google notes Agent Substrate is not an officially supported Google product and is not eligible for the Google Open Source Software Vulnerability Rewards Program.", "body_md": "NOTE: This is not an officially supported Google product. This project is not\neligible for the [Google Open Source Software Vulnerability Rewards Program](https://bughunters.google.com/open-source-security).\n\nAgent Substrate is a secure-by-default agent execution runtime engineered to run millions of sandboxes with 10x higher density than standard container runtimes. Purpose-built for the era of autonomous agents, Substrate delivers sub-500ms resume operations at over 500 suspend/resume activations per second with native zero-trust kernel and network isolation. It supports multiple sandbox technologies including microVMs and gVisor, enabling consistent lifecycle operations for all sandbox types.\n\nAt its core, Agent Substrate maps a larger set of “actors” (applications such as agents) onto a smaller set of ready “workers”, relying on the fact that agent-like applications tend to be idle most of the time to achieve heavy multiplexing. It provides functionality to manage an actor’s lifecycle (e.g. create/destroy, suspend/resume), to assign actors to workers in real time, and to route incoming traffic to them.\n\nAgent Substrate is intended to be a low-opinion system. The workloads it manages don't have to be literal AI agents, but those are the best example of the kind of applications it is designed for. It is not an SDK for building agents, but rather a system for running them at scale.\n\nAgent Substrate leverages Kubernetes for the infrastructure provisioning and worker lifecycle management (Kubernetes Pods). It builds on top of Kubernetes features like Pods and Pod autoscaling, while Agent Substrate provides agent-specific scheduling and control to achieve lower latency. Using Kubernetes as the underlying system enables consistent infrastructure management across all workloads types that are required for end to end agentic deployments and allows holistic infrastructure optimizations for RL scenarios that span agentic, inference and training cycles.\n\n*Watch the Agent Substrate cluster multiplex ~250 stateful actors across just 8 physical pods.*\n\nThis demo highlights the core developer experience and \"Agentic Infrastructure\" capabilities of Substrate:\n\n1. **Actor Teleport:** High-performance suspend and resume of actors onto any available worker in the pool with sub-second activation.\n2. **State Persistence:** Persistent working memory (volatile RAM) and filesystem state preserved perfectly across hibernation cycles via full-state snapshots.\n3. **Agent Multiplexing:** Demonstrates 30x+ oversubscription by \"juggling\" a large registry of stateful actors onto a small pool of shared physical pods.\n\nTo reproduce this demo in your own cluster, please refer to the detailed walkthrough in the **[Counter Demo](https://github.com/agent-substrate/substrate/blob/main/demos/counter/README.md)**.\n\nFor more videos and walkthroughs, visit our YouTube channel: **[agent-substrate](https://www.youtube.com/channel/UCN9PPqlTtVxlcpbQ-NWpfZQ)**.\n\nAgent Substrate is designed to be **framework and agent harness agnostic**. Because it manages standard OCI containers at the kernel level (via gVisor), it can host agents built on any stack.\n\n- **Agent Development Kit (ADK):** Support for ADK agents with session state preservation across invocations as actor state. Ideal for all types of agents and stateful tool or subagent calls.\n- **LangChain:** Ideal execution environment for LangChain agents and tool calls.\n- **Claude Code, CodeX, and Antigravity:** Support for high-density, stateful coding environments that preserve system state and filesystem state across sessions.\n- **Model Context Protocol (MCP):** Support for deploying secure, sandboxed MCP servers as Substrate Actors to provide durable tools for any model.\n\n- **[Agent Executor](https://github.com/google/ax):** A distributed agent runtime that demonstrates building a secure, hyper-scalable agent harness on Agent Substrate (see the[announcement blog](https://cloud.google.com/blog/products/ai-machine-learning/agent-executor-googles-distributed-agent-runtime) and[integration guide](https://github.com/google/ax/blob/main/manifests/README.md) ).\n- **[kagent](https://github.com/kagent-dev/kagent):** A CNCF Sandbox project and Kubernetes-native framework for building, deploying, and managing AI agents that uses Agent Substrate to run sandboxed, stateful agent workloads (see the[announcement blog](https://kagent.dev/blog/the-future-of-kagent) ).\n\nAgent Substrate is currently in early development. It is not ready for production use, and the APIs are almost guaranteed to change. We are not making any guarantees about backward compatibility at this stage, and everything in this project may be changed.\n\nCurrently we aim to support the [latest stable release](https://kubernetes.io/releases/) of Kubernetes, and the previous minor release.\n\nFor announcements, technical discussions, and community support, please join\nthe **[ate-dev](https://groups.google.com/g/ate-dev)** Google Group.\n\nWe host a weekly community meeting every Thursday from 10:00am - 11:00am PST.\n\n- Video call link: [https://meet.google.com/uhq-cxvn-dhy](https://meet.google.com/uhq-cxvn-dhy)\n- Or dial: (US) +1 253-289-6971 PIN: 787 664 574 59#\n- More phone numbers: [https://tel.meet/uhq-cxvn-dhy?pin=9044088223662](https://tel.meet/uhq-cxvn-dhy?pin=9044088223662)\n- [Meeting notes](https://docs.google.com/document/d/1obSIvfcafLNniLYTQCcT2eCgxHqa2AQ3Ga7YTsju49s) for the weekly sync meeting\n- [Recordings and transcripts](https://drive.google.com/corp/drive/u/0/folders/1rX1S6vPxPrR8dA1lEBuBEXkGKjHtG-mL) of all community meetings\n\nWe also have channels in the CNCF slack; [request an invite here](https://slack.cncf.io/)\nif you don't have access.\n\n- [#substrate-users](https://cloud-native.slack.com/archives/C0B6RCAJULW) to discuss using substrate.\n- [#substrate-dev](https://cloud-native.slack.com/archives/C0B6M3E2J3D) to discuss developing substrate.\n\nPlease see [CONTRIBUTING.md](https://github.com/agent-substrate/substrate/blob/main/CONTRIBUTING.md) for guidelines on contributing to\nthe project.  We welcome contributions of all kinds, but the project is VERY\nyoung.  Our immediate focus is on building out the core system and demos, so we\nmay not be able to review or merge contributions that don't align with those\ngoals in the near term.\n\nTo quickly set up the complete environment:\n\n1. \nMake sure you have [Go](https://go.dev/doc/install) ,[`kubectl`](https://kubernetes.io/docs/tasks/tools/) , and[` docker`](https://www.docker.com/) installed and configured on your dev machine. We will automatically manage other dependencies via Go, including[`kind`](https://kind.sigs.k8s.io/) .\n2. \nRun the following steps:\n\n```\n# create cluster and local registry (IPv4; IP_FAMILY=dual|ipv6 overrides)\nhack/create-kind-cluster.sh\n\n# install ate, PostgreSQL, rustfs\nhack/install-ate-kind.sh --deploy-ate-system\n\n# install counter demo\nhack/install-ate-kind.sh --deploy-demo-counter\n\n# install kubectl-ate\ngo install ./cmd/kubectl-ate\n\n# create a counter actor in the demo's atespace (--template names the\n# actor template, resolved in the actor's atespace)\nkubectl ate create actor my-counter-1 -a ate-demo-counter --template counter\n\n# port-forward the network router to bind to local port `8000`\nkubectl port-forward -n ate-system svc/atenet-router 8000:80\n```\n\n1. In a **separate terminal** , send an HTTP request to increment the counter:\n\n```\ncurl -X POST \\\n   -H \"ate-target-actor: ate-demo-counter/my-counter-1\" \\\n   -i http://localhost:8000/\n```\n\nWorker capacity is versioned: the dataplane (the atelet DaemonSet and the\nworker pods) schedules only on nodes that carry the\n`ate.dev/substrate-version` label, and the install stamps it on every node\nthat exists when it runs. A node added later hosts no workers until you label\nit with the installed version\n(`kubectl label node <node> ate.dev/substrate-version=<build version>`).\n`kubectl get ds -n ate-system -l app=atelet -L ate.dev/substrate-version`\nprints the installed version, off the atelet DaemonSet the install created.\n\n1. \nCreate and configure your environment file: \n\n```\ncp hack/ate-dev-env.sh.example .ate-dev-env.sh\n\n# Edit .ate-dev-env.sh to match your project and preferences, then source it:\nsource .ate-dev-env.sh\n```\n\n2. \nEnable application-default credentials for gcloud: \n\n```\ngcloud auth application-default login --project=${PROJECT_ID}\n```\n\n3. \nProvision the required GCP resources (GKE cluster, GCS, and IAM bindings): \n\n```\ngo run ./tools/setup-gcp bootstrap\n```\n\n On a fresh project this step also creates the atelet Workload Identity IAM grants that snapshots depend on — see [what `create iam` actually grants](https://github.com/agent-substrate/substrate/blob/main/tools/setup-gcp/README.md#what-create-iam-actually-grants) to audit them or apply them manually. If you bring your own cluster instead,\nnote the required Kubernetes beta APIs can only be enabled**at cluster\ncreation** — see the[Create Cluster warning](https://github.com/agent-substrate/substrate/blob/main/tools/setup-gcp/README.md#2-create-cluster) .\n4. \nDeploy the Agent Substrate system to your cluster: \n\n```\n./hack/install-ate.sh --deploy-ate-system\n```\n\n Nodes that GKE adds later (autoscaling, auto-repair, node upgrades) are born with the node pool's labels, so the pool needs `ate.dev/substrate-version` too; see[Node version labels](https://github.com/agent-substrate/substrate/blob/main/tools/setup-gcp/README.md) .\n5. \nYou can then deploy the sample applications. See [demos/counter/README.md](https://github.com/agent-substrate/substrate/blob/main/demos/counter/README.md) or[demos/sandbox/README.md](https://github.com/agent-substrate/substrate/blob/main/demos/sandbox/README.md) for detailed walkthroughs.\n\n```\n./hack/install-ate.sh --deploy-demo-counter\n```\n\nYou can run individual setup steps to create GCP resources as needed. See `go run ./tools/setup-gcp --help` for available options. For example:\n\n```\ngo run ./tools/setup-gcp create cluster\ngo run ./tools/setup-gcp create bucket\n```\n\nTo run the PostgreSQL store backend on Cloud SQL — with IAM database\nauthentication and no passwords — see [tools/setup-gcp/cloud-sql.md](https://github.com/agent-substrate/substrate/blob/main/tools/setup-gcp/cloud-sql.md).\n\nSimilarly, you can deploy or cleanup specific Agent Substrate components using the installation script. See `./hack/install-ate.sh --help` for all options.\n\n```\n# Re-deploy only ate-apiserver of the ATE system\n./hack/install-ate.sh --deploy-ate-apiserver\n\n# Delete everything (core system and all demos)\n./hack/install-ate.sh --delete-all\n```\n\nIf you need to delete the resources created by the setup script, you can use the provided script `hack/teardown.sh`. This script will delete resources in the reverse order of creation and handles partial failures gracefully.\n\n```\n./hack/teardown.sh --all\n```\n\nOr run individual teardown steps as needed (see `./hack/teardown.sh` for available options).\n\nIf you need to delete the local `kind` cluster and its registry (if it was created by `hack/create-kind-cluster.sh`):\n\n```\n./hack/delete-kind-cluster.sh\n```\n\nWe provide several sample applications demonstrating Agent Substrate's capabilities:\n\n1. **[Counter Demo](https://github.com/agent-substrate/substrate/blob/main/demos/counter/README.md)** : A stateful Go HTTP server demonstrating state preservation across suspends/resumes, and on-demand actor resumption and routing via the Substrate router.\n2. **[Sandbox Demo (Antigravity)](https://github.com/agent-substrate/substrate/blob/main/demos/sandbox/README.md)** : A secure, sandboxed execution environment (running Alpine Linux) that allows arbitrary shell execution while preserving filesystem state across sessions.\n3. **[Claude Code Multiplex](https://github.com/agent-substrate/substrate/blob/main/demos/claude-code-multiplex/README.md)** : Demonstrates oversubscribing physical hardware by multiplexing multiple Claude Code agents onto a limited pool of workers.\n4. **[Multi-Template](https://github.com/agent-substrate/substrate/blob/main/demos/multi-template/README.md)** : Two`ActorTemplate` s running different binaries share one`WorkerPool` , even though the templates live in different atespaces.\n5. **[Request Parking](https://github.com/agent-substrate/substrate/blob/main/demos/parking/README.md)** : An oversubscribed pool where the router holds inbound requests until a worker frees up, instead of returning`503` .\n6. **[Autoscaled WorkerPool](https://github.com/agent-substrate/substrate/blob/main/demos/autoscaled-workerpool/README.md)** : Scales a`WorkerPool` on its assigned-worker count with an HPA fed by prometheus-adapter.\n\n- [Architecture](https://github.com/agent-substrate/substrate/blob/main/docs/architecture.md) : How the control plane, node supervisor, and networking stack fit together.\n- [API Configuration Guide](https://github.com/agent-substrate/substrate/blob/main/docs/api-guide.md) : Detailed reference for configuring WorkerPools, ActorTemplates, Secrets, and Volumes.\n- [Full CLI Documentation](https://github.com/agent-substrate/substrate/blob/main/cmd/kubectl-ate/README.md) : Installation and usage for`kubectl-ate` .\n- [Glossary](https://github.com/agent-substrate/substrate/blob/main/docs/glossary.md) : Core terms (Actor, Atespace, ActorTemplate, WorkerPool, Worker, ate-api-server, atenet, atelet, ateom) and how they relate.\n- [Integration Repositories](https://github.com/agent-substrate/substrate/blob/main/docs/integration-repos.md) : Where integrations live, how their repositories are named, and how fixes flow back to core.\n- [Observability Guide](https://github.com/agent-substrate/substrate/blob/main/docs/observability.md) : Guide to actor logging, metrics, and distributed tracing.\n- [Authentication Guide](https://github.com/agent-substrate/substrate/blob/main/docs/authentication.md) : Configure trusted JWT providers and human credentials.\n- [Enabling man-in-the-middle (MITM) interception for Actor Egress policy](https://github.com/agent-substrate/substrate/blob/main/docs/egress-trust-bundle.md) : Egress policies such as header injection depend on MITM interception of Actor traffic. This guide explains how an Actor should be configured to enable interception.\n- [Request Parking](https://github.com/agent-substrate/substrate/blob/main/docs/request-parking.md) : How the router parks requests through transient worker-pool saturation.\n- [Rolling Upgrade Runbook](https://github.com/agent-substrate/substrate/blob/main/docs/upgrade.md) : Upgrade a running substrate node by node without losing actor state.\n- [Threat Model](https://github.com/agent-substrate/substrate/blob/main/docs/threat-model.md) : Trust boundaries, assumptions, and known risks.\n- [Roadmap](https://github.com/agent-substrate/substrate/blob/main/docs/roadmap.md) : Current limitations and what is planned next.\n- [Benchmarking Guide](https://github.com/agent-substrate/substrate/blob/main/benchmarking/README.md) : Locust-based load tests, monitoring stack, and the orchestrated benchmark harness.\n\n- `cmd/ateapi` : The core control plane API server exposing gRPC endpoints to manage actor and worker lifecycles.\n- `cmd/atelet` : A node-level DaemonSet that supervises physical worker pods, coordinates snapshotting, and manages state transfers.\n- `cmd/atecontroller` : A Kubernetes controller that reconciles WorkerPool custom resources.\n- `cmd/atenet` : A combined networking controller providing Envoy routing and proxy sidecars.\n- `cmd/ateom-gvisor` : An interior-pod helper running inside sandboxed worker pods to execute`runsc` checkpoint and restore commands.\n- `cmd/ateom-microvm` : The micro-VM peer of`ateom-gvisor` , running actors as cloud-hypervisor VMs.\n- `cmd/podcertcontroller` : A \"polyfill\" that provides Pod Certificate signers that\nwill eventually ship in upstream Kubernetes (with different names).\n- `cmd/kubectl-ate` : A CLI tool for managing Agent Substrate resources. See its[README](https://github.com/agent-substrate/substrate/blob/main/cmd/kubectl-ate/README.md) .\n- `cmd/benchmarking` : Synthetic workloads used by the load tests, including`glutton` , which consumes RAM, disk, and file descriptors on demand.\n- `tools/setup-gcp` : A provisioning utility to set up the necessary GCP infrastructure resources (GKE, GCS, IAM).\n- `demos/` : Sample applications demonstrating Agent Substrate capabilities.", "url": "https://wpnews.pro/news/agent-execution-runtime-engineered-to-run-millions-of-sandboxes", "canonical_source": "https://github.com/agent-substrate/substrate", "published_at": "2026-09-21 14:08:13+00:00", "updated_at": "2026-09-21 14:24:40.354481+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "ai-tools", "agent-protocols", "developer-tools"], "entities": ["Google", "Agent Substrate", "Kubernetes", "gVisor", "Agent Development Kit", "LangChain", "Claude Code", "Model Context Protocol"], "alternates": {"html": "https://wpnews.pro/news/agent-execution-runtime-engineered-to-run-millions-of-sandboxes", "markdown": "https://wpnews.pro/news/agent-execution-runtime-engineered-to-run-millions-of-sandboxes.md", "text": "https://wpnews.pro/news/agent-execution-runtime-engineered-to-run-millions-of-sandboxes.txt", "jsonld": "https://wpnews.pro/news/agent-execution-runtime-engineered-to-run-millions-of-sandboxes.jsonld"}}