Cutting Idle Agent Costs by 90% with Agent Substrate

Agent Substrate, a new platform for running AI agents, reduces idle agent costs by 90% compared to traditional Kubernetes Pod deployments. By using an actor model with checkpoint/restore and gVisor, it enables efficient resource sharing across multiple agents per worker, significantly lowering hardware requirements. The project provides a demo and CLI tool for users to test the cost savings.

Cost is everything. In just about every agentic conversation, the three things that come up for enterprises implementing AI workloads are: and as AI continues to throw everyone for a loop when it comes to cost management e.g - Uber running out of the yearly token budget in one quarter , the ability to shrink resource like hardware usage will be crucial moving forward. In this blog post, you will learn how to cust costs by 90% using Agent Susbtrate in comparison to Agents running in k8s Deployments/Pods. Agents need a place to run. The "place to run" needs to be a platform that's easily managed, orchestrated, and has the ability to cluster resources. Resources like CPU, GPU, and memory need to be able to scale and expand. Without this, it's a matter of manually managing servers that Agents are running on and clients to interact with said server. That's why so many organizations choose Kubernetes to run Agentic. When running Agents per Pod, however, that can get costly very quick in terms of hardware GPU, CPU, memory and performance can your cluster scale up and down quickly based on resource needs when it comes to Agents coming up and going down per use? . The tests in this blog post show: And the comparison will be 50 always-on Pods in comparison to 50 Actors across 5-7 Workers Pods . If there are 50 Agents running per Pod and 50 Agents running per Worker with 5-10 Actors per Pod, you can already imagine the hardware resource savings that can be accomplished. Right now, the majority of organizations start off with the "one Agent per Pod" approach as that's the fastest way to show value and get up and running. For the future, however, Agents in Actors via Agent Substrate will be how organizations deploy when they care about efficiency, optimization, and managing cost. Let's dive in from a hands-on perspective. To follow along in a hands-on fashion, you will need: kubectl-ate installedYou can install Agent Substrate and kubectl-ate from the Agent Substrate repo. Within the Agent Substrate repo, you will see a file in the hack directory called ate-dev-env.sh.example . Make a copy of the file: cp hack/ate-dev-env.sh.example .ate-dev-env.sh Then, edit the file with your cluster and account information. For example, if you are using GCP and deploy a GKE cluster, the .ate-dev-env.sh will look like the below: PROJECT ID=<your-project-id PROJECT NUMBER=<your-project-number GCE REGION=<bucket-region CLUSTER LOCATION=<cluster-zone-or-region CLUSTER NAME=<your-gke-cluster BUCKET NAME=<your-snapshot-bucket KO DOCKER REPO=gcr.io/<your-project-id /ate-images KUBECTL CONTEXT=<your-kube-context Once you've filled in your .ate-dev-env.sh file, you can source it and install the counter demo which exists in the Agent Substrate repo. source .ate-dev-env.sh ./hack/install-ate.sh --deploy-demo-counter You can then install the kubectl-ate CLI to interact with Actors, Workers, and Templates. go install ./cmd/kubectl-ate export PATH="$ go env GOPATH /bin:$PATH" Verify that Substrate is up and operational: kubectl get workerpools.ate.dev counter -n ate-demo-counter kubectl get pods -n ate-demo-counter kubectl ate get workers If your GKE cluster is regional or has Worker Nodes spread across zones, pin the demo WorkerPool to one zone before creating benchmark actors. Agent Substrate uses checkpoint/restore , and gVisor restores can fail if a snapshot created on oneunderlying CPU platform is restored on a node with a different CPU feature set. Choose the zone where you want the counter workers to run: export SUBSTRATE WORKER ZONE=us-east1-d Then patch the counter WorkerPool to schedule Workers in that zone. kubectl patch workerpools.ate.dev counter \ -n ate-demo-counter \ --type=merge \ -p "{\"spec\":{\"template\":{\"nodeSelector\":{\"topology.kubernetes.io/zone\":\"${SUBSTRATE WORKER ZONE}\"}}}}" With the installation and configuration complete, you can now start setting up the benchmark environment and tests. There are several environment variables below. The ACTOR COUNT is the number of logical counter agents to test in both scenarios. BENCHMARK NAMESPACE | is the namespace for the always-on Kubernetes baseline workloads and the in-cluster benchmark client Pod. BASELINE PREFIX is the name prefix for Kubernetes baseline Deployments and Services. SUBSTRATE PREFIX is the name prefix for Substrate actors created by the benchmark. TEMPLATE REF is the Substrate actor template reference in <namespace /<name format. The counter demo creates ate-demo-counter/counter . SUBSTRATE ROUTER URL is the in-cluster URL for atenet-router ; benchmark client sends Substrate actor traffic through this service. BASELINE CPU REQUEST is the requests assigned to each always-on Kubernetes baseline Pod. Used to make baseline resource consumption explicit. BASELINE MEMORY REQUEST is the memory request assigned to each always-on Kubernetes baseline Pod and is used to make baseline resource consumption explicit. BASELINE RESULTS FILE is the Local TSV file for Kubernetes baseline latency results. SUBSTRATE RESULTS FILE and SUMMARY FILE are the files that contain the results. export ACTOR COUNT=50 export BENCHMARK NAMESPACE=cost-comparison export BASELINE PREFIX=k8s-counter export SUBSTRATE PREFIX=substrate-counter export TEMPLATE REF=ate-demo-counter/counter export SUBSTRATE ROUTER URL=http://atenet-router.ate-system.svc:80 export BASELINE CPU REQUEST=50m export BASELINE MEMORY REQUEST=64Mi export SUBSTRATE WORKER CPU REQUEST=50m export SUBSTRATE WORKER MEMORY REQUEST=64Mi export BASELINE RESULTS FILE=baseline-kubernetes-results.tsv export SUBSTRATE RESULTS FILE=substrate-results.tsv export SUMMARY FILE=cost-comparison-summary.txt ActorTemplate . This keeps the Kubernetes baseline on the same counter server image used by the Substrate demo. export COUNTER IMAGE=$ kubectl get actortemplates.ate.dev counter \ -n ate-demo-counter \ -o jsonpath='{.spec.containers 0 .image}' printf "Counter image: %s\n" "$COUNTER IMAGE" case "$COUNTER IMAGE" in ko:// printf "Counter image was not resolved: %s\n" "$COUNTER IMAGE" exit 1 ;; esac export WORKER REPLICAS=$ kubectl get workerpools.ate.dev counter \ -n ate-demo-counter \ -o jsonpath='{.spec.replicas}' printf "Logical agents: %s\nSubstrate workers: %s\n" \ "$ACTOR COUNT" "$WORKER REPLICAS" In this section, you will deploy the Kubernetes Pods that will be running the counter demo. kubectl create namespace "$BENCHMARK NAMESPACE" Deployment and Service object per logical counter Agent. for i in $ seq 1 "$ACTOR COUNT" ; do name=$ printf "%s-%03d" "$BASELINE PREFIX" "$i" kubectl apply -f - <<EOF apiVersion: apps/v1 kind: Deployment metadata: name: ${name} namespace: ${BENCHMARK NAMESPACE} labels: app.kubernetes.io/name: counter app.kubernetes.io/part-of: cost-comparison cost-comparison/model: always-on-kubernetes spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: counter app.kubernetes.io/instance: ${name} template: metadata: labels: app.kubernetes.io/name: counter app.kubernetes.io/instance: ${name} app.kubernetes.io/part-of: cost-comparison cost-comparison/model: always-on-kubernetes spec: containers: - name: counter image: ${COUNTER IMAGE} command: - /ko-app/counter ports: - containerPort: 80 resources: requests: cpu: ${BASELINE CPU REQUEST} memory: ${BASELINE MEMORY REQUEST} --- apiVersion: v1 kind: Service metadata: name: ${name} namespace: ${BENCHMARK NAMESPACE} labels: app.kubernetes.io/name: counter app.kubernetes.io/part-of: cost-comparison cost-comparison/model: always-on-kubernetes spec: selector: app.kubernetes.io/name: counter app.kubernetes.io/instance: ${name} ports: - name: http port: 80 targetPort: 80 EOF done kubectl get deployments -n "$BENCHMARK NAMESPACE" \ -l cost-comparison/model=always-on-kubernetes kubectl get pods -n "$BENCHMARK NAMESPACE" \ -l cost-comparison/model=always-on-kubernetes You'll see an output with 50 objects like in the example below: NAME READY STATUS RESTARTS AGE k8s-counter-001-9f9f44464-h526d 1/1 Running 0 128m k8s-counter-002-9678fb86f-bwpwm 1/1 Running 0 128m k8s-counter-003-54bccfd7db-k5wgz 1/1 Running 0 128m k8s-counter-004-5957785959-59k24 1/1 Running 0 128m k8s-counter-005-5df4559dd4-pgm4v 1/1 Running 0 128m k8s-counter-006-5597c6cd9-nj9mm 1/1 Running 0 128m k8s-counter-007-8d5c5bb74-j2n6p 1/1 Running 0 128m k8s-counter-008-ff9898c5f-hc7hw 1/1 Running 0 128m k8s-counter-009-77bc4cf8bd-rk2sq 1/1 Running 0 128m k8s-counter-010-578684f8c8-d7xg8 1/1 Running 0 128m k8s-counter-011-8697447c4f-5jz9p 1/1 Running 0 128m k8s-counter-012-5d7bc67f4d-hhhkh 1/1 Running 0 128m xxxxx xxxxx xxxxx xxxxx xxxxx In this configuration, you can create one Actor per logical counter Agent. for i in $ seq 1 "$ACTOR COUNT" ; do actor=$ printf "%s-%03d" "$SUBSTRATE PREFIX" "$i" kubectl ate create actor "$actor" --template "$TEMPLATE REF" || true done When you run the above, you will see 50 Actors deployed via kubectl ate get actors However, you will only see 7 Workers. That's the optimization and efficiency right there. Same exact deployment/configuration as the Kubernetes section above, except with Agent Substrate, you can run the same workloads in 7 Pods Workers insteadof 50. You’ll see 50 Agent Substrate Actors and a smaller set of Workers Pods . The Actors are logical workloads, but they are not actively running while they are STATUS SUSPENDED . By default, Actors are in a "suspended" state until they are used, which is why Actors are so great from an efficiency perspective. When traffic arrives for an Actor, Agent Substrate assigns that actor to an available Worker, resumes it, serves the request, and can suspend it again afterward. This is the efficiency model: many idle actors can exist without each requiring its own always-on Kubernetes Pod. The benchmark client runs inside the cluster so both paths avoid local port-forward overhead. benchmark-client = temporary in-cluster curl Pod. Why it exists: kubectl delete pod benchmark-client \ -n "$BENCHMARK NAMESPACE" \ --ignore-not-found kubectl run benchmark-client \ -n "$BENCHMARK NAMESPACE" \ --image=curlimages/curl:8.10.1 \ --restart=Never \ --command -- sleep 3600 kubectl wait --for=condition=Ready pod/benchmark-client \ -n "$BENCHMARK NAMESPACE" \ --timeout=2m Each baseline agent receives two requests: Because these are always-on Pods, both requests should be served by already running Kubernetes workloads. kubectl exec -n "$BENCHMARK NAMESPACE" benchmark-client -- sh -c ' set -eu actor count="$1" prefix="$2" namespace="$3" printf "agent\tfirst seconds\twarm seconds\n" for i in $ seq 1 "$actor count" ; do name=$ printf "%s-%03d" "$prefix" "$i" url="http://${name}.${namespace}.svc.cluster.local" first seconds=$ curl -sS -o /dev/null -w "%{time total}" -X POST "$url" warm seconds=$ curl -sS -o /dev/null -w "%{time total}" -X POST "$url" printf "%s\t%s\t%s\n" "$name" "$first seconds" "$warm seconds" done ' sh "$ACTOR COUNT" "$BASELINE PREFIX" "$BENCHMARK NAMESPACE" "$BASELINE RESULTS FILE" Inspect the baseline results: column -t -s $'\t' "$BASELINE RESULTS FILE" You'll see an output similar to the below for 50 counters. agent first seconds warm seconds k8s-counter-001 0.023404 0.003911 k8s-counter-002 0.023275 0.005233 k8s-counter-003 0.015850 0.003773 k8s-counter-004 0.017657 0.005033 k8s-counter-005 0.014946 0.004443 k8s-counter-006 0.015616 0.004212 k8s-counter-007 0.016875 0.004261 k8s-counter-008 0.014731 0.004317 k8s-counter-009 0.017053 0.004707 k8s-counter-010 0.013013 0.003273 k8s-counter-011 0.014281 0.004552 k8s-counter-012 0.018644 0.003734 xxxxx xxxxx Each Substrate actor receives two requests: After each actor is measured, the Actor is suspended so the worker can serve the next Actor. printf "actor\twake seconds\twarm seconds\n" "$SUBSTRATE RESULTS FILE" for i in $ seq 1 "$ACTOR COUNT" ; do actor=$ printf "%s-%03d" "$SUBSTRATE PREFIX" "$i" actor host="${actor}.actors.resources.substrate.ate.dev" result=$ kubectl exec -n "$BENCHMARK NAMESPACE" benchmark-client -- sh -c ' set -eu router url="$1" actor host="$2" wake seconds=$ curl -sS -o /dev/null -w "%{time total}" \ -X POST \ -H "Host: ${actor host}" \ "$router url" warm seconds=$ curl -sS -o /dev/null -w "%{time total}" \ -X POST \ -H "Host: ${actor host}" \ "$router url" printf "%s\t%s" "$wake seconds" "$warm seconds" ' sh "$SUBSTRATE ROUTER URL" "$actor host" printf "%s\t%s\n" "$actor" "$result" "$SUBSTRATE RESULTS FILE" kubectl ate suspend actor "$actor" /dev/null done Inspect the Substrate results: column -t -s $'\t' "$SUBSTRATE RESULTS FILE" Now it's time to measure the results and see if Substrate really saves resources and helps optimize workloads running in k8s. export BASELINE RUNNING PODS=$ kubectl get pods \ -n "$BENCHMARK NAMESPACE" \ -l cost-comparison/model=always-on-kubernetes \ --field-selector=status.phase=Running \ --no-headers | wc -l | tr -d ' ' export SUBSTRATE WORKLOAD PODS=$ kubectl get pods \ -n ate-demo-counter \ --field-selector=status.phase=Running \ --no-headers | wc -l | tr -d ' ' The results for k8s always on: printf "baseline cpu request per pod=%s\n" "$BASELINE CPU REQUEST" printf "baseline memory request per pod=%s\n" "$BASELINE MEMORY REQUEST" baseline cpu request per pod=50m baseline memory request per pod=64Mi The results for Substrate: baseline cpu request per pod=50m baseline memory request per pod=64Mi substrate worker cpu request per pod=50m substrate worker memory request per pod=64Mi baseline total cpu request millicores=2500 baseline total memory request mib=3200 substrate total cpu request millicores=250 substrate total memory request mib=320 The above shows the requested-capacity savings: Kubernetes baseline: 50 Pods 50m CPU / 64Mi = 2500m CPU / 3200Mi Substrate: 5 Pods 50m CPU / 64Mi = 250m CPU / 320Mi Which results in a 90% reduction You can also capture actual CPU and memory usage if metrics-server is installed: kubectl top pods -n "$BENCHMARK NAMESPACE" \ -l cost-comparison/model=always-on-kubernetes || true kubectl top pods -n ate-demo-counter || true Notice how many resources are saved with just a few Substrate deployments on the bottom vs k8s running workloads 50 pods, 50 agents . The Kubernetes baseline is running one always-on Pod per logical workload. With ACTOR COUNT=50 , that means 50 k8s-counter- Pods are running even when they are mostly idle. Each baseline Pod has explicit CPU and memory requests, so the always-on capacity grows linearly with the number of logical workloads. The Substrate side is running the same logical workload count as actors, but only the worker pool stays hot. In this run, the counter WorkerPool has 5 counter-deployment- Pods. Each worker Pod has explicit CPU and memory requests, so the requested always-on capacity grows with worker count, not Actor count. The main difference is the always-on footprint: Kubernetes baseline: 50 running workload PodsAgent Substrate: 5 running worker Pods for 50 logical actors Your result shows the core optimization: 50 idle logical workloads do not require50 always-on workload Pods when they run as Substrate actors.