GPUs For Actors In Agent Substrate

A developer demonstrates how to enable NVIDIA GPU support for actors in the Substrate agent runtime, allowing multiple actors to share a smaller pool of GPU-backed workers via time-slicing. The approach uses the NVIDIA Operator and gVisor sandbox class to allocate nvidia.com/gpu resources per actor, reducing hardware and cost overhead for intermittent GPU workloads.

The increase in cost of GPUs and AI is hitting an all-time high, so much so that it's hard to even get GPU usage approved within a public cloud environment. Because of that, the ability to splice/share GPUs across workloads that need GPUs apps, Agents is a necessity not only for cost savings, but for hardware/resource savings as GPUs aren't an unlimited resource especially nowadays . In this blog post, you'll learn how to implement CUDA NVIDIA's parallel processing inside an Agent Substrate Actor and see how many intermittent GPU workloads can take turns using a smaller pool of GPU-backed Substrate Workers. 💡How Agents work today tool calls, planning, loops, etc. are CPU-centric tasks. That means the majority of the time, you won't see an Agent that needs a GPU. However, there are a few edge cases neural-network math that's local, the LLM is in-process where the GPU is for the LLM, but it lives inside the Agent process, a tool the Agent runs is a GPU job like local image/vid gen where an Agent using a GPU makes sense. This blog, however, uses a standard HTTP app inside of the Actor. To follow along with this blog post from a hands-on perspective, you will need: PodCertificate API needs the ability to be enabled on the k8s API server. KO DOCKER REPO and BUCKET NAME from your Substrate env file found in the repo from step 3 that you can edit . substrate/optimization/gpu-backed-actors to see what you're deploying and you'll need it to build the images later.GPU splicing isn't new. With the NVIDIA Operator, we've been able to do that within Kubernetes for quite a while. Example: apiVersion: v1 kind: ConfigMap metadata: name: time-slicing-config data: any: |- version: v1 flags: migStrategy: none sharing: timeSlicing: renameByDefault: false failRequestsGreaterThanOne: false resources: - name: nvidia.com/gpu replicas: 4 The above shows that the NVIDIA GPU is deployed on a Worker Node within your k8s cluster and up to four 4 Pods can request a "piece" of the one GPU. You'd do that by putting a parameter like this within your Pods Manifest: nvidia.com/gpu: 1 Instead of splicing per Pod, you can splice per Actor an Agent or HTTP app runs inside an Actor , which saves hardware resources and therefore cost because multiple Actors can run in one Worker Pod. 💡As it stands right now, one Actor can run at a time within a Worker, which means the GPU is being used by said Actor. In the next section, you'll see how to ensure an Actor can use a NVIDIA GPU. A Substrate Actor can use a GPU the same way a Pod does, which is by putting nvidia.com/gpu on the Worker/WorkerPool. There is no GPU field on ActorTemplate . Instead, Substrate passes the assigned device into every container in the Actor. source /path/to/substrate/.ate-dev-env.sh export SUBSTRATE DIR=/path/to/substrate kubectl get pods -n ate-system kubectl get sandboxconfig gvisor-default kubectl get nodes -o json | jq -r \ ' "NAME","GPU","GKE ACCEL","PRODUCT" , .items | .metadata.name, .status.allocatable "nvidia.com/gpu" // "-" , .metadata.labels "cloud.google.com/gke-accelerator" // "-" , .metadata.labels "nvidia.com/gpu.product" // "-" | @tsv' | column -t You'll see an output similar to the below: In the next section, you'll implement an Actor with a usable NVIDIA GPU. At the time of writing this, only gVisor software-level isolation based Actors can support GPUs. CRD rule on WorkerPool: nvidia.com/gpu is only supported when sandboxClass is 'gvisor' The source of this is via the kubebuilder CEL marker on WorkerPoolSpec . Because of that, you will have to ensure that you have gVisor enabled on your cluster running Substrate. 💡The default sandboxClass is gVisor, so if you didn't specify microVM during installation, you're good to go. The images in this step are for the next step, but let's break down the "why" in terms of why we need them: gcr.io/distroless/static-debian13 . It cannot exec nvidia-ctk, so it cannot inject a GPU into the sandbox. workload/ Which you can find here, is the Actors application it's a Go app that calls out to the GPU. It runs nvidia-smi inside the Actor as a child process. export ATEOM GPU IMAGE=$ cd "$SUBSTRATE DIR" && KO DOCKER REPO="$KO DOCKER REPO" \ KO DEFAULTPLATFORMS=linux/amd64 \ KO DEFAULTBASEIMAGE=debian:stable-slim \ ./hack/run-tool.sh ko build ./cmd/ateom-gvisor echo "$ATEOM GPU IMAGE" ActorTemplate can use, which is used as a golden image to deploy an Actor with GPU needs. cd agentic-demo-repo/substrate/optimization/gpu-backed-actors/workload docker buildx build \ --platform linux/amd64 \ --push \ --provenance=false \ --metadata-file /tmp/gpu-agent.json \ --tag "${KO DOCKER REPO}/gpu-actor-workload:gpu-agent" \ workload/ export GPU WORKLOAD IMAGE="${KO DOCKER REPO}/gpu-actor-workload@$ jq -er '."containerimage.digest"' /tmp/gpu-agent.json " echo "$GPU WORKLOAD IMAGE" ActorTemplate . export SNAPSHOT LOCATION="gs://${BUCKET NAME}/ate-demo-gpu/" With the proper images built for both the WorkerPool to have the ability to inject GPUs and the GPU-based image so the Actor uses an image that requires a GPU for the workload to run, let's deploy the resources. WorkerPool nvidia.com/gpu: "1" in requests and limits is what injects the GPU into the sandbox. kubectl apply -f - <<EOF apiVersion: v1 kind: Namespace metadata: name: ate-demo-gpu labels: apiVersion: ate.dev/v1alpha1 kind: WorkerPool metadata: name: gpu-workers namespace: ate-demo-gpu labels: workload: gpu spec: replicas: 1 sandboxClass: gvisor workerImage: ${ATEOM GPU IMAGE} template: tolerations: - key: nvidia.com/gpu operator: Exists effect: NoSchedule resources: requests: cpu: 500m memory: 2Gi nvidia.com/gpu: "1" limits: cpu: "2" memory: 4Gi nvidia.com/gpu: "1" EOF ActorTemplate , which is the golden image/template that an Actor uses as a blueprint when it's created. Notice that it's using the GPU workload image.💡There is no GPU field on the template. The GPU comes from the WorkerPool . kubectl apply -f - <<EOF apiVersion: ate.dev/v1alpha1 kind: ActorTemplate metadata: name: gpu-app namespace: ate-demo-gpu spec: sandboxClass: gvisor workerSelector: matchLabels: workload: gpu containers: With the WorkerPool where the Actor runs and the Actor's golden image/template/blueprint created, you can now create the Actor. kubectl ate create atespace gpu-demo kubectl ate create actor gpu-1 --atespace gpu-demo --template ate-demo-gpu/gpu-agent kubectl ate resume actor gpu-1 --atespace gpu-demo --boot kubectl ate logs actors gpu-1 --atespace gpu-demo You can test the Actor to ensure that it's working as expected: kubectl -n ate-system port-forward svc/atenet-router 8000:80 curl -sS \ -H 'Host: gpu-1.gpu-demo.actors.resources.substrate.ate.dev' \ http://localhost:8000/gpu http://localhost:8000/gpu "No running processes found" is expected. GET /gpu runs nvidia-smi as a child and then exits. The Processes table only lists jobs that currently hold GPU memory a CUDA context .