{"slug": "gpus-for-actors-in-agent-substrate", "title": "GPUs For Actors In Agent Substrate", "summary": "A developer demonstrates how to enable NVIDIA GPU support for actors in the Substrate agent runtime, allowing multiple actors to share a smaller pool of GPU-backed workers via time-slicing. The approach uses the NVIDIA Operator and gVisor sandbox class to allocate nvidia.com/gpu resources per actor, reducing hardware and cost overhead for intermittent GPU workloads.", "body_md": "The increase in cost of GPUs and AI is hitting an all-time high, so much so that it's hard to even get GPU usage approved within a public cloud environment. Because of that, the ability to splice/share GPUs across workloads that need GPUs (apps, Agents) is a necessity not only for cost savings, but for hardware/resource savings as GPUs aren't an unlimited resource (especially nowadays).\n\nIn this blog post, you'll learn how to implement CUDA (NVIDIA's parallel processing) inside an Agent Substrate Actor and see how many intermittent GPU workloads can take turns using a smaller pool of GPU-backed Substrate Workers.\n\n💡How Agents work today (tool calls, planning, loops, etc.) are CPU-centric tasks. That means the majority of the time, you won't see an Agent that needs a GPU. However, there are a few edge cases (neural-network math that's local, the LLM is in-process where the GPU is for the LLM, but it lives inside the Agent process, a tool the Agent runs is a GPU job like local image/vid gen) where an Agent using a GPU makes sense. This blog, however, uses a standard HTTP app inside of the Actor.\n\nTo follow along with this blog post from a hands-on perspective, you will need:\n\n`PodCertificate`\n\nAPI needs the ability to be enabled on the k8s API server.`KO_DOCKER_REPO`\n\nand `BUCKET_NAME`\n\nfrom your Substrate env file (found in the repo from step 3 that you can edit).`substrate/optimization/gpu-backed-actors`\n\nto see what you're deploying and you'll need it to build the images later.GPU splicing isn't new. With the NVIDIA Operator, we've been able to do that within Kubernetes for quite a while.\n\nExample:\n\n```\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: time-slicing-config\ndata:\n any: |-\n version: v1\n flags:\n migStrategy: none\n sharing:\n timeSlicing:\n renameByDefault: false\n failRequestsGreaterThanOne: false\n resources:\n - name: nvidia.com/gpu\n replicas: 4\n```\n\nThe above shows that the NVIDIA GPU is deployed on a Worker Node within your k8s cluster and up to four (4) Pods can request a \"piece\" of the one GPU.\n\nYou'd do that by putting a parameter like this within your Pods Manifest:\n\n```\nnvidia.com/gpu: 1\n```\n\nInstead of splicing per Pod, you can splice per Actor (an Agent or HTTP app runs inside an Actor), which saves hardware resources and therefore cost because multiple Actors can run in one Worker Pod.\n\n💡As it stands right now, one Actor can run at a time within a Worker, which means the GPU is being used by said Actor.\n\nIn the next section, you'll see how to ensure an Actor can use a NVIDIA GPU.\n\nA Substrate Actor can use a GPU the same way a Pod does, which is by putting `nvidia.com/gpu`\n\non the Worker/WorkerPool. There is no GPU field on `ActorTemplate`\n\n. Instead, Substrate passes the assigned device into every container in the Actor.\n\n```\nsource /path/to/substrate/.ate-dev-env.sh\nexport SUBSTRATE_DIR=/path/to/substrate\n\nkubectl get pods -n ate-system\nkubectl get sandboxconfig gvisor-default\nkubectl get nodes -o json | jq -r \\\n '[\"NAME\",\"GPU\",\"GKE_ACCEL\",\"PRODUCT\"],\n (.items[] | [\n .metadata.name,\n (.status.allocatable[\"nvidia.com/gpu\"] // \"-\"),\n (.metadata.labels[\"cloud.google.com/gke-accelerator\"] // \"-\"),\n (.metadata.labels[\"nvidia.com/gpu.product\"] // \"-\")\n ]) | @tsv' | column -t\n```\n\nYou'll see an output similar to the below:\n\nIn the next section, you'll implement an Actor with a usable NVIDIA GPU.\n\nAt the time of writing this, only gVisor (software-level isolation) based Actors can support GPUs.\n\n```\nCRD rule on WorkerPool:\n\nnvidia.com/gpu is only supported when sandboxClass is 'gvisor'\n```\n\nThe source of this is via the kubebuilder CEL marker on `WorkerPoolSpec`\n\n.\n\nBecause of that, you will have to ensure that you have gVisor enabled on your cluster running Substrate.\n\n💡The default `sandboxClass`\n\nis gVisor, so if you didn't specify microVM during installation, you're good to go.\n\nThe images in this step are for the next step, but let's break down the \"why\" in terms of why we need them:\n\n`gcr.io/distroless/static-debian13`\n\n. It cannot exec nvidia-ctk, so it cannot inject a GPU into the sandbox.`workload/`\n\nWhich you can find here, is the Actors application (it's a Go app) that calls out to the GPU. It runs nvidia-smi inside the Actor as a child process.\n\n```\nexport ATEOM_GPU_IMAGE=$(\n cd \"$SUBSTRATE_DIR\" &&\n KO_DOCKER_REPO=\"$KO_DOCKER_REPO\" \\\n KO_DEFAULTPLATFORMS=linux/amd64 \\\n KO_DEFAULTBASEIMAGE=debian:stable-slim \\\n ./hack/run-tool.sh ko build ./cmd/ateom-gvisor\n)\necho \"$ATEOM_GPU_IMAGE\"\n```\n\n`ActorTemplate`\n\ncan use, which is used as a golden image to deploy an Actor with GPU needs.\n\n```\ncd agentic-demo-repo/substrate/optimization/gpu-backed-actors/workload\n\ndocker buildx build \\\n --platform linux/amd64 \\\n --push \\\n --provenance=false \\\n --metadata-file /tmp/gpu-agent.json \\\n --tag \"${KO_DOCKER_REPO}/gpu-actor-workload:gpu-agent\" \\\n workload/\n\nexport GPU_WORKLOAD_IMAGE=\"${KO_DOCKER_REPO}/gpu-actor-workload@$(jq -er '.\"containerimage.digest\"' /tmp/gpu-agent.json)\"\necho \"$GPU_WORKLOAD_IMAGE\"\n```\n\n`ActorTemplate`\n\n.\n\n```\nexport SNAPSHOT_LOCATION=\"gs://${BUCKET_NAME}/ate-demo-gpu/\"\n```\n\nWith the proper images built for both the `WorkerPool`\n\nto have the ability to inject GPUs and the GPU-based image so the Actor uses an image that requires a GPU for the workload to run, let's deploy the resources.\n\n`WorkerPool`\n\n`nvidia.com/gpu: \"1\"`\n\nin requests and limits is what injects the GPU into the sandbox.```\n\nkubectl apply -f - <