The increase in cost of GPUs and AI is hitting an all-time high, so much so that it's hard to even get GPU usage approved within a public cloud environment. Because of that, the ability to splice/share GPUs across workloads that need GPUs (apps, Agents) is a necessity not only for cost savings, but for hardware/resource savings as GPUs aren't an unlimited resource (especially nowadays).
In this blog post, you'll learn how to implement CUDA (NVIDIA's parallel processing) inside an Agent Substrate Actor and see how many intermittent GPU workloads can take turns using a smaller pool of GPU-backed Substrate Workers.
💡How Agents work today (tool calls, planning, loops, etc.) are CPU-centric tasks. That means the majority of the time, you won't see an Agent that needs a GPU. However, there are a few edge cases (neural-network math that's local, the LLM is in-process where the GPU is for the LLM, but it lives inside the Agent process, a tool the Agent runs is a GPU job like local image/vid gen) where an Agent using a GPU makes sense. This blog, however, uses a standard HTTP app inside of the Actor.
To follow along with this blog post from a hands-on perspective, you will need:
PodCertificate
API needs the ability to be enabled on the k8s API server.KO_DOCKER_REPO
and BUCKET_NAME
from your Substrate env file (found in the repo from step 3 that you can edit).substrate/optimization/gpu-backed-actors
to see what you're deploying and you'll need it to build the images later.GPU splicing isn't new. With the NVIDIA Operator, we've been able to do that within Kubernetes for quite a while.
Example:
apiVersion: v1
kind: ConfigMap
metadata:
name: time-slicing-config
data:
any: |-
version: v1
flags:
migStrategy: none
sharing:
timeSlicing:
renameByDefault: false
failRequestsGreaterThanOne: false
resources:
- name: nvidia.com/gpu
replicas: 4
The above shows that the NVIDIA GPU is deployed on a Worker Node within your k8s cluster and up to four (4) Pods can request a "piece" of the one GPU.
You'd do that by putting a parameter like this within your Pods Manifest:
nvidia.com/gpu: 1
Instead of splicing per Pod, you can splice per Actor (an Agent or HTTP app runs inside an Actor), which saves hardware resources and therefore cost because multiple Actors can run in one Worker Pod.
💡As it stands right now, one Actor can run at a time within a Worker, which means the GPU is being used by said Actor.
In the next section, you'll see how to ensure an Actor can use a NVIDIA GPU.
A Substrate Actor can use a GPU the same way a Pod does, which is by putting nvidia.com/gpu
on the Worker/WorkerPool. There is no GPU field on ActorTemplate
. Instead, Substrate passes the assigned device into every container in the Actor.
source /path/to/substrate/.ate-dev-env.sh
export SUBSTRATE_DIR=/path/to/substrate
kubectl get pods -n ate-system
kubectl get sandboxconfig gvisor-default
kubectl get nodes -o json | jq -r \
'["NAME","GPU","GKE_ACCEL","PRODUCT"],
(.items[] | [
.metadata.name,
(.status.allocatable["nvidia.com/gpu"] // "-"),
(.metadata.labels["cloud.google.com/gke-accelerator"] // "-"),
(.metadata.labels["nvidia.com/gpu.product"] // "-")
]) | @tsv' | column -t
You'll see an output similar to the below:
In the next section, you'll implement an Actor with a usable NVIDIA GPU.
At the time of writing this, only gVisor (software-level isolation) based Actors can support GPUs.
CRD rule on WorkerPool:
nvidia.com/gpu is only supported when sandboxClass is 'gvisor'
The source of this is via the kubebuilder CEL marker on WorkerPoolSpec
.
Because of that, you will have to ensure that you have gVisor enabled on your cluster running Substrate.
💡The default sandboxClass
is gVisor, so if you didn't specify microVM during installation, you're good to go.
The images in this step are for the next step, but let's break down the "why" in terms of why we need them:
gcr.io/distroless/static-debian13
. It cannot exec nvidia-ctk, so it cannot inject a GPU into the sandbox.workload/
Which you can find here, is the Actors application (it's a Go app) that calls out to the GPU. It runs nvidia-smi inside the Actor as a child process.
export ATEOM_GPU_IMAGE=$(
cd "$SUBSTRATE_DIR" &&
KO_DOCKER_REPO="$KO_DOCKER_REPO" \
KO_DEFAULTPLATFORMS=linux/amd64 \
KO_DEFAULTBASEIMAGE=debian:stable-slim \
./hack/run-tool.sh ko build ./cmd/ateom-gvisor
)
echo "$ATEOM_GPU_IMAGE"
ActorTemplate
can use, which is used as a golden image to deploy an Actor with GPU needs.
cd agentic-demo-repo/substrate/optimization/gpu-backed-actors/workload
docker buildx build \
--platform linux/amd64 \
--push \
--provenance=false \
--metadata-file /tmp/gpu-agent.json \
--tag "${KO_DOCKER_REPO}/gpu-actor-workload:gpu-agent" \
workload/
export GPU_WORKLOAD_IMAGE="${KO_DOCKER_REPO}/gpu-actor-workload@$(jq -er '."containerimage.digest"' /tmp/gpu-agent.json)"
echo "$GPU_WORKLOAD_IMAGE"
ActorTemplate
.
export SNAPSHOT_LOCATION="gs://${BUCKET_NAME}/ate-demo-gpu/"
With the proper images built for both the WorkerPool
to have the ability to inject GPUs and the GPU-based image so the Actor uses an image that requires a GPU for the workload to run, let's deploy the resources.
WorkerPool
nvidia.com/gpu: "1"
in requests and limits is what injects the GPU into the sandbox.```
kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
name: ate-demo-gpu
labels:
apiVersion: ate.dev/v1alpha1
kind: WorkerPool
metadata:
name: gpu-workers
namespace: ate-demo-gpu
labels:
workload: gpu
spec:
replicas: 1
sandboxClass: gvisor
workerImage: ${ATEOM_GPU_IMAGE}
template:
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
resources:
requests:
cpu: 500m
memory: 2Gi
nvidia.com/gpu: "1"
limits:
cpu: "2"
memory: 4Gi
nvidia.com/gpu: "1"
EOF
`
ActorTemplate
, which is the golden image/template that an Actor uses as a blueprint when it's created. Notice that it's using the GPU workload image.💡There is no GPU field on the template. The GPU comes from the WorkerPool
.
kubectl apply -f - <<EOF
apiVersion: ate.dev/v1alpha1
kind: ActorTemplate
metadata:
name: gpu-app
namespace: ate-demo-gpu
spec:
sandboxClass: gvisor
workerSelector:
matchLabels:
workload: gpu
containers:
With the `WorkerPool`
where the Actor runs and the Actor's golden image/template/blueprint created, you can now create the Actor.
kubectl ate create atespace gpu-demo
kubectl ate create actor gpu-1 --atespace gpu-demo --template ate-demo-gpu/gpu-agent
kubectl ate resume actor gpu-1 --atespace gpu-demo --boot
kubectl ate logs actors gpu-1 --atespace gpu-demo
You can test the Actor to ensure that it's working as expected:
kubectl -n ate-system port-forward svc/atenet-router 8000:80
curl -sS \
-H 'Host: gpu-1.gpu-demo.actors.resources.substrate.ate.dev' \
`
"No running processes found" is expected. GET /gpu
runs nvidia-smi
as a child and then exits. The Processes table only lists jobs that currently hold GPU memory (a CUDA context).