{"slug": "memory-enforced-gpu-sharing-in-kubernetes-with-nvidia-cuda-13-4", "title": "Memory-Enforced GPU Sharing in Kubernetes with NVIDIA CUDA 13.4", "summary": "NVIDIA's CUDA 13.4 adds a memory-enforcement boundary to Multi-Process Service, letting each workload see and be capped at a fraction of GPU memory under a new time-sharing mode alongside the existing sm-sharing compute mode. A new open-source Kubernetes operator, gpu-fractioning, hosted under the KAI Scheduler organization on GitHub, runs alongside the NVIDIA GPU Operator to set up MPS on every GPU node, apply each container's memory limits and compute mode from Pod annotations, and measure memory and compute use per fraction rather than per GPU. KAI Scheduler is the first scheduler integrated with gpu-fractioning, offering a GPU-sharing mode called NvFractions starting with v0.18.0.", "body_md": "Many AI workloads need only a fraction of a GPU. Giving each one an entire device can leave capacity idle; sharing GPUs lets platform teams serve more workloads with the infrastructure they already have.\n\nNVIDIA GPUs already support several approaches to sharing. Multi-instance GPU (MIG) splits a GPU into hardware-isolated instances that are transparent to the application, time-slicing lets several processes take turns without memory enforcement, and [Multi-Process Service](https://docs.nvidia.com/deploy/mps/latest/index.html) (MPS) lets them run concurrently in separate address spaces (sm-sharing).\n\nCUDA 13.4 now enforces a memory boundary that is transparent to the application with a new time-sharing mode, alongside the compute sharing mode (sm-sharing) that already existed in MPS.\n\nHere is what the [CUDA release](https://developer.nvidia.com/cuda-downloads) brings:\n\n- **An exact memory amount:** A workload asks for a memory size and CUDA holds it there. An allocation above that size fails, just as it would on a smaller GPU.\n- **A range:** A workload gets a guaranteed minimum plus a maximum it can grow into when memory is free.\n- **A compute mode:** Each workload can choose to use its compute in an sm-sharing mode rather than using the default time-sharing mode.\n- **A transparent memory limit:** Each workload sees only a fraction of the GPU memory.\n\n## The missing piece: Getting there from Kubernetes\n\nCUDA provides the memory enforcement capability, but a Kubernetes cluster can’t use it on its own. On every GPU node, something has to:\n\n- Run and supervise the daemons that these features depend on\n- Apply each Pod container’s memory limits and compute mode before the process starts\n- Report what each fraction actually uses, rather than what the whole GPU uses\n\nThis work belongs on the node, independent of whichever scheduler placed the workload.\n\n## GPU fractioning: An open-source Kubernetes operator\n\nA new open-source Kubernetes operator, [`gpu-fractioning`](https://github.com/kai-scheduler/gpu-fractioning), is  hosted under the [KAI Scheduler](https://github.com/kai-scheduler/KAI-scheduler#readme) organization on GitHub. It runs next to the [NVIDIA GPU Operator,](https://github.com/NVIDIA/gpu-operator#readme) which provides the driver and the container runtime setup.\n\nGPU Fractioning Operator handles three jobs:\n\n- It sets up and runs MPS on every GPU node.\n- It applies each workload’s memory limits and compute mode.\n- It measures memory and compute use per fraction rather than per GPU.\n\n### How schedulers pass GPU allocations to the operator\n\nThe developers write the resource and mode annotations in the pod YAML. Then the scheduler selects the GPUs for a workload and records its memory allocation and compute-sharing mode in per-container Pod annotations. GPU Fractioning Operator reads those annotations and configures the assigned GPUs to apply the workload’s memory limits and compute mode:\n\n| Annotation | Meaning | \n|---|---|\n| `nvidia.com/container.<name>.gpu-memory.request` | Guaranteed GPU memory | \n| `nvidia.com/container.<name>.gpu-memory.limit` | Maximum the container may grow into | \n| `nvidia.com/container.<name>.gpu-compute.mode` | time-slicing (default) or sm-sharing | \n| `nvidia.com/container.<name>.gpus.devices` | Which GPUs the container was assigned | \n\n*Table 1: Per-container Pod annotations that GPU Fractioning Operator reads to enforce a GPU fraction*\n\nAny Kubernetes scheduler that can write these annotations can enforce GPU fractions with CUDA. The scheduler still has its own job: choosing a GPU with enough free memory and never mixing compute modes on one GPU.\n\n## Using GPU Fractioning with KAI Scheduler\n\nKAI Scheduler is the first scheduler integrated with `gpu-fractioning`. Starting with v0.18.0, KAI Scheduler has a GPU-sharing mode called [`NvFractions`](https://github.com/kai-scheduler/KAI-Scheduler/tree/main/docs/gpu-sharing/nv-fraction). KAI Scheduler picks which slice of which GPU each workload gets and writes the annotations (see Table 1, above), and `gpu-fractioning` enforces them on the node. \n\nKAI Scheduler’s other sharing modes stay available, and administrators choose one per cluster. It can be enabled when KAI is installed:\n\n```\nhelm upgrade -i kai-scheduler oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler \\\n  -n kai-scheduler --create-namespace \\\n  --set global.nvFractions.set=true\n```\n\nThis also installs `gpu-fractioning` as a dependency. From then on, you describe a fractional workload with Pod annotations. \n\nThere are four important annotations that answer four questions:\n\n1. How much memory is needed \n\n2. How far the workload may grow\n\n3. How compute is shared\n\n4. How many GPUs it spans\n\n### How much memory: The request\n\nThere are two ways to request memory:\n\n**1. As an absolute size:** Use the per-container NVIDIA annotation and standard Kubernetes units:\n\nThe container name (`app`) is part of the key, so it always targets the right container.**2. As a portion of a GPU by using KAI Scheduler’s existing annotation:**\n\n```\ngpu-fraction: \"0.5\"\n```\n\n### How far it may grow: The limit\n\nThis ability is new in KAI Scheduler. A workload can declare a guaranteed minimum (the request) and a maximum it may burst into when memory is free. Which limit annotation you use depends on how you wrote the request:\n\n| Request Type | Annotation | Example | \n|---|---|---|\n| Absolute Size | nvidia.com/container.<c>.gpu-memory.limit | “80Gi” | \n| A Portion | kai.scheduler/container.<c>.gpu-memory.portion.limit | “0.8” | \n\n*Table 2*\n\n**:** Which limit annotation to use for each type of memory request\nThe limit is optional. If it is omitted, the limit equals the request and the workload is capped at exactly what it asked for. A portion limit must be greater than the `gpu-fraction` and less than 1.\n\n### How compute is shared: Time sharing or sm-sharing\n\nCompute can now be divided. Call n the number of workloads sharing a GPU: either way each gets roughly 1/n of the compute, and what differs is whether they take turns or run together.\n\n- **Time-slicing is serial:** Workloads take turns on the whole GPU, and when every one of them uses its full slice each gets about 1/n of the time. Time left unused by one workload is shared equally between the others. This is the default. Each workload runs in its own error domain, i.e. full error isolation.\n- **SM-sharing is parallel:** The GPU’s streaming multiprocessors (SMs) are split between workloads and all run at once, continuously. Nothing is carved out in advance. The GPU hardware allocates and releases SMs as workloads need them. When every workload is busy the split settles at roughly 1/n; when one goes quiet its SMs go to the others, and a single active workload can use them all. In this mode, workloads share a single error domain.\n\nCompute mode is now explicit and can be set per container:\n\n```\nnvidia.com/container.app.gpu-compute.mode: \"sm-sharing\"   # or \"time-slicing\"\n```\n\nA shared GPU runs one compute mode at a time, so KAI Scheduler never places a time-slicing workload and an sm-sharing workload on the same GPU. KAI Scheduler tracks this per GPU, not per node, so one node can host both modes on different GPUs.\n\n### How many GPUs are utilized\n\nTo get fractional capacity on several devices, add:\n\n```\ngpu-fraction-num-devices: \"4\"\n```\n\nThe request applies to each device. A 40Gi request with four devices gets 40Gi on each of four GPUs.\n\n### Putting it together: Examples of GPU fractioning\n\nAn absolute size with room to burst on four GPUs with SM-sharing:\n\n```\napiVersion: v1\nkind: Pod\nmetadata:\n  name: inference\n  labels:\n    kai.scheduler/queue: default-queue\n  annotations:\n    nvidia.com/container.app.gpu-memory.request: \"40Gi\"\n    nvidia.com/container.app.gpu-memory.limit: \"60Gi\"       # optional\n    nvidia.com/container.app.gpu-compute.mode: \"sm-sharing\" # optional, default time-slicing\n    gpu-fraction-num-devices: \"4\"                           # optional, default 1\nspec:\n  schedulerName: kai-scheduler\n  containers:\n    - name: app\n      image: nvcr.io/nvidia/cuda:13.0.2-base-ubuntu24.04\n```\n\nThis is an inference server split across four GPUs that needs a guaranteed 40Gi on each GPU, can use up to 60Gi when neighbors are idle, and runs alongside other workloads instead of waiting its turn. SM-sharing puts the workloads in one error domain, so use it for workloads you trust to run together.\n\nHalf a GPU that may grow to 80% on two GPUs with time slicing (default):\n\n```\nmetadata:\n  annotations:\n    gpu-fraction: \"0.5\"\n    gpu-fraction.limit: \"0.8\"  # optional\n    nvidia.com/container.app.gpu-compute.mode: \"time-slicing\"    # optional\n    gpu-fraction-num-devices: \"2\"                                # optional\nspec:\n  schedulerName: kai-scheduler\n  containers:\n    - name: app\n```\n\nThis is an example for a development or batch job, such as notebooks or small fine-tuning runs, that need a guaranteed half of each of two GPUs and can grow to 80% when memory is free. Time-slicing gives each workload its own error domain, so a crash in one job doesn’t affect its neighbors. Running `nvidia-smi` in the container should show two GPUs, each with this workload’s fractional memory share.\n\n## Getting started\n\nKAI Scheduler and the [GPU Fractioning](https://github.com/kai-scheduler/gpu-fractioning) Operator are both on GitHub under [the KAI Scheduler organization](https://github.com/kai-scheduler), with installation and configuration instructions for all three sharing mechanisms (See [admin](https://github.com/kai-scheduler/KAI-Scheduler/tree/main/docs/gpu-sharing#admins) and [user](https://github.com/kai-scheduler/KAI-Scheduler/tree/main/docs/gpu-sharing#users) guides). Join [the Slack channel](https://github.com/kai-scheduler/KAI-Scheduler#slack) or [bi-weekly community meetings](https://github.com/kai-scheduler/KAI-Scheduler#bi-weekly-community-call) to ask your questions and contribute.\n\nFor more information about CUDA 13.4, [refer to the announcement.](https://developer.nvidia.com/blog/cuda-toolkit-13-4-adds-windows-on-arm-support-and-greater-control-over-shared-gpus/)\n\n**You can also get involved during** [KubeCon + CloudNativeCon NA 2026](https://www.nvidia.com/en-us/events/kubecon-and-cloudnative-con-north-america/) **in Salt Lake City, Utah**, Nov. 9-12.", "url": "https://wpnews.pro/news/memory-enforced-gpu-sharing-in-kubernetes-with-nvidia-cuda-13-4", "canonical_source": "https://developer.nvidia.com/blog/memory-enforced-gpu-sharing-in-kubernetes-with-nvidia-cuda-13-4/", "published_at": "2026-10-27 20:50:39+00:00", "updated_at": "2026-10-27 21:14:29.939624+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "developer-tools", "mlops"], "entities": ["NVIDIA", "CUDA 13.4", "Multi-Process Service", "gpu-fractioning", "KAI Scheduler", "NVIDIA GPU Operator", "NvFractions", "Kubernetes"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/memory-enforced-gpu-sharing-in-kubernetes-with-nvidia-cuda-13-4", "markdown": "https://wpnews.pro/news/memory-enforced-gpu-sharing-in-kubernetes-with-nvidia-cuda-13-4.md", "text": "https://wpnews.pro/news/memory-enforced-gpu-sharing-in-kubernetes-with-nvidia-cuda-13-4.txt", "jsonld": "https://wpnews.pro/news/memory-enforced-gpu-sharing-in-kubernetes-with-nvidia-cuda-13-4.jsonld"}}