SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints

Amazon SageMaker Inference now supports NVIDIA Multi-Instance GPU (MIG) partitioning on inference components, letting customers split a physical GPU into as many as seven hardware-isolated instances and host more models than an instance has accelerators. The feature addresses two constraints: an ml.p4d.24xlarge holds only eight models because it has eight A100s, and deployments previously had to choose a whole GPU from an older or newer generation rather than a fraction of one. Partitioning is limited to inference-component endpoints; single-model endpoints, serverless endpoints and heterogeneous instance pools cannot be partitioned.

Artificial Intelligence https://aws.amazon.com/blogs/machine-learning/ SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints When you deploy a model to a GPU endpoint, the smallest unit of GPU you can request is one whole accelerator. This constraint can cost you twice. First, you can’t host more models on an instance than it has accelerators. An ml.p4d.24xlarge has eight A100s, so it holds eight models, whether each one needs 40 GB or 14 GB. Second, and less obvious, it makes every deployment come down to a choice of which full GPU to deploy to. As your concurrency and prompt lengths grow, a full GPU on an older instance stops being enough, but a full GPU on a newer one is more than your model can use. You end up either under-served or paying for capability you never reach. Today, Amazon SageMaker Inference supports GPU partitioning, which uses NVIDIA Multi-Instance GPU https://docs.nvidia.com/datacenter/tesla/mig-user-guide/latest/ MIG to divide a physical GPU into independent partitions. Each partition receives a fixed share of the GPU’s compute units and a dedicated block of its memory, with isolation enforced in hardware. That changes both constraints: you can place a different model in each partition, and you can choose a fraction of a newer accelerator instead of a whole one of either generation. This post covers both. The first part is the direct benefit: deploy more models than you have GPUs, with isolation between them. The second is the one customers ask about when the question turns to cost: how the granularity of partitioning helps you pick the right-sized unit, and what we measured when we did. One thing to know before you read further: GPU partitioning works on endpoints that use inference components https://aws.amazon.com/blogs/machine-learning/reduce-model-deployment-costs-by-50-on-average-using-sagemakers-latest-features/ , the construct you use to deploy several models to one endpoint and set the compute each model gets. Partitioning builds directly on it. A partition is a resource a component asks for, in the same place it already asks for CPU and memory. If you haven’t used inference components, this post https://aws.amazon.com/blogs/machine-learning/reduce-model-deployment-costs-by-50-on-average-using-sagemakers-latest-features/ introduces them. Single-model endpoints, serverless endpoints, and heterogeneous instance pools cannot be partitioned. What customers do today, and why each option falls short If you have more models than accelerators, you have had three choices. Run fewer, larger endpoints and batch harder – This raises utilization, but only when the models are interchangeable. A portfolio of fine-tuned variants or per-tenant models is not. Use multi-model endpoints – These load and unload models from a shared GPU on demand, which fits sparse, bursty traffic. But models share the device without isolation, so a heavy request for one model affects the others. A model that isn’t loaded also pays a load penalty on first use. Move to self-managed Kubernetes – This is where customers who need real GPU partitioning have gone. MIG works there, but you take on driver versioning, partition configuration, a device plugin, scheduling, and keeping it all consistent across a fleet. That’s a lot of work for a hardware capability your instances already have. Why MIG NVIDIA Multi-Instance GPU MIG splits one GPU into as many as seven instances, depending on the GPU. Each instance has its own compute, cache, memory and memory bandwidth. Because the split is in hardware rather than software, you get guarantees that other sharing methods can’t give: 1. Predictable latency – Each partition has dedicated memory bandwidth, so one model’s traffic spike doesn’t show up in another model’s p99. 2. Fault isolation – A crash or memory error stays inside its partition instead of taking down every model on the GPU. 3. Right-sized capacity – Small and mid-sized models get a slice that fits them instead of a whole GPU. You can mix slice sizes on one device. 4. No cold starts – Every model keeps its own partition and stays loaded, with no load penalty and no eviction. 5. Tenant isolation – Memory protection between partitions is enforced in hardware, which makes security and compliance reviews simpler. 6. Reliable planning and chargeback – A partition performs the same way regardless of its neighbors, so load tests predict production. GPU cost also maps cleanly to a model, team or tenant. Partitioning has boundaries worth knowing up front. The maximum number of partitions depends on the accelerator and never exceeds seven. MIG itself only allows a layout to change while its partitions are idle, and on SageMaker AI that means choosing a layout when you create the endpoint. A model that needs more than one GPU still needs whole devices. Within those boundaries, you get the isolation of separate GPUs without dedicating one to each model. Partitioning without the platform work Until now, getting MIG meant running the Kubernetes stack yourself. GPU partitioning on Amazon SageMaker AI removes that step: you pick a partition profile for your model, and SageMaker AI handles GPU configuration, placement and scaling. You get hardware-isolated GPU sharing on a managed endpoint, without having to run the platform underneath it. Salesforce has been part of this work from the start, helping define the requirements and validating the initial implementation. “NVIDIA MIG partitions a single GPU into multiple isolated instances, each with dedicated compute and memory resources. Salesforce sees this as an important capability because many inference workloads do not require an entire GPU. MIG would allow us to consolidate workloads safely, significantly improve GPU utilization, reduce inference costs, and maintain predictable performance and strong isolation. This capability would help us scale AI Inference more efficiently across different use cases. Salesforce has helped define requirements, validated the initial solution. We are now preparing to validate the solution with representative production workload and demonstrate strong customer demand with AWS. We believe MIG support would deliver meaningful technical and business value for enterprise customers.” — Richard Chang, Software Engineering Architect, Salesforce How it works GPU partitioning adds two optional fields to API calls you already make. There are no new API operations, no new resource types, and no new AWS Identity and Access Management IAM actions, and an endpoint that omits both fields behaves exactly as it does today. AcceleratorPartitionLayouts goes on a production variant in CreateEndpointConfig and declares how each accelerator is divided: An A100 has seven compute slices to divide. The preceding layout uses all seven, so each of the eight A100s on the p4d instance carries three partitions and the endpoint provides 24, eight of each type. The carve applies uniformly. One layout entry describes how one accelerator is divided, and every accelerator on every instance in the variant is divided the same way. The list isn’t positional, so an entry doesn’t map to a specific GPU. AcceleratorPartitionType goes on an inference component and selects which partition the model runs on: Each component copy occupies one partition of the requested type. With eight partitions of each type on the instance, raising CopyCount places additional copies on free partitions. This is useful for a small model serving high concurrency, where a single process leaves the GPU waiting on requests. NumberOfAcceleratorDevicesRequired is always 1 here. With a partition type set it means one partition rather than one accelerator. NumberOfCpuCoresRequired and MinMemoryRequiredInMb still describe host CPU and host memory, because GPU memory comes from the partition type. Invocation doesn’t change. InvokeEndpoint addresses a model by inference component name, and the partition it occupies never appears in the request. Partitioning happens while the endpoint is provisioned. When it reaches InService every accelerator is carved and the partitions are empty, which is a valid state as components land on them afterward. In our testing, carving accelerators added roughly 17 seconds to provisioning. The rest of the time is the usual instance launch and container pull. Partition profiles by GPU The profiles available to you are a property of the GPU, so start by finding out which GPU your instance type carries and look it up in NVIDIA’s supported MIG profiles https://docs.nvidia.com/datacenter/tesla/mig-user-guide/latest/supported-mig-profiles.html , which is the authority on the full profile set and the combinations valid on one device. | Instance family | Accelerator | GPU memory | Granularity compute / memory | Available partition profiles | Max partitions per GPU | | P4d | NVIDIA A100 | 40 GB | 7 / 8 | mig-1g.5gb, mig-1g.10gb see driver note , mig-2g.10gb, mig-3g.20gb, mig-4g.20gb, mig-7g.40gb | 7 | | P5 | NVIDIA H100 | 80 GB | 7 / 8 | mig-1g.10gb, mig-1g.20gb, mig-2g.20gb, mig-3g.40gb, mig-4g.40gb, mig-7g.80gb | 7 | | G7 | NVIDIA RTX PRO 4500 Blackwell | 32 GB | 2 / 2 | mig-1g.16gb, mig-2g.32gb | 2 | | G7e | NVIDIA RTX PRO 6000 Blackwell Server Edition | 96 GB | 4 / 4 | mig-1g.24gb, mig-2g.48gb, mig-4g.96gb | 4 | Read the profile name as compute units followed by GPU memory: mig-3g.40gb takes three compute units and 40 GB. The granularity column is what limits a layout. An A100 and an H100 divide into sevenths on compute and eighths on memory, while G7 divides into halves and G7e into quarters. Profile names differ between GPUs even where the slice counts match, so don’t carry a layout from one instance family to another without checking it against this table. Within those limits you have real choice. On a P4d or P5 you can carve seven of the smallest profile, or two 3g partitions, or the 4g plus 2g plus 1g layout used in the earlier example, or take the whole device as one. On a G7e you can take four of the smallest, or two 1g beside a 2g, or two 2g, or one whole device. On a G7, either two partitions or one. The 3g profile is the one to know, because its two dimensions don’t match: it takes three of the seven compute units but four of the eight memory units. Two of them fill the GPU and nothing else will fit beside them. The high-level rule: Every profile in your layout must be one your GPU offers, and the compute and memory units your profiles add up to must not exceed what one device provides. Both are checked when you create the endpoint configuration, and either one failing is a rejection rather than a fallback to a whole accelerator. Driver note for P4d: mig-1g.10gb requires an R525 or newer NVIDIA driver, and the default P4d image ships 470.256.02. Request it on the default image and the carve fails without an error naming the profile. Set InferenceAmiVersion to al2023-ami-sagemaker-inference-gpu-4-1 on the production variant to get an image on the 580 driver line, which offers it. Choosing a layout for your workload Choosing well is a separate question from choosing a layout that is merely accepted. A partition hands you compute and memory in fixed proportions, and the two do different jobs during inference. Compute units are what you need for prefill. Prefill reads the prompt and builds its representation. This is the compute-heavy phase, and it scales with the streaming multiprocessors, and so the TFLOPs, your partition holds. The longer your inputs and the more prompts arriving at once, the more it governs how quickly you return a first token. Memory units are what you need for decode and for capacity. Generating tokens re-reads the model weights on every step, so decode is bounded by memory bandwidth rather than by math, and memory also caps how much KV cache you can hold. That cap in turn sets the context length you can serve and how many sequences you can serve concurrently. Almost no real workload exercises only one of these, so the goal isn’t to pick a dimension but to choose a combination that supports your use case. Choose a profile whose proportions cover your heaviest phase while leaving the other adequately served. On the compute side, what matters is how long your prompts are and how many arrive at once. On the memory side, it’s your output length, your context window, and your concurrency. Memory is the easier of the two to underestimate, because a partition has to hold three things and not just the model: the weights, which for an 8B model at FP16 are about 16 GB. The KV cache, sized for the context length and the number of concurrent sequences you intend to allow, which is the term people leave out and the one that grows with traffic. And 1–2 GB of runtime overhead for the CUDA context and the serving process. Add those up before choosing a profile, because a partition sized to the weights alone will load the model and then fail under load, or quietly serve a much shorter context than you configured. For a concrete reference point, that same 8B model at FP16 comfortably serves an 8,000-token context on a 40 GB partition. Where MIG adds the most value A portfolio of small and mid-size models – The direct case. Fine-tuned variants, embedding models, rerankers, a