Gemma 4 on Amazon SageMaker: The NVIDIA T4 Decodes at 0.8x of the L4 With the Same Answers

A developer deployed Gemma 4 (E2B, E4B, 12B and 26B A4B) to Amazon SageMaker on a single NVIDIA T4 (ml.g4dn.xlarge), measuring decode throughput at 0.77x to 0.82x of an L4 while producing identical answers, with 12B the largest build that fits in 16 GB. The work required a Turing attention patch to vLLM 0.30.0, since the Triton attention kernel requests 98,304 bytes of shared memory per block against the T4's 65,536-byte limit, and includes a 26B A4B W4A16 repack that keeps Google's QAT weights on their 4-bit grid (0 of 11,534,336 layer-0 query-projection weights change level, versus 30.3% for an AWQ build). A companion MCP server exposes tools including check_quotas for reading per-region SageMaker endpoint quotas.

This article deploys Gemma 4 to the smallest GPU Amazon SageMaker offers, an NVIDIA T4, and measures it against the L4 from the earlier parts. A suite of Python MCP tools is built to simplify management of the vLLM hosted deployment. https://github.com/xbill9/sagemaker-gemma https://github.com/xbill9/sagemaker-gemma | Models | Gemma 4 E2B, E4B, 12B and 26B A4B, 4-bit weights and 4-bit embeddings | | Hardware | SageMaker ml.g4dn.xlarge , 1x NVIDIA T4, 16 GB; compared with ml.g6.xlarge , 1x L4 | | Region | us-east-2 | | Software | AWS vLLM SageMaker container 0.30.0, with a Turing attention patch | | Result | The T4 decodes at 0.77x to 0.82x of the L4 and gives identical answers; 12B is the largest build that fits | This is part four of a series. Part one deploys Gemma 4 to a SageMaker endpoint with the aws CLI and an MCP server: https://dev.to/aws-builders/gemma-4-on-an-amazon-sagemaker-endpoint-aws-cli-nvidia-l4-and-an-mcp-server-2c9d https://dev.to/aws-builders/gemma-4-on-an-amazon-sagemaker-endpoint-aws-cli-nvidia-l4-and-an-mcp-server-2c9d Part two measures Google's QAT checkpoint against the full-size bf16 release: https://dev.to/aws-builders/gemma-4-on-amazon-sagemaker-qat-weights-decode-205x-faster-than-bf16-on-one-l4-318m https://dev.to/aws-builders/gemma-4-on-amazon-sagemaker-qat-weights-decode-205x-faster-than-bf16-on-one-l4-318m Part three repacks the QAT weights with 4-bit embeddings: https://dev.to/aws-builders/gemma-4-on-amazon-sagemaker-4-bit-embeddings-decode-up-to-139x-faster-on-one-l4-36mf https://dev.to/aws-builders/gemma-4-on-amazon-sagemaker-4-bit-embeddings-decode-up-to-139x-faster-on-one-l4-36mf This article takes part three's builds down one GPU generation. SageMaker JumpStart lists Gemma 4 from ml.g6e.xlarge , one NVIDIA L40S, for E2B, and 12B only on ml.g6e.16xlarge ; none of its Gemma 4 entries lists a T4. Gemma 4 on Turing GPUs is an open vLLM issue, 38918, "Gemma4 on Turing GPUs SM 7.5 : all attention backends hit shared memory limits", reported again on vLLM 0.29.0 in September. The builds served here, with 4-bit embeddings, are the author's repacks of Google's QAT weights on Hugging Face. The 26B A4B build is the first W4A16 checkpoint of that model to keep Google's QAT weights on their 4-bit grid. In layer 0's query projection, 0 of its 11,534,336 weights change level, where an AWQ build of the same QAT export, published in June, moves 30.3% of them by a full level or more. grid check.py reads that one tensor from each checkpoint over HTTP and compares them. make test passing aws login session in an account with SageMaker endpoint quota for ml.g4dn.xlarge The project's MCP server has a check quotas tool that reads the account's endpoint quota for each instance type in every US region: | Instance | us-east-1 | us-east-2 | us-west-1 | us-west-2 | | ml.g4dn.xlarge | 2 | 2 | 2 | 2 | | ml.g5g.xlarge | - | - | - | - | | ml.g5.xlarge | 2 | 2 | - | 2 | | ml.g6.xlarge | 1 | 1 | - | 1 | | ml.inf2.xlarge | 0 | 2 | - | 0 | A dash means SageMaker has no quota entry for the type. ml.g4dn.xlarge , one T4 with 16 GB, is the smallest GPU on offer. The Graviton T4G g5g exists on EC2 only. Inferentia2 is listed, but it runs the Neuron SDK and needs a different container and a compiled model. Gemma 4's full-attention layers are 512 wide. vLLM serves them with its Triton attention kernel, whose tile asks for 98,304 bytes of shared memory per block. A T4 is a Turing GPU and allows 65,536, so the engine stops at start-up: triton.runtime.errors.OutOfResources: shared memory, Required: 98304, Hardware limit: 65536 vLLM 0.30.0, the version in the SageMaker container, has no fix for it. The author's Compute Engine T4 serves Gemma 4 with a script that halves the tiles on pre-Ampere GPUs, and the SageMaker image gets the same script, in a Dockerfile built FROM the stock container: ARG BASE IMAGE FROM ${BASE IMAGE} COPY patch triton turing.py /opt/turing/patch triton turing.py RUN set -eu; \ target="$ python3 -c 'import importlib.util, os; print os.path.join importlib.util.find spec "vllm" .submodule search locations 0 , "v1/attention/ops/triton unified attention.py" ' "; \ python3 /opt/turing/patch triton turing.py "$target"; \ python3 /opt/turing/patch triton turing.py --check "$target"; \ python3 -c 'import torch, sys; a = torch. C. cuda getArchFlags ; print "torch arch:", a ; sys.exit 0 if "sm 75" in a else 1 ' The last line refuses the build if PyTorch in the image has no Turing kernels. CodeBuild builds the image and pushes it to a private ECR repository, so the 8 GB base image never touches the local disk: 8 1.092 patch triton turing: patched /usr/local/lib/python3.12/dist-packages/vllm/v1/attention/ops/triton unified attention.py 8 1.092 smem budget : 60000 B of Turing's 65536 hard limit 8 5.652 torch arch: sm 75 sm 80 sm 86 sm 90 sm 100 sm 120 The patch is a no-op on an L4 or newer, and the image keeps the stock entrypoint, so every SM VLLM setting works as before. The Dockerfile and buildspec.yml are in the repository, in the turing directory. A SageMaker production variant runs on one of several host images, each with its own NVIDIA driver, chosen by InferenceAmiVersion : al2-ami-sagemaker-inference-gpu-2 NVIDIA driver 535, CUDA 12.2 al2-ami-sagemaker-inference-gpu-3-1 NVIDIA driver 550, CUDA 12.4 al2023-ami-sagemaker-inference-gpu-4-1 NVIDIA driver 580, CUDA 13.0 The vLLM 0.30.0 container is built on CUDA 13: NVIDIA REQUIRE CUDA=cuda =13.0 ... CUDA VERSION=13.0.2 sm.py passes the host image through from .env : INFERENCE AMI VERSION=al2023-ami-sagemaker-inference-gpu-4-1 Without InferenceAmiVersion , an ml.g4dn.xlarge endpoint with this image ends about six minutes after Creating with: CannotStartContainerError. Please ensure the model container for variant AllTraffic starts correctly when invoked with 'docker run <image serve' and no CloudWatch log group at all. A missing log group points at the host, before any code in the image runs: compare the container's NVIDIA REQUIRE CUDA with the driver of the host image. With the setting above, the same image on the same instance type reached InService in 13.2 minutes. The sweep serves part three's text-only builds with 4-bit embeddings, one endpoint at a time: deploy, measure, tear down. Settings match the L4 runs except for the data type, since Turing has no bf16: IMAGE URI=<account .dkr.ecr.us-east-2.amazonaws.com/sagemaker-gemma-vllm:0.30.0-sagemaker-v1.3-sm75 INFERENCE AMI VERSION=al2023-ami-sagemaker-inference-gpu-4-1 INSTANCE POOLS=ml.g4dn.xlarge,ml.g4dn.2xlarge MAX MODEL LEN=8192 SM VLLM DTYPE=float16 Each endpoint went from the deploy call to InService in 13.3 minutes. compare.py then measures single-request decode, 1 to 16 parallel requests and 40 questions at temperature 0, as in parts two and three, and compare.py combine sets each T4 run beside its L4 run: decode tokens per second 35.0 28.5 0.81 load c16 tokens per second 411.8 216.45 0.53 quality correct 40 40 identical answers: 40/40 | Model | Decode, T4 tok/s | Decode, L4 tok/s | T4 / L4 | 16 parallel, T4 / L4 | |---|---|---|---|---| | E2B | 108.5 | 141.7 | 0.77 | 0.63 | | E4B | 65.6 | 79.9 | 0.82 | 0.57 | | 12B | 28.5 | 35.0 | 0.81 | 0.53 | One user at a time, the T4 runs at about 0.8x of the L4. Under load the gap widens with model size: at 16 parallel requests the T4 delivers 0.63x of the L4 at E2B and 0.53x at 12B. The Compute Engine T4 serving the same E2B build with the same patch decodes at 109.7 tok/s, against 108.5 here. That rig runs vLLM 0.29.0 from pip with a 16,384-token context, so the agreement is a cross-check and not a controlled comparison. All three models give the same 40 answers on the T4 as on the L4, byte for byte: 36 of 40 correct for E2B and E4B, 40 of 40 for 12B. The T4 computes in fp16 and runs the clamped attention tiles, and neither changed an answer. | Model | Weights GiB | KV cache, T4 tokens | KV cache, L4 tokens | |---|---|---|---| | E2B | 2.86 | 660,033 | 1,209,977 | | E4B | 4.52 | 194,336 | 376,156 | | 12B | 7.36 | 20,354 | 52,541 | The weights take the same memory on both GPUs. The T4 has less left for the KV cache: 12B keeps 20,354 tokens, which is 2.48 requests at the full 8,192-token context. The 26B A4B build holds 14.2 GiB of weights on the L4. On the T4 it ran with the reduced settings 31B needed in part three: memory 0.97, a 1,024-token context and four sequences. The weights loaded and the engine then ran out of memory: memory allocation failed with OOM on device 0 while trying to allocate 253755392 bytes free: 191627264, total: 15636037632 . The T4 reports 14.56 GiB in total. Of the builds tested, 12B is the largest that serves on one T4. SageMaker's on-demand hosting price in us-east-2 , from the AWS Price List API on 2026-09-30, is $0.736 an hour for ml.g4dn.xlarge and $1.1267 for ml.g6.xlarge . Per million output tokens at 16 parallel requests: | Model | T4 | L4 | T4 / L4 | |---|---|---|---| | E2B | $0.260 | $0.249 | 1.05 | | E4B | $0.419 | $0.369 | 1.14 | | 12B | $0.945 | $0.760 | 1.24 | The T4 costs 0.65x the L4 per hour and delivers 0.53x to 0.63x of its throughput under load, so a busy T4 costs more per token, by 5% at E2B and 24% at 12B. A mostly idle endpoint is billed by the hour, and there the T4 costs 0.65x. sm.py sets ContainerStartupHealthCheckTimeoutInSeconds to 1800, which leaves room for a large model to download and load. A container that crashes at start holds the endpoint in Creating for that whole window: the 26B endpoint ran out of memory at 15:33 and was marked Failed at 15:59, 36.5 minutes after Creating . For a small model that loads in under two minutes, a few hundred seconds is enough, and a failed start comes back that much sooner. | Workload | GPU | Why | |---|---|---| | 🥇 E2B or E4B, light traffic | T4 | 0.65x the hourly price, 0.8x the single-user speed | | 🥈 Any size, steady load | L4 | Lower cost per token, 1.8x to 2.6x the KV cache | | 🥉 12B | L4 | 12B fits the T4 with room for 2.48 full-length requests | | 26B A4B and 31B | L4 or larger | They do not fit a T4 | Every endpoint was deleted after its measurement, and a watchdog deleted anything left behind when the sweep exited: 2026-09-30T15:59:45Z gemma-4-e2b-emb4-t4: gone 2026-09-30T15:59:46Z gemma-4-e4b-emb4-t4: gone 2026-09-30T15:59:46Z gemma-4-12b-emb4-t4: gone 2026-09-30T15:59:47Z gemma-4-26b-emb4-t4: gone The build resources cost nothing while idle: an ECR repository, a CodeBuild project, its role and a source bucket in us-east-2 . The goal of this article was to serve Gemma 4 on the smallest GPU SageMaker offers and measure what it gives up against the L4. The key to the solution was a Turing patch for vLLM in a derived container, on the host image with driver 580. The T4 results were: InferenceAmiVersion set to the driver 580 host on ml.g4dn Scope: one account, SageMaker ml.g4dn.xlarge in us-east-2 , vLLM 0.30.0 from the AWS container with the Turing patch, fp16, one deployment per model on 2026-09-30. The L4 figures are part three's, measured on 2026-09-29 in bf16 with the stock container. Throughput was measured through the aws CLI from one client machine, and prices are on-demand list prices. The strategy for using MCP for SageMaker deployment and benchmarking was validated with an incremental step by step approach.