{"slug": "running-gemma-4-on-ec2-g5g-graviton2-amd-with-nvidia-gpu", "title": "Running Gemma 4 on EC2 G5g: Graviton2 AMD with NVIDIA GPU", "summary": "An engineer successfully ran Google's Gemma 4 E2B model on AWS EC2 G5g, a Graviton2 (aarch64) instance with an NVIDIA T4G GPU, achieving 43.1 tokens per second after patching vLLM. The deployment required overcoming three major obstacles: missing architecture support in official builds, a version floor that only the newest vLLM clears, and a 64 KiB shared memory limit. The engineer found that AWS's Deep Learning ARM64 AMI includes PyTorch with sm_75 support, avoiding a from-source PyTorch build, but vLLM's own kernels required compilation for the Turing architecture.", "body_md": "*A field report on serving Google's Gemma 4 E2B on AWS EC2 **G5g** — a Graviton2 (aarch64)\n\nhost with an NVIDIA **T4G** (Turing, SM 7.5) GPU. Three obstacles: an **arch list** nobody\n\npublishes for this combination, a **version floor** that only the newest vLLM clears, and\n\n**64 KiB of shared memory** that stops the model dead. Plus the seven things I documented\n\nwrong before I had a box.*\n\n| Model |\n`google/gemma-4-E2B-it` (reference bf16 release) |\n| Hardware | AWS EC2 `g5g.4xlarge` — Graviton2 + 1x NVIDIA T4G, compute capability 7.5, 15,360 MiB |\n| Base image | Deep Learning ARM64 AMI OSS Nvidia Driver GPU PyTorch 2.12 (Ubuntu 24.04) |\n| Software | torch 2.12.0+cu132 · CUDA 13.2 · vLLM v0.27.2rc0 built from source for `sm_75`\n|\n| Result |\n43.1 tok/s single-stream greedy, 329,579-token KV cache — after one patch to vLLM |\n\nG5g is the only instance AWS has ever shipped that puts an NVIDIA GPU behind a Graviton\n\nhost. It launched in 2020, it never got a successor, and Graviton is now on its fifth\n\ngeneration without one.\n\nThat matters more than it sounds. The Arm-plus-CUDA world moved on to NVIDIA's own Arm CPU\n\n— Grace, paired with SM 9.0 and 10.0 parts. Turing stayed well supported, on x86. G5g is\n\nthe only hardware that is aarch64 *and* compute capability 7.5, and almost nobody publishes\n\na build for that combination.\n\nI put a rig on one anyway. **The packaging problem was the quick part.** Everything after it\n\n— a compiler that was not there, a version floor I did not expect, and 32 KiB of shared\n\nmemory — took far longer, because none of it fails where you are looking.\n\nStart with the obvious candidate. `vllm/vllm-openai:v0.27.1`\n\npublishes both platforms under\n\none tag, and you can read the arch lists straight out of the image config without pulling a\n\nlayer:\n\n```\ndocker buildx imagetools inspect vllm/vllm-openai:v0.27.1 --format '{{json .Image}}'\nlinux/amd64   7.5 8.0 8.6 8.9 9.0 10.0 12.0\nlinux/arm64       8.0 8.7 8.9 9.0 10.0 11.0 12.0\n```\n\nThe one architecture this hardware needs is the only entry the two images disagree on. The\n\narm64 list is Ampere and up, because that is what ships as an Arm-plus-NVIDIA system: A100,\n\nJetson Orin, GH200, Blackwell. Turing is not on that list and never will be.\n\nNormally a missing target degrades to JIT from embedded PTX. Not here. The Dockerfile says\n\nso, with a comment:\n\n```\n# Do not add +PTX here: vLLM filters torch's top-level PTX flag when it\n# converts global gencode flags into per-kernel arch lists.\n```\n\nSo it does not run slowly. It fails outright, with `no kernel image is available for`\n\n.\n\nexecution on the device\n\nThe rest of the ecosystem splits the same way. Check before you plan anything:\n\n| Artifact | 7.5 on arm64 | State |\n|---|---|---|\n`vllm/vllm-openai` arm64 |\nno | Current. Never had it. |\n`nvcr.io/nvidia/pytorch` arm64 |\nthrough 24.10 | Dropped by 24.12. |\n`drikster80/vllm-aarch64` |\nyes | Abandoned Sept 2024. vLLM 0.6.1, far too old for Gemma 4. |\n| PyPI torch aarch64 | no | Built for 9.0 / 10.0 / 12.0. |\nAWS ARM64 GPU DLAMI |\nyes |\nMaintained. PyTorch 2.2 through 2.12. |\n\nThis is the finding that saves the whole exercise, and I nearly wrote it off. I had assumed\n\nPyTorch's aarch64 CUDA wheels lacked `sm_75`\n\nand that a from-source PyTorch build was\n\ncoming. That is true of the PyPI wheels. It is not true of AWS.\n\nRead on two different DLAMIs, on the box:\n\n```\ntorch 2.7.0+cu128    ['sm_75', 'sm_90', 'sm_100', 'sm_120']\ntorch 2.12.0+cu132   ['sm_75', 'sm_80', 'sm_90', 'sm_100', 'sm_110', 'sm_120']\n```\n\nAWS sells G5g, so AWS keeps Turing in the build — right through PyTorch 2.12 on CUDA 13.2,\n\nan image cut three months ago. **PyTorch never needs building.** Only vLLM's own kernels do,\n\nand CMake takes the arch list without argument:\n\n```\n-- CUDA target architectures: 7.5\nCMake Warning: Pytorch version 2.11.0 expected for CUDA build, saw 2.12.0 instead.\n```\n\nThat warning is worth reading twice, and I come back to it below.\n\nTwo things the DLAMI does not give you, neither of them documented anywhere I could find.\n\nThere is no `nvcc`\n\n. The image ships the driver and a torch built against CUDA, not the\n\ntoolkit. You need the keyring and `cuda-toolkit-13-2`\n\nfrom NVIDIA's **sbsa** repo — not the\n\nx86 one, which is an easy reflex to get wrong on an Arm box.\n\nAnd vLLM now wants Rust. Its `vllm-rs`\n\nfrontend needs `setuptools_rust`\n\nplus a toolchain,\n\nand the failure is a bare `ModuleNotFoundError: No module named 'setuptools_rust'`\n\nthrown\n\nfrom metadata generation, several minutes in.\n\nNo vLLM tag pins torch 2.12. They go 2.11, then jump to 2.13. I reasoned that building older\n\ncode against a newer runtime was the safer direction, took v0.26.0, and spent an hour being\n\nwrong about it.\n\nIt builds fine. It then dies on model load:\n\n```\ntransformers.integrations.heterogeneity.configuration_utils.AmbiguousGlobalPerLayerAttributeError:\n'head_dim' is a per-layer attribute and may vary across layers.\n```\n\nGemma 4's `head_dim`\n\nis not one number, and current `transformers`\n\nrefuses to hand out a\n\nglobal value for it. vLLM's config converter was still doing a flat\n\n`getattr(config, \"head_dim\", 0)`\n\n. The `per_layer_config`\n\nhandling that copes with it landed\n\nin **v0.27.2rc0** — not v0.27.1, which I also checked. The newest tag was the only one that\n\nworked.\n\nIf you take one process lesson from this: reach for the latest release first, and make the\n\nconstraint say out loud what stopped you when you fall back.\n\nWith the build working the server still would not start, and this failure has nothing to do\n\nwith Arm or packaging. It is this model against this chip.\n\n```\nGemma4 model has heterogeneous head dimensions\n{'sliding_attention': 256, 'full_attention': 512}.\nFA4 not available, forcing TRITON_ATTN backend.\n```\n\nRead that as a chain, because every link is load-bearing:\n\n`TRITON_ATTN`\n\n.`VLLM_ATTENTION_BACKEND`\n\nis not a recognised variable\nin v0.27 — it logs `Unknown vLLM environment variable detected`\n\nand carries on. I set it\ntwice before I read the warning.`head_size=512`\n\nwants about 96 KiB of shared memory\nper block.Turing's shared memory is two numbers, and both are real. The **default** static limit per block\n\nis 48 KiB — that is what `torch.cuda.get_device_properties().shared_memory_per_block`\n\nreports,\n\n49,152 bytes. A kernel that needs more has to opt in through the dynamic shared-memory\n\nattribute, and even then it tops out at **64 KiB**. Ampere and later have 164 KiB and up.\n\nTriton opts in, so it is measuring against the 64 KiB ceiling. It still does not fit:\n\n```\ntriton.runtime.errors.OutOfResources: out of resource: shared memory,\nRequired: 98304, Hardware limit: 65536\n```\n\nRefused outright. Not slow, not degraded — the kernel will not launch, and it takes the\n\nengine down during CUDA graph capture, which is late enough that you have already watched\n\nthe weights load and the KV cache get sized.\n\nThe fix is small. Shrink the KV tile until the query block and the K/V tiles fit inside the\n\nbudget, and drop the software pipeline to one stage. Gate it on pre-Ampere so it is a no-op\n\non every other card:\n\n```\nif current_platform.get_device_capability()[0] < 8:\n    _smem_budget = 60000\n    _esz = q.element_size()\n    def _fits(t): return (BLOCK_M + 2 * t) * head_size * _esz <= _smem_budget\n    while TILE_SIZE_PREFILL > 16 and not _fits(TILE_SIZE_PREFILL): TILE_SIZE_PREFILL //= 2\n    while TILE_SIZE_DECODE  > 16 and not _fits(TILE_SIZE_DECODE):  TILE_SIZE_DECODE  //= 2\n    launch_num_stages = 1\n```\n\nWith that in `vllm/v1/attention/ops/triton_unified_attention.py`\n\n, graphs capture, the engine\n\ncomes up in 76 seconds, and the model serves. **This is not upstream.** It lives on my\n\ninstance and has to be reapplied on any vLLM upgrade, which makes it the obvious thing to\n\nsend back.\n\n67 minutes on a `g5g.4xlarge`\n\nat `MAX_JOBS=12`\n\n, and the majority of it is FlashAttention.\n\nvLLM compiles FA2 and FA3 **regardless of TORCH_CUDA_ARCH_LIST** — I watched it grind\n\n`sm90`\n\nHopper instantiations on a build targeting 7.5 only. FA2 needsConstraining `VLLM_FA_CMAKE_GPU_ARCHES`\n\nshould cut that dramatically. I did not try it,\n\nbecause by the time I understood what I was looking at the build was 45 minutes in and\n\ninterrupting it would have cost more than finishing.\n\nI wrote the rig's documentation before provisioning anything. Seven claims in it were wrong,\n\nand every correction came off the machine rather than out of an argument. This is the part I\n\nwould keep if I kept nothing else.\n\n| What I wrote | What the box said |\n|---|---|\nPyTorch aarch64 lacks `sm_75`\n|\nAWS DLAMI has it, on both versions I checked |\n| bfloat16 is a hard failure here | Torch upconverts; vLLM logs `Casting torch.bfloat16 to torch.float16` and proceeds |\n| The backend is XFORMERS |\n`TRITON_ATTN` , forced, not selectable |\n`VLLM_ATTENTION_BACKEND` picks it |\nNot a recognised variable. I had shipped dead config. |\n| w4a16 needs sm80+ Marlin | The build compiled `sm75_kernel_float16_u4b8_float16.cu.o`\n|\n| The GPU has 16 GB | 15,360 MiB |\n`/v1/completions` returns an empty body |\nIt returns `': ok: ok: ok: ok'` — garbage, not silence |\n\nThat last one has teeth. If you health-check by testing for an empty response, this endpoint\n\npasses while producing nonsense. Use `/v1/chat/completions`\n\nand read the text.\n\nOne claim is still standing only because I never tested it: whether `g5g.xlarge`\n\n's 8 GiB of\n\nhost RAM can stage 9.5 GiB of weights. Safetensors loading is mmap-backed, so I suspect it\n\ncan. It is labelled untested rather than stated as fact, which is where it should have been\n\nall along.\n\n```\ncontent: 'Site Reliability Engineering (SRE) is a discipline that applies\n          software engineering principles to infrastructure and operations\n          problems to create highly reliable, scalable, and efficient systems.'\nfinish_reason: stop      usage: 19 prompt / 32 completion / 51 total\n```\n\n| Measure | Value |\n|---|---|\n| Throughput, single stream greedy | 42.9 tok/s @ 64, 43.1 @ 256 |\n| KV cache | 2.95 GiB, 329,579 tokens |\n| Concurrency at 16k context | 20.12x |\n| GPU memory while serving | 13,501 / 15,360 MiB |\n| Engine init | 76.4 s, graph capture 17 s |\n| Memory bandwidth, measured | 277.0 GB/s read · 234.3 GB/s copy (320.1 theoretical) |\n\nBefore reading too much into 43 tok/s, note what the memory does. The T4G has **GDDR6, not\nHBM** — 256-bit bus at 5,001 MHz, so 320 GB/s theoretical. I measured\n\nSingle run, single stream, no repeats and no variance figure. One sample per cell, and taken\n\nwith the clamped tiles, so it is a floor rather than a characterisation. My Inferentia port\n\nmeasured about 44 tok/s for E2B on one core, which is the same neighbourhood — but that is a\n\ndifferent harness on different silicon and I would not put the two in one table.\n\n| Symptom | Cause |\n|---|---|\n`no kernel image is available` |\nStock arm64 image. No 7.5, no PTX. Build from source. |\n`OutOfResources: shared memory` |\nTuring's 64 KiB against a 512-wide head. Clamp the tiles. |\n`AmbiguousGlobalPerLayerAttributeError` |\nvLLM older than v0.27.2rc0. |\n`No module named 'setuptools_rust'` |\nMissing Rust toolchain for `vllm-rs` . |\n`nvcc: not found` |\nPyTorch DLAMI has no toolkit. Install `cuda-toolkit-13-2` (sbsa). |\n`Unknown vLLM environment variable` |\nYou set `VLLM_ATTENTION_BACKEND` . It does nothing. |\n| Healthy endpoint, nonsense output | You checked `/v1/completions` . Use chat completions. |\n\nTake the AWS ARM64 GPU PyTorch DLAMI — it is the only maintained aarch64 stack that still\n\ncarries `sm_75`\n\n. Add `cuda-toolkit-13-2`\n\nfrom the sbsa repo and a Rust toolchain, because the\n\nimage ships neither. Build vLLM v0.27.2rc0 or newer from source with\n\n`TORCH_CUDA_ARCH_LIST=7.5`\n\nand `use_existing_torch.py`\n\n, and patch the Triton attention kernel\n\nto fit Turing's shared memory before you try to start it. Serve with `--dtype float16`\n\nand\n\n`--kv-cache-dtype auto`\n\n.\n\nNothing here failed loudly, and nothing failed where I was looking. The packaging gap I built\n\nthe rig around was already solved by AWS; the thing that actually stopped me was 32 KiB of\n\nshared memory and a model whose global attention heads are twice as wide as its sliding ones.\n\nHardware this far off the mainstream will keep producing that shape of surprise — the fix is\n\nnot to reason harder about it, but to get to a box sooner and let it tell you.\n\n*Measured on EC2 g5g.4xlarge spot, us-east-1a. NVIDIA T4G, compute capability 7.5,*", "url": "https://wpnews.pro/news/running-gemma-4-on-ec2-g5g-graviton2-amd-with-nvidia-gpu", "canonical_source": "https://dev.to/aws-builders/running-gemma-4-on-ec2-g5g-graviton2-amd-with-nvidia-gpu-13j", "published_at": "2026-08-13 18:45:41+00:00", "updated_at": "2026-08-13 19:20:45.155237+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Google", "AWS", "NVIDIA", "vLLM", "PyTorch", "Gemma 4", "Graviton2", "T4G"], "alternates": {"html": "https://wpnews.pro/news/running-gemma-4-on-ec2-g5g-graviton2-amd-with-nvidia-gpu", "markdown": "https://wpnews.pro/news/running-gemma-4-on-ec2-g5g-graviton2-amd-with-nvidia-gpu.md", "text": "https://wpnews.pro/news/running-gemma-4-on-ec2-g5g-graviton2-amd-with-nvidia-gpu.txt", "jsonld": "https://wpnews.pro/news/running-gemma-4-on-ec2-g5g-graviton2-amd-with-nvidia-gpu.jsonld"}}