{"slug": "the-cheapest-cuda-gpu-on-aws-has-an-arm-cpu-and-you-probably-want-the-intel-one", "title": "The Cheapest CUDA GPU on AWS Has an Arm CPU — and You Probably Want the Intel One", "summary": "A developer's benchmark of the two cheapest CUDA GPU instances on AWS reveals that while the Arm-based g5g.xlarge is 20% cheaper per hour than the Intel-based g4dn.xlarge, the Intel instance is more cost-effective for token generation due to a packaging issue in vLLM's Docker images. The arm64 image lacks SM 7.5 kernels, forcing a source build that adds overhead, whereas the amd64 image includes precompiled kernels for the T4 GPU.", "body_md": "This article provides a step by step deployment guide for Gemma 4 E2B onto the two cheapest\n\nwhole GPU CUDA instances AWS sells, and compares what they cost to run. A suite of Python MCP\n\ntools is built to simplify management of the vLLM hosted deployment. Everything was measured on\n\n2026-08-30.\n\nThe question is simple: if you want a CUDA GPU on AWS as cheaply as possible, which one do you\n\nrent?\n\nTwo instance families sit at the bottom of the price list, and they carry the same generation of\n\nNVIDIA Turing silicon. G5g pairs a T4G with a Graviton2 host on aarch64. G4dn pairs a T4 with an\n\nIntel host on x86_64. The GPUs are effectively the same part. The host CPU is the variable.\n\nThe answer turns out to depend entirely on whether you are buying hours or tokens.\n\nStart with price. Every NVIDIA instance type in us-east-1 was priced from the AWS Pricing API\n\nrather than from documentation:\n\n```\naws pricing get-products --region us-east-1 --service-code AmazonEC2 \\\n  --filters Type=TERM_MATCH,Field=instanceType,Value=g4dn.xlarge \\\n            Type=TERM_MATCH,Field=location,Value=\"US East (N. Virginia)\" \\\n            Type=TERM_MATCH,Field=operatingSystem,Value=Linux \\\n            Type=TERM_MATCH,Field=tenancy,Value=Shared \\\n            Type=TERM_MATCH,Field=preInstalledSw,Value=NA \\\n            Type=TERM_MATCH,Field=capacitystatus,Value=Used\n```\n\nSixty four types came back. Keeping only those that give you a whole GPU rather than a\n\nfractional slice:\n\n| Rank | Instance | Host CPU | $/hr On-Demand | $/hr Spot | GPU | VRAM |\n|---|---|---|---|---|---|---|\n| 🥇 | `g5g.xlarge` |\nGraviton2 arm64 | 0.4200 | 0.1458 | T4G | 15,360 MiB |\n| 🥈 | `g4dn.xlarge` |\nIntel x86_64 | 0.5260 | 0.3559 | T4 | 15,360 MiB |\n| 🥉 | `g5g.2xlarge` |\nGraviton2 | 0.5560 | — | T4G | 15,360 MiB |\n`g4dn.2xlarge` |\nIntel | 0.7520 | — | T4 | 15,360 MiB | |\n`g6.xlarge` |\nAMD x86_64 | 0.8048 | 0.7033 | L4 | 22,888 MiB |\n\n**The cheapest real CUDA GPU on AWS is an Arm box.** It is 20 percent cheaper per hour on demand\n\nand 59 percent cheaper on spot.\n\nThree instances are cheaper still, at 0.2020, 0.2375 and 0.4750, but all three are fractional L4\n\nslices with 2,861 to 5,722 MiB, and none of them can map Gemma 4 E2B's 10.2 GB checkpoint. The\n\ncheapest slice that could, `g6f.4xlarge`\n\nat 11,444 MiB, costs 0.95 per hour.\n\nSo the headline is the Arm box. It is also the wrong number to buy on, and the rest of this\n\narticle is why.\n\nRead off the running instances rather than the spec sheets:\n\n| Property | T4 on G4dn | T4G on G5g |\n|---|---|---|\n| Compute capability | 7.5 | 7.5 |\nVRAM, `nvidia-smi`\n|\n15,360 MiB | 15,360 MiB |\n| Memory clock | 5,001 MHz | 5,001 MHz |\n| Bus width | 256 bit | 256 bit |\n| Theoretical peak bandwidth | 320.1 GB/s | 320.1 GB/s |\n| GPU KV cache allocated by vLLM | 329,579 tokens | 329,579 tokens |\n\nEvery measurable property matches, down to vLLM independently arriving at a KV cache of exactly\n\n329,579 tokens on both. Whatever separates these deployments, it is not the accelerator.\n\nThis is the mechanism, and it has nothing to do with how fast either CPU runs.\n\n`vllm/vllm-openai`\n\npublishes one manifest list with two platforms, and they are not compiled for\n\nthe same GPUs. Read straight out of the registry config blobs:\n\n```\nlinux/amd64  sha256:2286e8533ca8\n  TORCH_CUDA_ARCH_LIST=7.5 8.0 8.6 8.9 9.0 10.0 12.0    sm_75 present\n\nlinux/arm64  sha256:2a7cde230b59\n  TORCH_CUDA_ARCH_LIST=8.0 8.7 8.9 9.0 10.0 11.0 12.0   sm_75 absent\n```\n\nSame tag, same day. **Only the amd64 image carries SM 7.5.**\n\nThe host architecture selects the manifest. An Intel host pulls kernels that run on its T4. A\n\nGraviton2 host pulls an image with no kernels for its own GPU, and the Dockerfile sets no `+PTX`\n\n,\n\nso there is not even a JIT fallback. That rig compiles vLLM from source before it serves a\n\ntoken.\n\nThis is a packaging decision by the vLLM project, not a property of either CPU, and it is the\n\nsingle largest cost difference between the two families.\n\nG4dn gives you 4 GiB per vCPU. G5g gives you 2:\n\n| Instance | vCPU | Host RAM |\n|---|---|---|\n`g4dn.xlarge` |\n4 | 16 GiB |\n`g5g.xlarge` |\n4 | 8 GiB |\n\nGemma 4 E2B's checkpoint is 9.54 GiB. On `g5g.xlarge`\n\n, with about 7.5 GiB usable, the kernel\n\ndeclines to map it and vLLM crash loops before a single page is faulted in:\n\n```\nRuntimeError: unable to mmap 10246621918 bytes from model.safetensors:\nCannot allocate memory (12)\n```\n\nThat is a failure of the mapping, not of residency, and a swapfile fixes it. But the cheapest\n\nCUDA instance on AWS needs configuration the next one up does not. `g4dn.xlarge`\n\nhas 16 GiB and\n\nmaps the checkpoint with no swapfile at all.\n\n`g4dn.xlarge`\n\nneeds 4`vllm/hf-token`\n\n`AmazonSSMManagedInstanceCore`\n\n```\ngit clone https://github.com/xbill9/gemma4-dev\ncd gemma4-dev/gpu-vllm-g4dn-2b\npip install -r requirements.txt\n```\n\nThe requirements are small:\n\n```\nmcp\nhttpx\nboto3\n```\n\nThe Hugging Face token is fetched at boot and never placed in user data, because instance\n\nmetadata is readable by anything on the box. The fetch is wrapped in `set +x`\n\n, because the\n\nbootstrap runs under `set -x`\n\nand bash traces assignments with their values.\n\nThe MCP server is a single Python file started over stdio. The standard MCP libraries abstract\n\nthe transport, so the tool implementations are identical no matter which client connects.\n\nRegistration lives in four places and all four must name the server identically: `.mcp.json`\n\n,\n\nthe plugin manifest, `.codex/config.toml`\n\n, and `enabledMcpjsonServers`\n\n. A rename that updates\n\nthree of four leaves an approval gate naming a tool that does not exist, and a gate on a tool\n\nname that does not exist fails open and says nothing.\n\n```\n./project-setup.sh --server-name gpu-vllm-g4dn-2b\n```\n\nQuota first:\n\n```\ncheck_g4dn_quotas\n| Running On-Demand G and VT instances (vCPU) | 16 |\n| All G and VT Spot Instance Requests (vCPU) | 16 |\ng4dn.xlarge needs 4 vCPUs.\n```\n\nQuota is not capacity. G family spot in us-east-1 has been exhausted in every AZ but one with\n\nquota to spare, and the one AZ with capacity was the most expensive. Price is not a proxy for\n\navailability:\n\n```\naws ec2 get-spot-placement-scores --region us-east-1 --instance-types g4dn.xlarge \\\n  --target-capacity 1 --single-availability-zone --region-names us-east-1\nuse1-az1  1\nuse1-az2  3\nuse1-az4  3\nuse1-az5  3\nuse1-az6  3\n```\n\n`use1-az4`\n\nmaps to `us-east-1c`\n\n, which scored 3 and carried the lowest spot price at 0.3559.\n\n```\ncreate_g4dn_instance\n  subnet_id=subnet-0c2872fe4182b9ec1\n  security_group_id=sg-01ee54036d37aa770\n  iam_instance_profile=<profile>\n  instance_type=g4dn.xlarge\n  spot=true\n✅ Launching i-050dca2ed568dcc1b (g4dn.xlarge, spot, 1x T4) in us-east-1.\nAMI: ami-0216c4aa131462acf\nPatch sha: 26b1cead19f4 → vllm-openai:v0.28.0-sm75-patched\n```\n\nThe AMI is never hardcoded. It resolves from SSM at launch and returned the Deep Learning Base\n\nOSS Nvidia Driver GPU AMI on Ubuntu 26.04, built two days before this run. The base DLAMI is\n\nused rather than the PyTorch one, because the deployment serves from a container carrying its\n\nown CUDA and torch.\n\n```\nget_install_progress i-050dca2ed568dcc1b\n[stage] image-pull-start          +0s\n[stage] image-pull-done         +155s\n[stage] patch-applied           +176s\n[stage] image-build-done        +178s\n[stage] patch-verified-in-image +193s\n[stage] serving-started         +195s\n[stage] INSTALL_COMPLETE        +195s\n```\n\n**195 seconds, and nothing is compiled.** The image derivation is 23 seconds of that, because\n\nthe kernels are already correct and exactly one pure Python file is replaced.\n\nThat file exists because Gemma 4 has two attention geometries — 28 sliding attention layers at\n\nhead dimension 256 and 7 full attention layers at 512, verified against the safetensors headers.\n\nOnly FA4 and Triton handle heterogeneous head dims, FA4 is unavailable on Turing, so vLLM forces\n\n`TRITON_ATTN`\n\n. Its tile at head size 512 wants 98,304 bytes of shared memory per block against\n\nTuring's 65,536 hard limit, and 49,152 static. The patch clamps that one path from 32 tiles to\n\n16, and leaves the other three alone.\n\nTuring has no bfloat16 and no fp8 datapath, so the deployment runs float16. Be precise about\n\nwhy: bfloat16 does not fail on Turing, it upconverts, and vLLM logs the cast and proceeds.\n\nfloat16 is correct because it is what executes.\n\nTwo checks, and passing the first says nothing about the second. The arch gap and the shared\n\nmemory ceiling are independent problems, and the Intel host only deletes the first:\n\n```\nverify_gpu_arch i-050dca2ed568dcc1b\nTesla T4, 7.5, 15360 MiB\ncapability: (7, 5)\ntorch arch list: ['sm_75', 'sm_80', 'sm_86', 'sm_90', 'sm_100', 'sm_120']\nshared mem per block (static): 49152\nfp16 matmul ok: True\n```\n\nThe probe uses float16, not bfloat16. A bfloat16 probe would pass by upconversion and tell you\n\nnothing about what executes.\n\n```\nverify_triton_patch i-050dca2ed568dcc1b\n--- image ---             PATCHED IMAGE PRESENT\n--- module in image ---   CLAMP PRESENT\n--- running container --- vllm-openai:v0.28.0-sm75-patched\n```\n\nThe third line matters as much as the first two. An image can be correctly patched while the\n\ncontainer runs the stock tag, and everything else still reports healthy.\n\n```\nGPU KV cache size: 329,579 tokens, Maximum concurrency for 16,384 tokens per request: 20.12x\n```\n\n| Phase | Time |\n|---|---|\n| Weights download | 30.03 s |\n| Checkpoint load, 9.54 GiB | 23.33 s |\n| Model loading total | 55.75 s, 9.8 GiB |\n| Engine init | 150.90 s, 82.39 s of it compilation |\n| CUDA graph capture | 13 s, 0.16 GiB |\n\nThe health check uses `/v1/chat/completions`\n\n. Raw `/v1/completions`\n\nskips the chat template and\n\nreturns an empty body on `-it`\n\nmodels, so an empty response there is not evidence either way.\n\nIt also carries a degeneracy check, and that is not a quality metric. A broken deploy on this\n\nlineage once answered `': ok: ok: ok…'`\n\n— sixteen tokens, non empty, completely wrong. Testing\n\nfor a non empty response would have passed it.\n\nWhat the deployed model actually returned:\n\nThis is a\n\ndescription of a data collection processfor performance metrics. Here's a\n\nbreakdown of what the text tells us: *What is being measured:Throughput and latency.\n\nThe client runs on the box against localhost, so no network sits between it and the engine under\n\ntest. 512 input tokens, 128 output, `ignore_eos`\n\n, request count scaling at four times\n\nconcurrency, prompts sized with vLLM's own `/tokenize`\n\nso input length is a model token count\n\nand not a word count guess.\n\n```\npython3 benchmarking_suite.py --url http://127.0.0.1:8000 \\\n  --contexts 512 --concurrencies 1,4,8,16,32 \\\n  --output-tokens 128 --prompts-per-concurrency 4\n```\n\n| Concurrency | Output tok/s | Per-stream tok/s | TTFT p50 ms | TPOT p50 ms |\n|---|---|---|---|---|\n| 1 | 42.36 | 42.77 | 53 | 23.38 |\n| 4 | 140.80 | 35.88 | 95 | 27.87 |\n| 8 | 🥇 242.47 | 31.08 | 130 | 32.18 |\n| 16 | 243.93 | 31.10 | 4,304 | 32.15 |\n| 32 | 242.67 | 30.87 | 12,749 | 32.39 |\n\nFive cells, all measured, every request successful, every request exactly 128 output tokens.\n\nThe engine saturates at concurrency 8, which is `--max-num-seqs 8`\n\n. Above that, throughput is\n\nflat to within 0.6 percent while median TTFT rises from 130 milliseconds to 12.7 seconds.\n\n**Concurrency past your max-num-seqs buys latency, not throughput.**\n\nThe Arm deployment, measured on `g5g.4xlarge`\n\n:\n\n| Concurrency | G4dn Intel T4 | G5g Graviton2 T4G |\n|---|---|---|\n| 1 | 🥇 42.36 | 28.65 |\n| 4 | 🥇 140.80 | 97.48 |\n| 8 | 🥇 242.47 | 168.33 |\n| 16 | 🥇 243.93 | 169.96 |\n| 32 | 🥇 242.67 | 170.99 |\n\nBoth saturate at concurrency 8. The Intel box converts its hour into 44 percent more output, and\n\nit does that on a quarter of the host — 4 vCPU against 16.\n\nCheapest per hour and cheapest per token are different boxes. That is the finding:\n\n| Deployment | $/hr On-Demand | tok/s at c=8 | $/M Output Tokens |\n|---|---|---|---|\n🥇 `g4dn.xlarge` Intel T4 |\n0.5260 | 242.47 | 0.603 |\n🥈 `g5g.4xlarge` Graviton2 T4G |\n0.8280 | 168.33 | 1.366 |\n\nThe Arm family owns the cheapest hourly rate on AWS for a whole CUDA GPU. The Intel family\n\ndelivers the cheaper token, by 2.3 times.\n\nWithin the Intel box, the operating point matters more than the family choice does:\n\n| Operating point | tok/s | Spot 0.3559/hr | On-Demand 0.526/hr |\n|---|---|---|---|\n| Saturation, c=8 | 242.47 | 🥇 0.408 per M | 0.603 per M |\n| Single stream, c=1 | 42.36 | 2.334 per M | 3.449 per M |\n\nCompute only, excluding EBS and data transfer. **Serving one stream at a time costs 5.7 times\nmore per token on identical hardware.** Get your concurrency to\n\n`max-num-seqs`\n\nbefore you shopThe two hourly rates are not comparable as operating costs, because they buy different amounts\n\nof work before you serve anything.\n\nOn the Intel box a launch costs an image pull and a 23 second derivation, reaching a serving\n\nendpoint in 195 seconds. On the Arm box the published image has no kernels for the GPU in the\n\ninstance, so a launch costs a from source build first, and the `xlarge`\n\nalso needs a swapfile\n\nbefore the checkpoint will map.\n\nThat compounds on spot, which is where the Arm discount is largest. Spot instances get\n\nreclaimed. A reclaimed Intel instance costs an image pull and a model download to replace. A\n\nreclaimed Arm instance costs a build. **The architecture with the cheaper hour is the one that\npays most to come back**, and on spot those are the same decision.\n\nIf you want the cheapest CUDA hour on AWS, rent the Arm box. If you want the cheapest CUDA\n\ntoken, rent the Intel one.\n\nTermination is cheap on the Intel side precisely because nothing was compiled:\n\n```\nterminate_g4dn_instance i-050dca2ed568dcc1b\n```\n\nThen confirm nothing billable is left — instances, spot requests, orphaned volumes:\n\n```\naws ec2 describe-instances --region us-east-1 \\\n  --filters \"Name=instance-state-name,Values=pending,running,shutting-down,stopping,stopped\" \\\n  --query 'length(Reservations[].Instances[])' --output text\naws ec2 describe-volumes --region us-east-1 \\\n  --filters Name=status,Values=available --query 'length(Volumes)' --output text\n0\n0\n```\n\nThe cheapest whole GPU CUDA instance on AWS is `g5g.xlarge`\n\nat 0.4200 per hour on demand and\n\n0.1458 on spot, with a Graviton2 host. `g4dn.xlarge`\n\nwith an Intel host costs 20 percent more\n\nper hour on demand and 144 percent more on spot.\n\nThe GPUs are the same part — identical compute capability, VRAM, memory clock, bus width and\n\nbandwidth, and vLLM independently allocated a KV cache of exactly 329,579 tokens on both.\n\nWhat the host CPU changes is not speed but which container image runs at all. vLLM ships SM 7.5\n\nkernels in its amd64 manifest and not in its arm64 manifest, so the Intel host runs the\n\npublished image while the Graviton2 host must build from source. The Arm instance also carries\n\nhalf the RAM per vCPU and cannot map the 9.54 GiB checkpoint at the `xlarge`\n\nsize without swap.\n\nMeasured, the Intel deployment saturates at 242.47 output tokens per second against 168.33, and\n\ncosts 0.603 per million output tokens against 1.366. The instance with the higher hourly rate\n\nproduces the cheaper token by 2.3 times, and on spot it reaches 0.408 per million.\n\nScope: one instance per family, one region, one model size, one sweep with no repeats. The Arm\n\nfigures come from a `g5g.4xlarge`\n\nrunning vLLM 0.27.2rc1 built from source under `vllm bench`\n\n; the Intel figures from a\n\nserve`g4dn.xlarge`\n\nrunning vLLM 0.28.0 from the published image\n\nunder this rig's harness. These are properties of these two deployments on their measurement\n\ndates.\n\nThe strategy for using MCP for Gemma 4 deployment on AWS EC2 G4dn was validated with an\n\nincremental step by step approach.", "url": "https://wpnews.pro/news/the-cheapest-cuda-gpu-on-aws-has-an-arm-cpu-and-you-probably-want-the-intel-one", "canonical_source": "https://dev.to/xbill/the-cheapest-cuda-gpu-on-aws-has-an-arm-cpu-and-you-probably-want-the-intel-one-325b", "published_at": "2026-08-31 01:40:08+00:00", "updated_at": "2026-08-31 01:51:38.416495+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure", "machine-learning"], "entities": ["AWS", "NVIDIA", "vLLM", "Gemma 4 E2B", "Graviton2", "T4", "T4G"], "alternates": {"html": "https://wpnews.pro/news/the-cheapest-cuda-gpu-on-aws-has-an-arm-cpu-and-you-probably-want-the-intel-one", "markdown": "https://wpnews.pro/news/the-cheapest-cuda-gpu-on-aws-has-an-arm-cpu-and-you-probably-want-the-intel-one.md", "text": "https://wpnews.pro/news/the-cheapest-cuda-gpu-on-aws-has-an-arm-cpu-and-you-probably-want-the-intel-one.txt", "jsonld": "https://wpnews.pro/news/the-cheapest-cuda-gpu-on-aws-has-an-arm-cpu-and-you-probably-want-the-intel-one.jsonld"}}