{"slug": "three-gemma-4-deployments-on-one-t4g-for-under-3-what-the-runtime-changes-and-it", "title": "Three Gemma 4 Deployments on One T4G for Under $3: What the Runtime Changes, and What It Doesn't", "summary": "A developer benchmarked three Gemma 4 deployments on a single AWS T4G GPU for under $3, comparing vLLM, JAX, and PyTorch runtimes. The project built a common harness to measure decode throughput, revealing that previous comparisons were invalid due to inconsistent metrics. The exercise cost under three dollars and caught five incorrect assumptions through measurement.", "body_md": "This article provides a step by step comparison of three Gemma 4 deployments on a single AWS hosted GPU enabled system. A suite of Python MCP tools is built to simplify management of each deployment, and one benchmark harness is shared across all three so that the runtime is the only variable.\n\n[https://github.com/xbill9/gemma4-dev](https://github.com/xbill9/gemma4-dev)\n\nThe whole exercise cost under three dollars, and that is the part worth keeping. Nineteen instances and about four and a half instance-hours bought three serving sweeps, nine timed boots and a handful of A/B restarts. It also bought five wrong claims, each caught by measuring instead of reasoning. On hardware where a run is expensive, the cheapest of those five would have shipped with a caveat attached.\n\nThree rigs in this monorepo serve `google/gemma-4-E2B-it`\n\non an AWS G5g instance. One runs vLLM, one runs a pure JAX port, one runs PyTorch with transformers. The hardware is identical and only the runtime slot moves, so this should be the cleanest A/B available.\n\nFor months it was not, because each rig measured itself with its own harness and quoted its own number. Three harnesses computing three statistics is not a comparison.\n\n`us-east-1`\n\n. Each `g5g.2xlarge`\n\nis 8 vCPU, so 16 vCPU\nof spot quota runs two at once.`AmazonSSMManagedInstanceCore`\n\nplus read on the Hugging Face token secret.`EnvironmentFile`\n\nand never appears in user data.`boto3`\n\nand the standard credential chain. No AWS CLI shell-outs, no inbound SSH rule, and\nno private key anywhere in the flow.| Instance |\n`g5g.2xlarge` — 8 vCPU, 16 GiB host |\n| Host CPU | AWS Graviton2, aarch64 |\n| GPU | 1x NVIDIA T4G, Turing, SM 7.5 |\n| GPU memory | 15,360 MiB per `nvidia-smi` ; AWS lists 16,384 nominal |\n\nG5g is the only family AWS ships that puts an NVIDIA GPU behind a Graviton host, which makes it the only place to get aarch64 and compute capability 7.5 together.\n\n`google/gemma-4-E2B-it`\n\nis the reference instruction-tuned release. It is 2B effective from about 5B total, and the split matters here: most of what is resident is a per-layer-embedding table that decode reads as a gather and never streams through a matmul.\n\nThe dense checkpoint fits. 9.5 GiB of float16 weights go into 15,360 MiB of device memory with room for the KV cache, which at roughly 18 KiB per token is tens of megabytes at this context and never the binding constraint.\n\n| runtime | engine | how it serves |\n|---|---|---|\n| vLLM | v0.27.2rc0, built from source for `sm_75`\n|\ncontinuous batching, paged KV, prefix caching |\n| JAX | this project's own port | hand-written KV ring with a bucket ladder |\n| PyTorch |\n`AutoModelForCausalLM` + transformers |\n`past_key_values` , one request at a time |\n\nTuring has no bfloat16 datapath, so all three run float16. It has no fp8 either, which rules out the KV-cache tricks that work on newer parts.\n\n```\ncheck_g5g_quotas\n| Quota | vCPUs |\n| Running On-Demand G and VT instances (vCPU) | 16 |\n| All G and VT Spot Instance Requests (vCPU) | 16 |\n\n`g5g.2xlarge` needs 8 vCPUs.\n```\n\nThat is the constraint behind every launch below: two rigs in parallel, and no more.\n\nThe sweep script read its throughput figure straight out of the response body:\n\n```\n\"decode_tps\": usage.get(\"decode_tokens_per_second\", 0.0),\n```\n\n`usage.decode_tokens_per_second`\n\nis a field our own servers invent. vLLM does not emit it, and neither does anything else, so the harness could not be pointed at the vLLM rig at all. The three-way comparison had never actually been run.\n\nRe-running a rig does not fix that. Only a common statistic does.\n\nEvery OpenAI-compatible server streams, so the portable measurement is the gap between tokens on the wire.\n\n```\npython3 sweep.py --help | grep -A2 decode-source\n--decode-source {auto,usage,stream,both}\n                        where the decode figure comes from; see the module\n                        docstring\n```\n\nThe `stream`\n\npath uses `vllm bench serve`\n\n's exact TPOT definition, `(latency - ttft) / (output_len - 1)`\n\n, so a number from this harness is directly comparable to that tool's published figures. `auto`\n\nprobes the endpoint once and picks `both`\n\nwhere the server emits its own gauge, `stream`\n\nwhere it does not.\n\nNo, and that is worth a measurement rather than an assumption. Running `both`\n\nmeasures each rig's offset between the two statistics.\n\n| rig | server gauge | client stream | stream/gauge |\n|---|---|---|---|\n| JAX | 12.962 | 12.687 | 0.9799 |\n| PyTorch | 10.814 | 10.243 | 0.9543 |\n\nTwo percent against 4.6 percent, on the same day and the same instance shape. Borrowing one rig's ratio to convert the other's number would inject a 2.6 percent error into a comparison whose smallest interesting gap is 24 percent. The cross-rig table below is therefore built from `stream`\n\nthroughout.\n\nThe JAX server emitted its decode gauge with one decimal place:\n\n```\nf'tpu_jax_decode_tokens_per_second{{model=\"{MODEL_ID}\"}} {METRICS[\"last_tokens_per_second\"]:.1f}',\n```\n\nAt about 13 tok/s, one decimal is 0.78 percent resolution. Every sweep that rig had produced showed all three repeats of a cell as byte-identical: 12.8, 12.8, 12.8. That is not reproducibility, it is the measurement floor. The rig had been used to argue about two percent effects it could not resolve.\n\nTwo characters fixed it. The first run afterwards reads 12.962, where before it would have said 13.0.\n\nCapacity for the whole G5g family was exhausted across all four availability zones several times, so the launcher cycles them with a sixty second backoff.\n\n```\n[12:51:45] round 5 us-east-1c: ❌ AWS InsufficientInstanceCapacity\n[12:52:47] us-east-1a: ✅ Launching `i-02e79988a6cbeecbf` (g5g.2xlarge, spot, 1x T4G) in `us-east-1`.\n```\n\nThe walkthrough from here follows the PyTorch rig on `i-02e79988a6cbeecbf`\n\n; the other two run the same steps against their own instances. All three landed in `us-east-1a`\n\nwithin hours of each other. Note that AWS names the other zones as available in every one of those errors — that text describes on-demand capacity and says nothing about spot.\n\nCloud-init installs the runtime and then backgrounds itself, so the progress tool reports cloud-init's own state as well as the install log. A dead bootstrap and a slow one must not render identically.\n\n```\nget_install_progress i-02e79988a6cbeecbf\nINSTALL COMPLETE\n--- cloud-init ---\nstatus: done\nerrors: []\n```\n\nThis is a wheel install, not a build. Across the three timed boots the install finishes a median 113.55 s after launch, against the hours the vLLM rig needs for a from-source build.\n\nA config flag being accepted proves nothing, so the probe runs a real matmul on the device.\n\n```\nverify_gpu_arch i-02e79988a6cbeecbf\nNVIDIA T4G, 7.5, 15360 MiB\ntorch: 2.12.0+cu132\narch_list: ['sm_75', 'sm_80', 'sm_90', 'sm_100', 'sm_110', 'sm_120']\ncapability: (7, 5)\ncompute_dtype: float16\nfp16 matmul ok: True\n\n✅ torch reached the GPU and a real fp16 matmul executed.\n```\n\nThe DLAMI's torch carries `sm_75`\n\n. Upstream PyPI aarch64 wheels do not, so a `pip install torch`\n\non this box would serve on CPU without saying so.\n\nThe payload is the rig's own source, shipped over SSM as a gzipped tarball because user data caps at 16 KiB.\n\n```\ndeploy_torch_server i-02e79988a6cbeecbf\n✅ Deployed 3 files (16 KiB base64) to `i-02e79988a6cbeecbf`.\n\nPayload root: `/home/xbill/gemma4-dev/gpu-pytorch-g5g-2b`\nBuild id: `060a572aeb55` — verify_model_health checks the running server reports this.\n```\n\nA non-empty reply is not evidence of health. One sibling was once measured answering `': ok: ok: ok…'`\n\n, so the check reads the server's own degenerate-response counter either side of its probe, and compares the served build id against the local payload.\n\n```\nverify_model_health i-02e79988a6cbeecbf\n✅ health=200 tokens=5 reply='ok'\n\n- Degenerate (server's own verdict on the full text): **no**\n- Build id served: `060a572aeb55`\n- Build id matches the local payload (`060a572aeb55`).\n```\n\nThe same command runs against all three rigs. Only the endpoint changes.\n\n```\npython3 sweep.py --base http://<ip>:8000/v1 --out benchmarks/runs/<run> \\\n  --contexts 64,512,1024,2048,3072,3800 --outputs 32,128 --repeats 3 \\\n  --decode-source both\nphp\ndecode-source: both -> both\nctx~512 out=32: in=633 out=32 decode=10.96 tok/s  e2e=9.83 tok/s (warmup 11.08)  stream/usage=0.9616\nctx~2048 out=128: in=2501 out=89 decode=10.66 tok/s  e2e=9.16 tok/s (warmup 10.61)  stream/usage=0.9540\nctx~3800 out=32: FAILED HTTP Error 400 {\"detail\":\"prompt is 4630 tokens and the context\n  bound is 4096, leaving no room to decode. Start the server with a larger --seq.\"}\n```\n\nCells that cannot exist on the hardware are recorded as `infeasible`\n\nrather than dropped. An absent cell is indistinguishable from an untried one, which is how a sweep overstates its own coverage.\n\n| runtime | decode tok/s | % of ceiling | vs PyTorch | cells | |\n|---|---|---|---|---|---|\n| 🥇 | vLLM v0.27.2rc0 | 32.53 | 53.0% | 3.18x | 12/12 |\n| 🥈 | JAX | 12.69 | 20.7% | 1.24x | 10/12 |\n| 🥉 | PyTorch + transformers | 10.24 | 16.7% | 1.00x | 10/12 |\n\nThe ceiling is arithmetic, not a measurement. E2B streams 4.514 GB of weights per decode step against a measured 277 GB/s, giving 16.30 ms per step and **61.4 tok/s**.\n\nThe PLE table is excluded from that figure because it is a gather and never a matmul. Quantising it from 9.257 GB to 5.752 GB moved decode by 0.00 tok/s across three cells, which is what confirms it never streams.\n\nAll three runtimes sit far below the ceiling, so **none of them is bandwidth-bound at batch one**. The PyTorch profile shows why: about 5,650 kernel launches per step at one to three microseconds each, on a chip whose launch overhead is five to ten.\n\nTime to first token was the result of the run, right up until it was checked.\n\n| input tok | vLLM | JAX | PyTorch |\n|---|---|---|---|\n| 92 | 103 ms | 225 ms | 164 ms |\n| 1,259 | 118 ms | 1,615 ms | 657 ms |\n| 3,746 | 178 ms | 5,352 ms | 2,339 ms |\n\nA 30x advantage, far larger than the 3.2x on decode. It is also impossible. Prefill at 3,746 tokens is roughly 14 TFLOP against a T4G's realistic 20 to 30 TFLOP/s, which is 460 ms at best. vLLM's row says 178 ms.\n\nIt did not. vLLM ships `enable_prefix_caching=True`\n\n, and its own metrics say so:\n\n```\ngrep -E \"^vllm:prefix_cache_(queries|hits)_total\" metrics.prom\nvllm:prefix_cache_queries_total{engine=\"0\"} 102898.0\nvllm:prefix_cache_hits_total{engine=\"0\"}     97440.0\n```\n\n**A 94.7 percent hit rate.** vLLM genuinely prefilled 5.3 percent of the tokens it was sent, because the harness reused one prompt for a cell's warm-up and all three repeats. Neither sibling has a prefix cache, so both paid full prefill every time.\n\nThe fix places a nonce first in the prompt, since a shared prefix is exactly what the cache keys on and a trailing nonce would not have defeated it. That property is now a unit test.\n\nStrip the contaminated column and a real result remains. Neither of the other two runtimes caches prefixes, and both saw identical prompts.\n\n| TTFT slope | at 3,746 tokens | |\n|---|---|---|\n| JAX | 1.403 ms/token | 5,352 ms |\n| PyTorch | 0.595 ms/token | 2,339 ms |\n\n**JAX prefills 2.4x slower than PyTorch**, consistently across all five shared context lengths. On an interactive workload with real context that dominates user-visible latency, and it runs in the opposite direction to the 1.24x decode advantage the same rig enjoys.\n\nNine cold boots, three per runtime, plus nine warm reloads. The start line is the moment `run_instances`\n\nreturns an id, because capacity wait measures AWS rather than the rig.\n\n| runtime | cold boot | spread | warm reload | cold/warm | |\n|---|---|---|---|---|---|\n| 🥇 | PyTorch | 195.2 s | 11.8% | 24.5 s | 8.0x |\n| 🥈 | JAX | 242.2 s | 11.5% | 74.1 s | 3.3x |\n| 🥉 | vLLM | 1417.8 s | 12.6% | 264.3 s | 5.4x |\n\n**vLLM takes 23m 38s to serve, from a prebuilt AMI that downloads nothing.** PyTorch installs its runtime from wheels and pulls the 9.54 GiB checkpoint over the network, and is still 7.3x faster.\n\nBoot variance is 11.5, 11.8 and 12.6 percent — too consistent across three different runtimes to be a property of any of them. This family's decode noise floor is 1.7 percent, measured by running an identical build on two hosts, so boot is about seven times noisier and a single boot measurement is close to worthless.\n\nNot on every runtime, which is why the harness records two stop lines.\n\n| runtime | first completion, cold | warm |\n|---|---|---|\n| vLLM | 0.5 s | 0.2 s |\n| PyTorch | 1.0 s | 0.7 s |\n| JAX | 22.9 s | 9.2 s |\n\nJAX returns health 200 and then compiles XLA per shape bucket on the first real request. Quoting health alone understates its time to serving by 22 seconds, and the cost does not vanish when warm. vLLM is the mirror image: slowest to boot, fastest first token, because graph capture is paid before the port binds.\n\n| runtime | to change serving code |\n|---|---|\n| PyTorch | ship 3 files over SSM, restart — 25 s |\n| JAX | same mechanism, plus 9.2 s of compile — 83 s |\n| vLLM | no deploy path exists: rebuild from source, ~67 min, and reapply an out-of-tree Turing patch |\n\nvLLM's 264 s warm figure is a `systemctl restart`\n\n, not a code change, so it flatters the comparison. **vLLM wins decode 3.2x and loses the iteration loop by 3 to 100x.**\n\nvLLM's cold boot is dominated by weight loading: 468 to 561 seconds across four measurements. Explaining it took five attempts, four of which were wrong.\n\n`g5g.2xlarge`\n\nneeds no swapfile and buys that time back.`g5g.4xlarge`\n\nboot: available\nRAM 11.19 to 26.49 GiB, weight loading 546 to 468 s, and total boot 4.7 percent lower,\ninside the noise band.`--safetensors-load-strategy=prefetch`\n\n, which is 1.3 percent and therefore nothing.What that last run did find is the useful part.\n\n| weight load | n | |\n|---|---|---|\n| cold boot, fresh instance | 468-561 s | 4 |\n| warm restart, same box | 32-76 s | 3 |\n\nSame volume, same filesystem, same engine, differing only in whether the blocks had been read once. 9.54 GiB in 468 s is about 21 MiB/s, which is absurd for gp3 steady state and ordinary for first-touch reads against a snapshot-backed volume.\n\nTheory five is EBS lazily hydrating the volume from the AMI snapshot, and it is written down as untested. Given the strike rate it does not get promoted by reasoning.\n\nThe first boot campaign was discarded and re-run. Two independent instances had reported 214.4 s and 125.1 s, identical to the tenth. That is not consistency. It is a five second poll quantising two similar boots onto the same tick.\n\nThe data was not wrong; the campaign log shows 215 s and 216 s of wall clock. It was unusably coarse. Health polling went to half a second, and the next pair of boots came in at 216.90 s and 193.86 s — an 11.9 percent spread the old harness could not see.\n\n| $/hr | |\n|---|---|\n`g5g.2xlarge` on-demand |\n$0.556 |\n| spot, measured across four AZs | $0.3813 - $0.4416 |\n\nThe whole exercise — 19 instances, 4.06 hours of `g5g.2xlarge`\n\nplus 0.38 of `g5g.4xlarge`\n\n, a serving sweep, three cross-rig runs, nine boots and three A/B restarts — came to under $3. That is arithmetic rather than a bill: AWS drops terminated instances after an hour, so the derivation in `cost_derivation.md`\n\nbounds it at $1.84 all-spot and $2.68 all-on-demand.\n\nThat 26 to 46 percent premium is the entire spot versus on-demand decision on this hardware, which is to say there is not one. Try spot, fall back, keep working; the automatic fallback cost about $0.24 across a nine-boot campaign.\n\n**Cheap hardware is what made the method possible, not merely affordable.** Discarding a completed campaign over a poll-interval bug cost twenty minutes and pennies. Where a run is expensive, the same discovery argues for shipping the numbers with a caveat instead.\n\n| service | what it does here |\n|---|---|\n| EC2 | the `g5g.2xlarge` instances, spot and on-demand |\n| Systems Manager | every remote command; there is no inbound SSH rule and no private key |\n| Secrets Manager | the Hugging Face token, fetched at boot into a root-only `EnvironmentFile`\n|\n| IAM | one instance profile, `AmazonSSMManagedInstanceCore` plus read on that one secret |\n| EBS | gp3 root volumes, and the AMI snapshot behind the vLLM boot mystery |\n\nEvery instance is terminated as soon as its artifacts are captured. There is no built image to lose, only a pip install and a model cache.\n\n```\nterminate_g5g_instance i-02e79988a6cbeecbf\n🗑️ Terminating `i-02e79988a6cbeecbf`. Relaunch costs a pip install, not a build.\naws ec2 describe-instances --filters \"Name=instance-state-name,Values=running\" \\\n  --query 'Reservations[].Instances[?starts_with(InstanceType,`g5g`)].InstanceId' \\\n  --output text | grep . || echo \"🟢 none running\"\n🟢 none running\n```\n\nEverything above is concurrency one, which makes it a latency comparison rather than a serving one. Continuous batching is vLLM's whole value proposition and it is untested here.\n\nOnly vLLM can serve concurrently at all today. The PyTorch server holds an `asyncio.Lock`\n\nwith the comment *\"one GPU, one process -> serialize requests\"*, and the JAX rig has no batching machinery whatever. The engine-level batch sweep suggests what is on the table — batch eight reaches 84.16 tok/s for an extra 0.258 GB, with per-step time growing two percent across an eight-fold batch — but that number never leaves the engine.\n\nThree further gaps: the JAX leg ran its shipped quantised configuration against two dense runtimes; no output-quality axis was measured at all, on a comparison where one runtime uses a deliberately lossy LM head; and every TTFT figure predates the prompt-uniqueness fix, so only the JAX versus PyTorch half of that table is sound.\n\nThe goal of this article was to compare three inference runtimes on identical silicon without the harness being a variable. The key to the solution was a single client-side statistic that every OpenAI-compatible server can produce. The measured results were:\n\nScope: the decode and boot numbers are one `g5g.2xlarge`\n\nper runtime in `us-east-1a`\n\non 2026-08-31, three repeats per cell and three repeats per boot, mixed spot and on-demand with the market recorded per run. Three things differed between the legs and are named where they matter: vLLM ran `max_model_len`\n\n16384 against 4096 for the other two; the JAX leg ran its shipped quantised configuration against two dense runtimes; and vLLM booted from a prebuilt AMI carrying its model cache while the other two installed from wheels and downloaded the checkpoint, which is the point of the boot comparison rather than a flaw in it. Two runs sit outside that envelope and say so in the text: the RAM test was a single `g5g.4xlarge`\n\n, and the prefetch A/B ran 2026-09-01 on on-demand after a spot reclamation killed the first attempt. Decode is unaffected by the prefix-cache issue, which changes prefill only.\n\nThe strategy for using MCP for multi-runtime comparison was validated with a incremental step by step approach.", "url": "https://wpnews.pro/news/three-gemma-4-deployments-on-one-t4g-for-under-3-what-the-runtime-changes-and-it", "canonical_source": "https://dev.to/aws-builders/three-gemma-4-deployments-on-one-t4g-for-under-3-what-the-runtime-changes-and-what-it-doesnt-2cin", "published_at": "2026-09-01 18:25:50+00:00", "updated_at": "2026-09-01 18:54:24.363699+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "mlops"], "entities": ["Google", "AWS", "vLLM", "JAX", "PyTorch", "Gemma 4", "T4G", "Graviton2"], "alternates": {"html": "https://wpnews.pro/news/three-gemma-4-deployments-on-one-t4g-for-under-3-what-the-runtime-changes-and-it", "markdown": "https://wpnews.pro/news/three-gemma-4-deployments-on-one-t4g-for-under-3-what-the-runtime-changes-and-it.md", "text": "https://wpnews.pro/news/three-gemma-4-deployments-on-one-t4g-for-under-3-what-the-runtime-changes-and-it.txt", "jsonld": "https://wpnews.pro/news/three-gemma-4-deployments-on-one-t4g-for-under-3-what-the-runtime-changes-and-it.jsonld"}}