# g5g vs g6 for LLM Serving: the Same Code, and 3.7x the Throughput > Source: > Published: 2026-08-31 01:40:10+00:00 This article compares two AWS GPU instance families for serving a small language model, using a payload that is byte-identical on both. The older family loses **87% of decode** to dtype conversion, and nothing in any log, metric or health check says so. The code is here: [https://github.com/xbill9/gemma4-dev](https://github.com/xbill9/gemma4-dev) `g5g.2xlarge` pairs a Graviton2 (aarch64) host with an NVIDIA **T4G** — Turing, SM 7.5. `g6.2xlarge` is x86_64 with an NVIDIA **L4** — Ada, SM 8.9. Both were run on spot. `g5g.2xlarge` | `g6.2xlarge` | | |---|---|---| | GPU | NVIDIA T4G — Turing, SM 7.5 | NVIDIA L4 — Ada, SM 8.9 | | GPU memory | 15,360 MiB | 23,034 MiB | | Host | Graviton2, aarch64 | x86_64, `us-east-1d` | | Purchase model | spot | spot | | Run cited | `2026-08-28-full-run-cached-g5g` | `2026-08-28-first-serve-g6` | The workload is `google/gemma-4-E2B-it` , the dense reference checkpoint, served through a hand-written pure-JAX port — no PyTorch, no vLLM, no `torch_xla` . **The payload is byte-identical on both instances**: build id `51bc52c9e2e9` , config `ple4 + int8_lm_head` , and `tpu_jax_weight_bytes` reading **6,155,450,950** on each. Only the chip and its host differ. `g5g.2xlarge` and `g6.2xlarge` `google/gemma-4-E2B-it` `jax[cuda13]` supplies CUDA as wheelsThere is no build step on either instance. `jax[cuda13]` ships wheels carrying CUDA, including aarch64 wheels for the Graviton2 host. ``` Install: 117 s, with the cache restore included ``` XLA's persistent compilation cache is pushed to S3 and restored on boot. On the g5g rig it restores **805 files / 12 MB in 6 seconds** onto a fresh instance from a box that had already been terminated. `max_new_tokens` is a `static_argnames` entry, so `(bucket, max_tokens)` is the compiled shape. The first request off a fresh engine pays XLA compilation. On the g5g that first request took **18.06 s against 4.50 s warm** — a 4.0x whole-request ratio, from `2026-08-21-cuda13-py314-g5g` . A harness that skips warm-up misreports the instance by a factor of four. Note that the 56x figure quoted from the earlier first-serve baseline is **TTFT specifically**, which is a different measurement and not interchangeable with the 64 output tokens, concurrency 1, 3 repeats per cell, median reported. "Gauge" is the engine's steady-state decode counter; "end-to-end" is wall time over the whole request including prefill. | Input tokens | 🥈 g5g gauge | g5g end-to-end | 🥇 g6 gauge | g6 end-to-end | |---|---|---|---|---| | 41 | 12.9 tok/s | 12.43 tok/s | 48.5 tok/s | 46.23 tok/s | | 521 | 13.0 tok/s | 11.28 tok/s | 48.4 tok/s | 42.87 tok/s | | 2,057 | 12.9 tok/s | 8.22 tok/s | 48.3 tok/s | 34.57 tok/s | | 3,593 | — | — | 48.3 tok/s | 27.55 tok/s | **3.7x on decode**, for the same code and the same weights. Decode moves 0.8% across a 50x context range on the g5g and 0.4% on the g6. End-to-end falls hard on both — 12.43 to 8.22, and 46.23 to 27.55. That fall is prefill being linear in the padded bucket, not decode degrading. They are two different claims, and conflating them makes a benchmark a lie. A cost proportional to the **weights** rather than the context produces exactly this shape, which is why the KV cache is not what sets decode speed on either instance. Usable context on the g5g is `MAX_MODEL_LEN=4096` , and that is the honest number: 4,105 prompt tokens serve, 5,120 fails on a prefill transient. Profiling with xprof, 20 decode steps with the service stopped: | 🥈 g5g / T4G (SM 7.5) | 🥇 g6 / L4 (SM 8.9) | | |---|---|---| | dtype conversion | 54.1% | 0.0% | fp32 `gemvx` | 32.8% | absent | | Tensor Core | 0.0% | 0.0% | | Total kernel time | 1,466.0 ms | 362.8 ms | | Decode, gauge | 12.9 tok/s | 48.4 tok/s | | Peak HBM bandwidth | 298.083 GiB/s | 279.441 GiB/s | | Share of bandwidth roofline | 26% | ~100% | **87% of decode on the g5g is not math.** It is dtype conversion plus an fp32 `gemvx` path. The instance runs at 26% of its own memory-bandwidth roofline; the g6 runs at roughly all of it. **A wrong compute dtype does not raise. It emulates.** `bfloat16` on a pre-Ampere GPU does not fail — XLA routes it through fp32 and decode quietly disappears into conversion. Turing has neither bf16 nor fp8. Its only real 16-bit datapath is `float16` . So the port reads the live compute capability off the device rather than trusting a config file: ``` COMPUTE_DTYPE = float16 if IS_PRE_AMPERE else bfloat16 ``` The server states its decision on the first line it emits, so a misconfigured instance is one `grep` away rather than a mystery in the throughput: ``` INFO ports.gemma4.jax_e_model: jax_e_model device policy: platform=gpu compute_capability=8.9 compute_dtype=bfloat16 pallas_interpret=False ``` The obvious hypothesis was bf16 weights on a chip with no bf16 datapath, so the checkpoint was converted to float16 host-side and re-run. Parameter dtypes read `{'float16': 541, 'uint8': 1, 'int8': 1}` — and conversion **stayed at 54.0%**. Storage dtype was never the problem. The fp32 `gemvx` line is the tell: XLA was round-tripping through fp32 regardless of what the file on disk said. Only a card whose compute dtype matches its storage dtype removes it, which is what the g6 shows. The measurement reproduces: the same profile on a different instance, a different AMI and a restored cache landed at 1466.0 ms against 1467.1 ms. The g5g serves correctly the entire time. It returns HTTP `200` , valid completions, and a healthy `/health` . It is simply doing four times more work than it needs to. The related trap in this engine is a padding-eviction bug in the KV ring cache, whose failure mode is a token loop returning a clean `200` with `status: "success"` and output like `The The The The` . Nothing in the logs or the metrics is red. Only a degeneracy check on the response body catches it, which the server now runs on every request. **On this stack, HTTP 200 is not evidence of anything.** Both instances are spot and are terminated after collection. The XLA cache is pushed to S3 first, which is what makes the 6-second restore on the next fresh instance possible. The goal of this article was to compare two AWS GPU instance families for serving a small language model with a payload held byte-identical across both. The key to the solution was profiling decode rather than trusting throughput alone. The measured results were: `g6.2xlarge` over `g5g.2xlarge` Scope: two spot instances, one in `us-east-1d` , each measured once with 3 repeats per sweep cell and medians reported. The two differ in host architecture (aarch64 against x86_64) and base image as well as in GPU, so this is not a single-variable experiment; the payload is byte-identical across them — build `51bc52c9e2e9` , the same config and the same 6,155,450,950 bytes of weights — which is the basis for attributing the difference to the chip. The g5g profile was reproduced on a second instance at 1466.0 ms against 1467.1 ms; the g6 profile was measured once. Price and price-per-token were not measured and are not claimed here. The strategy for using MCP for Gemma 4 serving across AWS GPU instance families was validated with an incremental step by step approach.