{"slug": "g5g-vs-g6-for-llm-serving-the-same-code-and-3-7x-the-throughput", "title": "g5g vs g6 for LLM Serving: the Same Code, and 3.7x the Throughput", "summary": "A developer benchmarked AWS g5g and g6 GPU instances for serving Google's Gemma-4-E2B language model with identical code and weights, finding the g6 delivers 3.7x the decode throughput. Profiling revealed the g5g's T4G GPU spends 87% of decode time on dtype conversion and an fp32 gemvx path, running at only 26% of its memory bandwidth roofline, while the g6's L4 GPU achieves near-peak bandwidth. The findings highlight how hidden inefficiencies can skew LLM serving performance benchmarks.", "body_md": "This article compares two AWS GPU instance families for serving a small language\n\nmodel, using a payload that is byte-identical on both. The older family loses\n\n**87% of decode** to dtype conversion, and nothing in any log, metric or health\n\ncheck says so.\n\nThe code is here:\n\n[https://github.com/xbill9/gemma4-dev](https://github.com/xbill9/gemma4-dev)\n\n`g5g.2xlarge`\n\npairs a Graviton2 (aarch64) host with an NVIDIA **T4G** — Turing,\n\nSM 7.5. `g6.2xlarge`\n\nis x86_64 with an NVIDIA **L4** — Ada, SM 8.9. Both were run\n\non spot.\n\n`g5g.2xlarge` |\n`g6.2xlarge` |\n|\n|---|---|---|\n| GPU | NVIDIA T4G — Turing, SM 7.5 |\nNVIDIA L4 — Ada, SM 8.9 |\n| GPU memory | 15,360 MiB | 23,034 MiB |\n| Host | Graviton2, aarch64\n|\nx86_64, `us-east-1d`\n|\n| Purchase model | spot | spot |\n| Run cited | `2026-08-28-full-run-cached-g5g` |\n`2026-08-28-first-serve-g6` |\n\nThe workload is `google/gemma-4-E2B-it`\n\n, the dense reference checkpoint, served\n\nthrough a hand-written pure-JAX port — no PyTorch, no vLLM, no `torch_xla`\n\n.\n\n**The payload is byte-identical on both instances**: build id `51bc52c9e2e9`\n\n,\n\nconfig `ple4 + int8_lm_head`\n\n, and `tpu_jax_weight_bytes`\n\nreading **6,155,450,950**\n\non each. Only the chip and its host differ.\n\n`g5g.2xlarge`\n\nand `g6.2xlarge`\n\n`google/gemma-4-E2B-it`\n\n`jax[cuda13]`\n\nsupplies CUDA as wheelsThere is no build step on either instance. `jax[cuda13]`\n\nships wheels carrying\n\nCUDA, including aarch64 wheels for the Graviton2 host.\n\n```\nInstall: 117 s, with the cache restore included\n```\n\nXLA's persistent compilation cache is pushed to S3 and restored on boot. On the\n\ng5g rig it restores **805 files / 12 MB in 6 seconds** onto a fresh instance from\n\na box that had already been terminated.\n\n`max_new_tokens`\n\nis a `static_argnames`\n\nentry, so `(bucket, max_tokens)`\n\nis the\n\ncompiled shape. The first request off a fresh engine pays XLA compilation.\n\nOn the g5g that first request took **18.06 s against 4.50 s warm** — a 4.0x\n\nwhole-request ratio, from `2026-08-21-cuda13-py314-g5g`\n\n. A harness that skips\n\nwarm-up misreports the instance by a factor of four.\n\nNote that the 56x figure quoted from the earlier first-serve baseline is **TTFT\nspecifically**, which is a different measurement and not interchangeable with the\n\n64 output tokens, concurrency 1, 3 repeats per cell, median reported. \"Gauge\" is\n\nthe engine's steady-state decode counter; \"end-to-end\" is wall time over the whole\n\nrequest including prefill.\n\n| Input tokens | 🥈 g5g gauge | g5g end-to-end | 🥇 g6 gauge | g6 end-to-end |\n|---|---|---|---|---|\n| 41 | 12.9 tok/s | 12.43 tok/s | 48.5 tok/s |\n46.23 tok/s |\n| 521 | 13.0 tok/s | 11.28 tok/s | 48.4 tok/s |\n42.87 tok/s |\n| 2,057 | 12.9 tok/s | 8.22 tok/s | 48.3 tok/s |\n34.57 tok/s |\n| 3,593 | — | — | 48.3 tok/s |\n27.55 tok/s |\n\n**3.7x on decode**, for the same code and the same weights.\n\nDecode moves 0.8% across a 50x context range on the g5g and 0.4% on the g6.\n\nEnd-to-end falls hard on both — 12.43 to 8.22, and 46.23 to 27.55.\n\nThat fall is prefill being linear in the padded bucket, not decode degrading. They\n\nare two different claims, and conflating them makes a benchmark a lie.\n\nA cost proportional to the **weights** rather than the context produces exactly\n\nthis shape, which is why the KV cache is not what sets decode speed on either\n\ninstance.\n\nUsable context on the g5g is `MAX_MODEL_LEN=4096`\n\n, and that is the honest number:\n\n4,105 prompt tokens serve, 5,120 fails on a prefill transient.\n\nProfiling with xprof, 20 decode steps with the service stopped:\n\n| 🥈 g5g / T4G (SM 7.5) | 🥇 g6 / L4 (SM 8.9) | |\n|---|---|---|\n| dtype conversion | 54.1% | 0.0% |\nfp32 `gemvx`\n|\n32.8% | absent |\n| Tensor Core | 0.0% | 0.0% |\n| Total kernel time | 1,466.0 ms | 362.8 ms |\n| Decode, gauge | 12.9 tok/s | 48.4 tok/s |\n| Peak HBM bandwidth | 298.083 GiB/s | 279.441 GiB/s |\n| Share of bandwidth roofline | 26% | ~100% |\n\n**87% of decode on the g5g is not math.** It is dtype conversion plus an fp32\n\n`gemvx`\n\npath. The instance runs at 26% of its own memory-bandwidth roofline; the\n\ng6 runs at roughly all of it.\n\n**A wrong compute dtype does not raise. It emulates.** `bfloat16`\n\non a pre-Ampere\n\nGPU does not fail — XLA routes it through fp32 and decode quietly disappears into\n\nconversion.\n\nTuring has neither bf16 nor fp8. Its only real 16-bit datapath is `float16`\n\n. So the\n\nport reads the live compute capability off the device rather than trusting a config\n\nfile:\n\n```\nCOMPUTE_DTYPE = float16 if IS_PRE_AMPERE else bfloat16\n```\n\nThe server states its decision on the first line it emits, so a misconfigured\n\ninstance is one `grep`\n\naway rather than a mystery in the throughput:\n\n```\nINFO ports.gemma4.jax_e_model: jax_e_model device policy: platform=gpu\ncompute_capability=8.9 compute_dtype=bfloat16 pallas_interpret=False\n```\n\nThe obvious hypothesis was bf16 weights on a chip with no bf16 datapath, so the\n\ncheckpoint was converted to float16 host-side and re-run. Parameter dtypes read\n\n`{'float16': 541, 'uint8': 1, 'int8': 1}`\n\n— and conversion **stayed at 54.0%**.\n\nStorage dtype was never the problem. The fp32 `gemvx`\n\nline is the tell: XLA was\n\nround-tripping through fp32 regardless of what the file on disk said. Only a card\n\nwhose compute dtype matches its storage dtype removes it, which is what the g6\n\nshows.\n\nThe measurement reproduces: the same profile on a different instance, a different\n\nAMI and a restored cache landed at 1466.0 ms against 1467.1 ms.\n\nThe g5g serves correctly the entire time. It returns HTTP `200`\n\n, valid completions,\n\nand a healthy `/health`\n\n. It is simply doing four times more work than it needs to.\n\nThe related trap in this engine is a padding-eviction bug in the KV ring cache,\n\nwhose failure mode is a token loop returning a clean `200`\n\nwith\n\n`status: \"success\"`\n\nand output like `The The The The`\n\n. Nothing in the logs or the\n\nmetrics is red. Only a degeneracy check on the response body catches it, which the\n\nserver now runs on every request.\n\n**On this stack, HTTP 200 is not evidence of anything.**\n\nBoth instances are spot and are terminated after collection. The XLA cache is\n\npushed to S3 first, which is what makes the 6-second restore on the next fresh\n\ninstance possible.\n\nThe goal of this article was to compare two AWS GPU instance families for serving a\n\nsmall language model with a payload held byte-identical across both. The key to the\n\nsolution was profiling decode rather than trusting throughput alone. The measured\n\nresults were:\n\n`g6.2xlarge`\n\nover `g5g.2xlarge`\n\nScope: two spot instances, one in `us-east-1d`\n\n, each measured once with 3 repeats\n\nper sweep cell and medians reported. The two differ in host architecture (aarch64\n\nagainst x86_64) and base image as well as in GPU, so this is not a single-variable\n\nexperiment; the payload is byte-identical across them — build `51bc52c9e2e9`\n\n, the\n\nsame config and the same 6,155,450,950 bytes of weights — which is the basis for\n\nattributing the difference to the chip. The g5g profile was reproduced on a second\n\ninstance at 1466.0 ms against 1467.1 ms; the g6 profile was measured once. Price\n\nand price-per-token were not measured and are not claimed here.\n\nThe strategy for using MCP for Gemma 4 serving across AWS GPU instance families was\n\nvalidated with an incremental step by step approach.", "url": "https://wpnews.pro/news/g5g-vs-g6-for-llm-serving-the-same-code-and-3-7x-the-throughput", "canonical_source": "https://dev.to/xbill/g5g-vs-g6-for-llm-serving-the-same-code-and-37x-the-throughput-5a39", "published_at": "2026-08-31 01:40:10+00:00", "updated_at": "2026-08-31 01:51:31.325361+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-research", "developer-tools"], "entities": ["AWS", "NVIDIA", "Google", "Gemma-4-E2B", "T4G", "L4", "Graviton2", "JAX"], "alternates": {"html": "https://wpnews.pro/news/g5g-vs-g6-for-llm-serving-the-same-code-and-3-7x-the-throughput", "markdown": "https://wpnews.pro/news/g5g-vs-g6-for-llm-serving-the-same-code-and-3-7x-the-throughput.md", "text": "https://wpnews.pro/news/g5g-vs-g6-for-llm-serving-the-same-code-and-3-7x-the-throughput.txt", "jsonld": "https://wpnews.pro/news/g5g-vs-g6-for-llm-serving-the-same-code-and-3-7x-the-throughput.jsonld"}}