{"slug": "serving-gemma-4-2b-on-a-single-tpu-v5e-chip-with-mcp-and-antigravity-cli", "title": "Serving Gemma 4 2B on a Single TPU v5e Chip with MCP and Antigravity CLI", "summary": "A developer detailed the deployment of Google's Gemma 4 2B model on a single Cloud TPU v5e chip using MCP and Antigravity CLI, achieving measured throughput and latency. The v5e chip, at roughly half the cost of v6e, proved well-suited for bandwidth-bound decode workloads, with the project consolidating configuration into a single tpu.env file.", "body_md": "This article is the v5e follow-on to the [v6e-1 debugging guide](https://xbill999.medium.com/debugging-deployments-with-gemma-4b-tpu-v6e-1-mcp-and-antigravity-cli-c9846231237a). Same MCP tooling, same Antigravity CLI driver, smaller and cheaper silicon — and a different set of failure modes. This time the model is `google/gemma-4-E2B-it`\n\non a single Cloud TPU v5e chip, with measured throughput, measured latency, and a cost breakdown that actually uses the numbers from the sweep.\n\nSame brief as before: a DevOps/SRE assistant whose brain is a self-hosted Gemma 4 model. The MCP server provisions the TPU, deploys the vLLM container, discovers the endpoint, and then uses that endpoint to analyze Cloud Logging output. 31 tools, one `server.py`\n\n, stdio transport.\n\nWhat changed is the target: **TPU v6e-1 → TPU v5e-1**, and **Gemma 4 4B → Gemma 4 2B**.\n\nThe interesting question this article answers: v5e is roughly half the price of v6e per chip-hour. Is it half the machine, or worse?\n\n| Spec (per chip) | TPU v5e (`v5litepod` ) |\nTPU v6e (Trillium) | Ratio |\n|---|---|---|---|\n| HBM capacity | 16 GB | 32 GB | 2.0× |\n| HBM bandwidth | 800 GiBps | 1,638 GBps | ~2× |\n| Peak BF16 | 197 TFLOPs | 918 TFLOPs | 4.66× |\n| Peak INT8 | 393 TOPs | 1,836 TOPs | 4.67× |\n| Single-chip machine type | `ct5lp-hightpu-1t` |\n`ct6e-standard-1t` |\n— |\n| On-demand list | ~$1.20 / chip-hr | ~$2.70 / chip-hr | 2.25× |\n| Flex-start (see below) | ~$0.60 / chip-hr | ~$1.35 / chip-hr | 2.25× |\n\nSpec figures are Google's own, from the [v5e](https://docs.cloud.google.com/tpu/docs/v5e) and [v6e](https://docs.cloud.google.com/tpu/docs/v6e) documentation. Note that Google quotes v5e bandwidth in **GiBps** and v6e in **GBps** — normalize the units and the real ratio is ~1.9×, not a clean 2×. I'm not going to pretend that's a meaningful difference at this altitude, but don't quote \"2× the bandwidth\" as if it were exact.\n\nThat table is the whole story in miniature. v6e is **2.25× the price** for **2× the memory and roughly 2× the bandwidth** but **4.7× the raw FLOPS**. For a 2B decode-bound workload — which is bandwidth-bound, not FLOPS-bound — v5e is priced almost exactly right. For prefill-heavy or long-context work, where you actually burn the matrix units, v6e's 4.7× starts to matter.\n\nThe sweep below bears this out.\n\nAntigravity CLI is the successor to Gemini CLI — the terminal-driven, agent-assisted coding tool. Install instructions:\n\n[Getting Started with Antigravity CLI](https://medium.com/google-cloud/getting-started-with-antigravity-cli-26c5da90951f)\n\nStart it and authenticate against a Google Cloud project:\n\n```\nagy\n```\n\nThe single biggest structural change from the v6e rig: deployment parameters live in exactly one place, `tpu.env`\n\n, and everything reads from it.\n\n```\n# tpu.env — the single source of truth for this rig.\nGOOGLE_CLOUD_PROJECT=aisprint-491218\nGOOGLE_CLOUD_REGION=us-west4\nGOOGLE_CLOUD_ZONE=us-west4-a\n\nMODEL_NAME=google/gemma-4-E2B-it\nACCELERATOR_TYPE=v5litepod-1\nTENSOR_PARALLEL_SIZE=1\n```\n\nIt is consumed by `server.py`\n\n(via `load_dotenv`\n\n), `mcp-run.sh`\n\n(which is what the MCP config actually launches), the `Makefile`\n\n(via `-include`\n\n), and `set_env.sh`\n\n. A real environment variable always wins over the file in all four — `load_dotenv`\n\ndoesn't overwrite, the wrapper only exports what's unset, and the Makefile uses `?=`\n\n— so `GOOGLE_CLOUD_ZONE=europe-west4-a make status`\n\nstill works as a one-off override.\n\nChange the zone once. Not in five places. This sounds like housekeeping until you spend an afternoon debugging a deploy that was reading a stale zone out of `mcp_config.json`\n\n.\n\nThe MCP config is correspondingly boring, which is the point:\n\n```\n{\n  \"mcpServers\": {\n    \"tpu-2B-v5e1-devops-agent\": {\n      \"command\": \"/home/xbill/gemma4-queens/tpu-2B-v5e1-devops-agent/mcp-run.sh\",\n      \"args\": [],\n      \"env\": {}\n    }\n  }\n}\n```\n\nCompare that to the v6e-1 version, which inlined seven environment variables into the JSON. Every one of those was a place the config could drift.\n\n(That absolute path is worth a second look if you're copying this: it points into a *different* checkout tree than the one this article was written from. Absolute paths in MCP configs are exactly the kind of thing that survives a directory rename and then fails silently at launch. `mcp-run.sh`\n\nexists to keep everything *else* out of this file — but the path to `mcp-run.sh`\n\nitself is still a hardcoded string.)\n\n`v5litepod`\n\nBefore anything else. gcloud does not know what a \"v5e-1\" is.\n\n`v5litepod-1`\n\n, `v5e-1`\n\n`v2-alpha-tpuv5-lite`\n\n`ct5lp-hightpu-1t`\n\n, `ct6e-standard-1t`\n\n\"v5e-1\" is fine in prose. It will never work in a gcloud argument. That last line matters more than it looks — `ct6e-standard-1t`\n\nis a *v6e* machine type, and if it shows up in your notes labelled as v5e, every memory number downstream of it is wrong by a factor of two. I found exactly that mislabel in this repo's own demo page while writing this article.\n\nThis is the one that cost the most time.\n\nThe MCP agent has a `get_zones_with_available_quota`\n\ntool that scans `TPUV5sLitepodPerProjectPerZoneForTPUAPI`\n\nacross every zone:\n\n```\n> get_zones_with_available_quota\n```\n\nIt came back with **44 zones with non-zero quota**. Every single one. That number is useless, and here's why: quota only means *creation is permitted*. It says nothing about whether capacity exists, and — critically — nothing about whether the **provisioning model** you asked for is supported there.\n\nFlex-start `v5litepod-1`\n\nis a much narrower thing than the quota table suggests:\n\n```\n> create a v5litepod-1 queued resource in europe-west4-a\n\n  ERROR: FLEX_START provisioning model is not supported for accelerator type\n  \"v5litepod-1\" in location \"europe-west4-a\"\n```\n\nSame rejection in `europe-west4-b`\n\n. Accepted in `us-west4-a`\n\n. Verified by attempting creation in each — there is no API that will tell you this in advance.\n\nSo the default zone this project shipped with (`europe-west4-a`\n\n, inherited from the v6e rig) **could never provision this rig**, in any amount of retrying. Worth noting: the reference documentation lists `europe-west4-b`\n\nas flex-start-capable for v5e, but its example uses `v5litepod-4`\n\n. The single-chip shape is narrower than the table implies.\n\nThe agent keeps mutable state in `tpu_zones_status.md`\n\n— `find_tpu`\n\nwrites failures into it and reads it back to skip known-bad zones. At one point it looked like this:\n\n```\n| Zone | Quota Available | TPU v5e-1 Started Successfully | Details |\n| europe-west4-a | Yes | No | creation failed |\n| europe-west4-b | Yes | No | creation failed |\n| europe-west1-b | Yes | No | creation failed |\n| ... 40 more rows ...\n```\n\nForty-odd zones \"out of capacity\" is not a capacity story. It's a flag story. Every one of those attempts was passing `--network=vpc-glitnir`\n\n— a VPC that does not exist in this project. `aisprint-491218`\n\nhas only the auto-mode `default`\n\nnetwork.\n\nThe fix was to let `TPU_NETWORK`\n\n/ `TPU_SUBNETWORK`\n\ndefault to empty so gcloud uses the project default, then reset the table and reseed it from a live quota scan. The lesson generalizes: **if a resource fails in every zone, it isn't the zones.**\n\nNote also that the file is *mutable state, not documentation*. Don't hand-edit it as if it were docs — `find_tpu`\n\nwill overwrite you.\n\nWith the zone sorted, the deploy is a single MCP call. The generated gcloud command:\n\n```\ngcloud alpha compute tpus queued-resources create vllm-gemma4-qr \\\n  --node-id=vllm-gemma4-qr-node \\\n  --project=aisprint-491218 \\\n  --zone=us-west4-a \\\n  --accelerator-type=v5litepod-1 \\\n  --runtime-version=v2-alpha-tpuv5-lite \\\n  --provisioning-model=flex-start \\\n  --max-run-duration=4h \\\n  --valid-until-duration=4h \\\n  --labels=purpose=flex-start \\\n  --metadata-from-file=startup-script=startup_script.sh\n```\n\nThose two duration flags are a **local policy choice, not a platform limit**. Flex-start itself allows up to **seven days** (`maxRunDurationSeconds`\n\n, defaulting to the full seven). This rig caps at 4h deliberately: it's a demo box, and an auto-expiring TPU is cheaper than a remembered one. `--valid-until-duration=4h`\n\nbounds the other end — how long the request sits in `WAITING_FOR_RESOURCES`\n\nbefore giving up rather than queuing indefinitely.\n\nThe startup script pulls `vllm/vllm-tpu:nightly`\n\nand serves:\n\n```\nvllm serve google/gemma-4-E2B-it \\\n  --max-model-len 16384 \\\n  --tensor-parallel-size 1 \\\n  --disable_chunked_mm_input \\\n  --max_num_batched_tokens 4096 \\\n  --limit-mm-per-prompt '{\"image\":4,\"audio\":1}' \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser gemma4 \\\n  --reasoning-parser gemma4\n```\n\n`--tensor-parallel-size`\n\nis **1**. v5e-1 is a single chip. If you see `4`\n\nanywhere in a v5e-1 config, it's copy-paste from a larger topology and the engine will fail to initialize.\n\nTwo structural details worth stealing:\n\n**The startup script fetches the HF token itself.** The rendered script is uploaded as *instance metadata*, so a baked-in token would be readable from the instance by anyone with `compute.instances.get`\n\n. Instead it reads `hf-token`\n\nfrom Secret Manager at boot via the metadata server, retrying for 30 minutes so an IAM grant applied *after* creation still lands. Shell tracing is off across the entire token section.\n\n**Serving flags live in exactly one function.** `_vllm_serve_flags()`\n\nbuilds the vLLM arg list from `MAX_MODEL_LEN`\n\n, `MAX_NUM_BATCHED_TOKENS`\n\n, `LIMIT_MM_PER_PROMPT`\n\n, and `TENSOR_PARALLEL_SIZE`\n\n. The startup script template takes the same values as placeholders. Both deploy paths and the generated one-liner therefore cannot disagree.\n\nThat template is consumed by `str.format()`\n\n, incidentally, which means **any literal { or } you add to that bash file breaks the deploy at format time** — a shell brace expansion, a\n\n`${VAR}`\n\n, a JSON literal. Escape as `{{`\n\n/ `}}`\n\n. The `--limit-mm-per-prompt`\n\nJSON above is exactly the trap.Here is where the 16 GB starts to bite, and it's not obvious from the parameter count.\n\nGemma 4 E2B is a \"2B\" model, but the resident weight footprint under vLLM is **~8.97 GiB**, not the ~4.5 GiB you'd naively compute from 2B parameters at bfloat16. The multimodal towers are not free.\n\nOn a v5e chip that arithmetic goes:\n\n```\nPhysical HBM                    16 GB   (~15.5 GiB)\nvLLM utilization cap (0.9)      ~13.9 GiB\nResident model weights          ~8.97 GiB\n─────────────────────────────────────────\nAvailable for KV cache          ~5.0 GiB\n```\n\nVersus the same model on a v6e chip, which has ~19.8 GiB left for KV — **roughly 4× the cache pool for a 2.25× price**.\n\nAt the observed ~18 KiB/token KV footprint (15 layers materialize a cache, bfloat16), ~5.0 GiB works out to roughly **290,000 tokens**, or about **17× concurrent requests** at the configured 16,384-token context.\n\n*Caveat, stated plainly:* the ~5.0 GiB and ~290K figures are arithmetic derived from the measured weight footprint and the published chip capacity, not read out of a v5e engine log. The 8.97 GiB weights and 18 KiB/token are measured. Treat the KV numbers as a good estimate, and confirm against your own `vllm`\n\ninit log before you size a fleet on them.\n\nThe 17× concurrency ceiling is not theoretical. It's exactly what falls over in the sweep.\n\nRun with `vllm bench serve`\n\ninside the container on the TPU VM, 128 output tokens per request, 20 grid points. **Zero failures across the whole grid** — worth noting, because the v6e-1 grid on the same model had 27 of 156 points fail or get skipped.\n\n| Conc | Context | req/s | Output tok/s | Total tok/s | Mean TTFT | p99 TTFT | TPOT |\n|---|---|---|---|---|---|---|---|\n| 1 | 128 | 0.99 | 126.9 | 254 | 22.5 ms | 75 ms | 7.76 ms |\n| 1 | 1,024 | 0.96 | 123.4 | 1,110 | 48.4 ms | 105 ms | 7.79 ms |\n| 1 | 4,096 | 0.87 | 110.8 | 3,655 | 149.5 ms | 208 ms | 7.92 ms |\n| 1 | 8,192 | 0.83 | 105.8 | 6,875 | 174.5 ms | 180 ms | 8.15 ms |\n| 1 | 15,000 | 0.70 | 90.1 | 10,643 | 351.2 ms | 358 ms | 8.42 ms |\n| 4 | 128 | 3.38 | 432.5 | 865 | 55.7 ms | 94 ms | 8.84 ms |\n| 4 | 1,024 | 3.40 | 435.1 | 3,916 | 32.4 ms | 42 ms | 8.97 ms |\n| 4 | 4,096 | 3.12 | 399.7 | 13,190 | 61.5 ms | 100 ms | 9.47 ms |\n| 4 | 8,192 | 2.78 | 355.8 | 23,126 | 92.9 ms | 169 ms | 10.41 ms |\n| 4 | 15,000 | 1.45 | 185.3 | 21,904 | 615.0 ms | 2,616 ms | 13.23 ms |\n| 16 | 128 | 8.41 | 1,076.9 | 2,154 | 169.5 ms | 289 ms | 13.59 ms |\n| 16 | 1,024 | 7.33 | 938.8 | 8,449 | 198.5 ms | 352 ms | 15.53 ms |\n| 16 | 4,096 | 4.79 | 612.8 | 20,221 | 365.8 ms | 974 ms | 23.10 ms |\n| 16 | 8,192 | 3.40 | 434.8 | 28,260 | 983.1 ms | 2,207 ms | 29.20 ms |\n| 16 | 15,000 | 1.32 | 169.3 | 20,003 | 3,319 ms | 8,220 ms | 68.73 ms |\n| 64 | 128 | 14.67 |\n1,877.7 |\n3,755 | 341.7 ms | 398 ms | 31.52 ms |\n| 64 | 1,024 | 11.52 | 1,474.9 | 13,274 | 686.5 ms | 1,347 ms | 37.68 ms |\n| 64 | 4,096 | 6.43 | 822.9 | 27,154 | 2,505 ms | 5,277 ms | 55.99 ms |\n| 64 | 8,192 | 2.66 | 340.3 | 22,119 | 3,077 ms | 6,870 ms | 69.24 ms |\n| 64 | 15,000 | 1.59 | 203.1 | 24,003 | 7,755 ms | 16,516 ms | 72.96 ms |\n\nPeak decode: **1,877.7 output tok/s** at 64 concurrent, short context.\n\nPeak aggregate: **28,260 total tok/s** at 16 concurrent, 8K context — that's prefill throughput, and it's the number to quote if your workload is RAG-shaped.\n\n**Single-stream decode is 90–127 tok/s and remarkably flat.** TPOT moves from 7.76 ms to 8.42 ms as context goes from 128 to 15,000 tokens — an 8% degradation across a 117× context increase. Decode on a 2B model is bandwidth-bound, and attention over even 15K tokens barely registers against the weight streaming cost. This is the number that matters for an interactive agent: **a single user gets a consistent ~120 tok/s regardless of how much context you stuff in.**\n\n**Batching scales cleanly up to about 4K context, then stops.** Speedup over single-stream, by context:\n\n| Context | 4 conc | 16 conc | 64 conc |\n|---|---|---|---|\n| 128 | 3.41× | 8.49× | 14.80× |\n| 1,024 | 3.53× | 7.61× | 11.96× |\n| 4,096 | 3.61× | 5.53× | 7.43× |\n| 8,192 | 3.36× | 4.11× | 3.22× |\n| 15,000 | 2.06× | 1.88× | 2.26× |\n\nLook at the 8,192 row. Going from 16 concurrent to 64 concurrent makes throughput **go down** — 434.8 tok/s to 340.3 tok/s. That is the ~5 GiB KV pool saturating. 64 requests × 8,192 tokens is ~524K tokens of KV demand against a pool that holds ~290K. vLLM starts preempting and recomputing, and you pay for the same prefill twice.\n\n**The 15,000-token column is a wall, not a slope.** At concurrency 1 it's fine — 90 tok/s, 351 ms TTFT. At concurrency 16, p99 TTFT is **8.2 seconds**. At 64, it's **16.5 seconds**. Nothing failed, which is the good news; vLLM queued rather than OOM'd. But a 16-second p99 is a broken user experience.\n\n**The practical envelope for this rig:**\n\n| Workload | Recommended concurrency | What you get |\n|---|---|---|\n| Chat / short prompts (≤1K) | 64 | ~14.7 req/s, sub-700 ms TTFT |\n| Tool-calling agent (≤4K) | 16 | ~4.8 req/s, ~366 ms TTFT, p99 under 1 s |\n| RAG / long docs (8K) | 16 | 28,260 tok/s aggregate, ~1 s TTFT |\n| Max context (15K) | 1–4 | Anything above 4 has an unusable p99 |\n\nNow the part the sweep pays for.\n\nFirst, an honesty note about the flex-start column, because I got this wrong on my first pass. **Google does not publish a per-chip-hour flex-start rate.** Flex-start is billed under [Dynamic Workload Scheduler pricing](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/dws), described only as *\"discounted (up to 53%) for vCPUs, GPUs, and TPUs.\"* So every flex number below is **on-demand list minus an assumed 50%**, which is the conservative end of \"up to 53%\". Treat them as upper bounds on the flex price, and therefore the cost-per-token figures as upper bounds too.\n\nOn-demand rates are list, us-west4:\n\n| Configuration | Flex-start (derived) | On-demand (list) |\n|---|---|---|\n| TPU v5e-1 (16 GB HBM) | ~$0.60 / hr | $1.20 / hr |\n| TPU v6e-1 (32 GB HBM) | ~$1.35 / hr | $2.70 / hr |\n| TPU v6e-4 (128 GB HBM) | ~$5.40 / hr | ~$10.80 / hr |\n| GCE 1× NVIDIA L4 (24 GB) | ~$0.30 / hr (spot) | ~$1.00 / hr |\n| GCE 8× NVIDIA L4 (192 GB) | ~$2.40 / hr (spot) | ~$8.00 / hr |\n| Cloud Run 1× L4 (serverless) | — | ~$1.20–1.50 / hr active |\nAWS EC2 `g6.xlarge` (1× L4) |\n~$0.35 / hr (spot) | ~$0.97 / hr |\nAzure `NVadsA10v5` (1× A10G) |\n~$0.35 / hr (spot) | ~$1.05 / hr |\n\nFor reference, published **spot** v5e is around $0.35/chip-hr and spot v6e $0.60–1.30/chip-hr — cheaper than flex-start, but preemptible with 30 seconds' warning, which is a different risk profile than flex-start's bounded run duration.\n\nRates vary by region and change; check current pricing before you commit. One warning from this codebase: `estimate_deployment_cost`\n\nin `server.py:880`\n\ncarries `\"v5e\": 0.12`\n\nin its rate table, which is off by an order of magnitude against the $1.20 list rate. Don't trust a hardcoded rate table — including this one.\n\n**Cost per million output tokens**, computed directly from the measured sweep:\n\n| Concurrency | Context | Output tok/s | Flex ($0.60/hr) | On-demand ($1.20/hr) |\n|---|---|---|---|---|\n| 1 | 128 | 126.9 | $1.314 / M | $2.627 / M |\n| 4 | 128 | 432.5 | $0.385 / M | $0.771 / M |\n| 16 | 128 | 1,076.9 | $0.155 / M | $0.310 / M |\n| 64 | 128 | 1,877.7 | $0.089 / M |\n$0.178 / M |\n| 16 | 1,024 | 938.8 | $0.178 / M | $0.355 / M |\n| 64 | 1,024 | 1,474.9 | $0.113 / M | $0.226 / M |\n| 16 | 4,096 | 612.8 | $0.272 / M | $0.544 / M |\n| 64 | 4,096 | 822.9 | $0.203 / M | $0.405 / M |\n\nThe headline: **$0.089 per million output tokens** at peak batching on flex-start.\n\nThe more useful headline is the spread. The *same chip*, *same model*, *same hour of billing*, run at concurrency 1 instead of 64, costs **14.8× more per token**. Idle TPU is the entire cost story. An agent rig that sits at concurrency 1 is paying $1.31/M for tokens it could be getting at $0.09/M.\n\nWhich leads to the real cost question for a devops agent: your workload probably *is* concurrency 1. An SRE assistant answering one operator's question is a single stream, and no amount of tuning turns one operator into a batch of 64.\n\nIn that regime **the cost control is run duration, not batching.** This is why the rig sets `--max-run-duration=4h`\n\nrather than accepting the seven-day flex-start default: an agent box that auto-expires costs a bounded amount, and a forgotten one costs $0.60/hr forever while serving nobody. Provision it for the session, let it expire. Do not leave a TPU idling at concurrency 1 and then compare $/M against a hosted API — you will lose that comparison badly, and you will deserve to.\n\nThe counterweight, and the reason not to set the cap too aggressively: re-provisioning is not instant. Flex-start requests sit in `WAITING_FOR_RESOURCES`\n\nuntil capacity appears, with **no documented SLA** on how long that takes. On this project it has run to a couple of hours. A 4-hour cap on a demo box is a reasonable trade; a 1-hour cap on something you actually need would not be.\n\nHere I have to be careful, and I'd rather flag it than quietly paper over it.\n\nThe v5e-1 numbers above come from `vllm bench serve`\n\nwith a **128-token output** per request. The v6e-1 Gemma 4 2B grid in this repo came from a different harness with a much **shorter output window**. Request-per-second figures between those two runs are **not directly comparable** — a shorter output inflates req/s roughly linearly, and the v6e grid's peak of ~140 req/s is largely measuring prefill-and-stop, not sustained generation.\n\nSo the honest comparison is on axes where the measurement method doesn't dominate:\n\n| Dimension | v5e-1 (measured) | v6e-1 | Verdict |\n|---|---|---|---|\n| Resident weights (E2B) | ~8.97 GiB | ~8.97 GiB | Model property, identical |\n| HBM left for KV |\n~5.0 GiB (derived) |\n~19.8 GiB (measured) | v6e ~4× the pool |\n| KV capacity | ~290K tok (derived) | 1,151,744 tok (measured) | v6e ~4× |\n| Sustained decode, 1 stream | 90–127 tok/s | ~214 tok/s | v6e ~1.7–1.8× |\n| TTFT, 1 stream, short ctx | 22.5 ms (on-host) | ~210 ms (over network) | Not comparable — different vantage points |\n| Grid failures | 0 / 20 |\n27 / 156 | v5e ran the whole grid |\n| Max usable context @ conc 16 | ~8K | 65K configured | v6e, decisively |\n| Price (flex) | ~$0.60/hr | ~$1.35/hr | v5e 2.25× cheaper |\n\n**Single-stream decode: v6e is roughly 1.7–1.8× faster** (~214 vs ~120 tok/s), for 2.25× the price. On a per-dollar basis for a single interactive user, **v5e is ahead**. That tracks the hardware — decode is bandwidth-bound, v6e has exactly 2× the bandwidth, and it converts about 85% of that into real tokens.\n\n**Memory is where v6e earns its premium, not compute.** The 4× KV pool is the difference between a model that batches 8K context at 16 concurrent and one that starts preempting. If your workload is short-context chat, you are paying for headroom you never touch. If it's RAG over long documents at any real concurrency, v5e's ~5 GiB pool is the binding constraint and no amount of tuning fixes it.\n\n**The FLOPS gap barely shows up.** v6e has 4.7× the BF16 throughput and delivers ~1.8× the decode tokens. That gap is the clearest possible signal that serving a 2B model is a memory-bandwidth problem, not a matrix-multiply problem. It also means the v6e-4 topology — 4 chips, 128 GB — is badly over-provisioned for a 2B model; the earlier v6e-4 sweep found the 2B model became bounded by *host CPU dispatch overhead*, with the TPU processing tokens faster than the host could schedule them.\n\n| If your workload is... | Use | Because |\n|---|---|---|\n| Single-agent SRE assistant, one operator | v5e-1 flex |\nBest tokens-per-dollar at concurrency 1; $0.60/hr |\n| Chat serving, short context, high concurrency | v5e-1 flex |\n14.7 req/s at $0.089/M output tokens |\n| Tool-calling agent, ≤4K context, ≤16 concurrent | v5e-1 flex |\nComfortably inside the KV envelope, p99 under 1 s |\n| RAG over 8K+ documents at concurrency >16 | v6e-1 |\nv5e's ~5 GiB KV pool preempts; throughput inverts |\n| Long context (15K+) at any real concurrency | v6e-1 |\nv5e p99 TTFT hits 8–16 s |\n| Anything 2B at 4-chip scale | Neither |\nHost-bound; spend the money on a bigger model instead |\n\nThe summary in one line: **for Gemma 4 2B, v5e-1 is the right default and v6e-1 is a memory upgrade, not a speed upgrade.**\n\nOnce the queued resource goes ACTIVE, endpoint discovery is dynamic — `discover_vllm_url()`\n\nlists queued resources in the zone, takes the first `ACTIVE`\n\none, resolves its node and external IP, and builds `http://{ip}:8000`\n\n. Never hardcode it; the IP changes every time the resource is recreated.\n\n```\n> verify_model_health\n\n  The health check passed successfully!\n\n  • Status: PASSED ✅\n  • Response: \"Hello! Yes, the model is working. I...\"\n  • Latency: 0.31 seconds\n> query_queued_gemma4_with_stats what is a TPU?\n\n  ### 📊 Performance Stats\n\n  • Time to First Token (TTFT): 0.022s\n  • Throughput: 126.9 tokens/s\n  • Mean TPOT: 7.76 ms\n```\n\nOne last trap: **raw /v1/completions returns an empty completion on -it models.**\n\n`make query`\n\nand `benchmarking_suite.py`\n\nboth use it, so an empty result there is expected, not a broken deploy. `server.py`\n\nuses `/v1/chat/completions`\n\nthroughout. Keep new code on the chat endpoint; raw completions are only useful for prefill-only benchmarks.\n\n```\n> make tools\n31 tools registered.\n```\n\n`GemmaTools.md`\n\nand the `get_help`\n\ntool both build their tool list from `mcp.list_tools()`\n\n, so they cannot drift from the `@mcp.tool()`\n\ndecorators. Add a tool, run `make tools`\n\n, the doc regenerates. Source of truth either way is `grep -n \"^@mcp.tool\" server.py`\n\n.\n\nOne split worth preserving if you fork this: `create_tpu_queued_resource`\n\nis **non-destructive** — it touches only the id it was given, which is what makes `find_tpu`\n\n's zone sweep safe to run. `manage_queued_resource`\n\nis **destructive** — it deletes every queued resource in the zone that isn't the named primary. Two verbs, two blast radii, no overlap.\n\nAnd: don't destroy a queued resource unless you mean it. Teardown is not part of routine debugging, and there is no documented SLA on how long a flex-start request waits for capacity — on this project it has run to a couple of hours.\n\nA single TPU v5e chip serves Gemma 4 2B at **90–127 tok/s single-stream** and **1,878 tok/s batched**, for **$0.60/hr flex-start** — about **$0.089 per million output tokens** at peak batching, or **$1.31/M** if you run it at concurrency 1 like most agent workloads actually do.\n\nAgainst v6e-1 at 2.25× the price, v5e gives up roughly **1.8× decode speed** and **4× KV cache pool**. The decode gap is a bandwidth story and v5e wins on a per-dollar basis. The KV gap is real and unfixable, and it is the actual reason to reach for v6e: not because v6e is faster, but because 16 GB runs out.\n\nThe three things that cost the most time were all configuration, not capacity: flex-start `v5litepod-1`\n\nis only accepted in `us-west4-a`\n\n; a nonexistent VPC in the create flags looked exactly like a global capacity shortage; and `ct6e-standard-1t`\n\nin a note labelled \"v5e\" quietly doubled every memory number downstream. Quota is not availability, availability is not the provisioning model, and a resource that fails in every zone is never the zones' fault.", "url": "https://wpnews.pro/news/serving-gemma-4-2b-on-a-single-tpu-v5e-chip-with-mcp-and-antigravity-cli", "canonical_source": "https://dev.to/gde/serving-gemma-4-2b-on-a-single-tpu-v5e-chip-with-mcp-and-antigravity-cli-33l8", "published_at": "2026-08-05 14:56:26+00:00", "updated_at": "2026-08-05 15:00:51.519185+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "developer-tools", "mlops"], "entities": ["Google", "Gemma 4 2B", "TPU v5e", "TPU v6e", "Antigravity CLI", "MCP", "vLLM", "Cloud TPU"], "alternates": {"html": "https://wpnews.pro/news/serving-gemma-4-2b-on-a-single-tpu-v5e-chip-with-mcp-and-antigravity-cli", "markdown": "https://wpnews.pro/news/serving-gemma-4-2b-on-a-single-tpu-v5e-chip-with-mcp-and-antigravity-cli.md", "text": "https://wpnews.pro/news/serving-gemma-4-2b-on-a-single-tpu-v5e-chip-with-mcp-and-antigravity-cli.txt", "jsonld": "https://wpnews.pro/news/serving-gemma-4-2b-on-a-single-tpu-v5e-chip-with-mcp-and-antigravity-cli.jsonld"}}