{"slug": "self-hosting-a-lite-agent-backend-on-one-tpu-gemma-4-e2b-vllm-on-a-v5e-1", "title": "Self-hosting a lite agent backend on one TPU: Gemma 4 E2B + vLLM on a v5e-1", "summary": "A developer self-hosted a lite agent backend on a single Google Cloud TPU v5e chip, achieving 1,496 output tokens/sec aggregate throughput with 8.02 ms per-token latency for Gemma 4 E2B under vLLM. The setup supports 8–16 concurrent agents at roughly $0.107 per million output tokens, and the developer documented several falsified predictions and zone-specific provisioning constraints.", "body_md": "A single Google Cloud TPU v5e chip — 16 GB of HBM, about $0.58/hour on spot — will serve `google/gemma-4-E2B-it`\n\nunder vLLM at **1,496 output tokens/sec** aggregate, with **8.02 ms** per-token latency at single stream and native tool-calling. That is enough to back a fleet of 8–16 concurrent \"lite\" agents for roughly **$0.107 per million output tokens**.\n\nThis is a build log with numbers. Everything here was measured on the hardware, and the sections that say \"I was wrong about this\" are the ones worth your time — four of my confident predictions were falsified by the benchmark, and each falsification was more useful than the guess.\n\n**Setup under test:** `v5litepod-1`\n\n(one v5e chip), `us-west4-a`\n\n, `vllm/vllm-tpu:nightly`\n\n, vLLM `0.26.1rc1.dev125+ga7a204cc6`\n\n, tpu-inference JAX backend, `google/gemma-4-E2B-it`\n\nat bf16.\n\n```\ngcloud auth login                     # for gcloud subprocess calls\ngcloud auth application-default login # ADC, for the Secret Manager client\n```\n\nPut your Hugging Face token in Secret Manager rather than in a script or an env file — the TPU VM's startup script is stored as instance metadata, and anything you bake in is readable from the instance:\n\n```\nprintf '%s' \"hf_xxxxxxxxxxxx\" | gcloud secrets create hf-token --data-file=- --project=YOUR_PROJECT\n```\n\n**Zone constraint that will waste your afternoon if you miss it:** flex-start `v5litepod-1`\n\nis only accepted in `us-west4-a`\n\n. `europe-west4-a`\n\nand `-b`\n\nreject it at the API with `FLEX_START provisioning model is not supported for accelerator type \"v5litepod-1\"`\n\n, *regardless of quota*. Non-zero quota in a zone tells you nothing — the provisioning model is the blocker.\n\nThree provisioning models, three different commands. Note `v5e`\n\nis spelled ** v5litepod** to gcloud — \"v5e-1\" is fine in prose and is never valid in a CLI argument.\n\n```\n# Spot — cheapest, preempted with ~30s notice, NO run limit (bills until you delete it)\ngcloud alpha compute tpus tpu-vm create gemma4-v5e \\\n  --zone=us-west4-a --type=v5litepod --topology=1x1 \\\n  --provisioning-model=spot --version=v2-alpha-tpuv5-lite\n\n# On-demand — full price, no preemption, also unbounded\ngcloud alpha compute tpus tpu-vm create gemma4-v5e \\\n  --zone=us-west4-a --type=v5litepod --topology=1x1 \\\n  --version=v2-alpha-tpuv5-lite\n```\n\nFlex-start goes through the Queued Resource API instead, and is the only model that accepts `--max-run-duration`\n\n, i.e. the only one that stops billing on its own:\n\n```\ngcloud alpha compute tpus queued-resources create gemma4-qr \\\n  --node-id=gemma4-qr-node --zone=us-west4-a \\\n  --accelerator-type=v5litepod-1 --runtime-version=v2-alpha-tpuv5-lite \\\n  --provisioning-model=flex-start --max-run-duration=4h\n```\n\nVerified 2026-08-09:this exact command was run — the QR reached`ACTIVE`\n\nwith`provisioningModel: FLEX_START`\n\nand`maxRunDuration: 14400s`\n\n, then deleted cleanly. Note there is no dry-run: a create either queues or provisions, and`PROVISIONING`\n\nstate cannot be deleted, so you will pay for at least a few minutes if capacity is immediately available.\n\nSpot and on-demand have no automatic stop.They bill until preempted or deleted. Set a calendar reminder, or use flex-start. See the cost section — flex-start is only 3.8% more than spot.\n\nSpot draws on a **separate quota** (`TPUV5sPreemptibleLitepodPerProjectPerZoneForTPUAPI`\n\n), not the standard TPU quota. A zone with plenty of on-demand quota can still refuse spot.\n\n`gcloud compute tpus tpu-vm ssh`\n\ncrashes with `ConnectionResetError`\n\nfrom some sandboxed environments (it fails inside its own internal API call, while plain gcloud API calls work fine). Direct SSH always works:\n\n```\nIP=$(gcloud compute tpus tpu-vm describe gemma4-v5e --zone=us-west4-a \\\n      --format='value(networkEndpoints[0].accessConfig.externalIp)')\n\n# Pipe the secret straight in — never through a shell variable or a log line\ngcloud secrets versions access latest --secret=hf-token \\\n  | ssh -i ~/.ssh/google_compute_engine xbill@$IP 'umask 077; cat > ~/.hf_token'\n\nssh -i ~/.ssh/google_compute_engine xbill@$IP 'sudo docker pull vllm/vllm-tpu:nightly'\n```\n\nThen start it. This is the configuration the rest of the article defends:\n\n```\nsudo docker run -d --name vllm-gemma4 --privileged --net=host \\\n  -v /dev/shm:/dev/shm --shm-size 10gb \\\n  -v ~/.cache/vllm:/root/.cache/vllm \\\n  -e HF_HOME=/dev/shm -e HF_TOKEN=\"$(cat ~/.hf_token)\" \\\n  vllm/vllm-tpu:nightly \\\n  vllm serve google/gemma-4-E2B-it \\\n    --dtype bfloat16 \\\n    --kv-cache-dtype auto \\\n    --max-model-len 32768 \\\n    --max-num-batched-tokens 4096 \\\n    --tensor-parallel-size 1 \\\n    --gpu-memory-utilization 0.92 \\\n    --enable-prefix-caching \\\n    --disable-chunked-mm-input \\\n    --limit-mm-per-prompt '{\"image\":4,\"audio\":1}' \\\n    --enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4\n```\n\n**That -v ~/.cache/vllm:/root/.cache/vllm line is the highest-value thing in this article.** The JAX compile cache lives there (197 MB measured) and is otherwise\n\n`docker rm`\n\n. Compilation is Cold start is **857 s** (14 min) and **80% of it is XLA compilation, not weight loading** — the 9.54 GiB checkpoint downloads in about 10 seconds. Be patient, and watch the log rather than the clock:\n\n```\nsudo docker logs -f vllm-gemma4 2>&1 | grep -E \"Memory statistics|Init kv-cache|startup complete\"\n```\n\nYou want to see this, which is the whole memory budget in one line:\n\n```\nMemory statistics | total_hbm_limit_gb=15.75GiB | total_hbm_limit_cap_gb=14.49GiB\n                  | total_hbm_used_gb=8.97GiB   | total_hbm_avail_gb=5.52GiB\n```\n\nThen smoke-test it. **Use /v1/chat/completions, not /v1/completions** — raw completions return an empty string on\n\n`-it`\n\nmodels, which looks exactly like a broken deploy and isn't:\n\n```\ncurl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{\n  \"model\":\"google/gemma-4-E2B-it\",\n  \"messages\":[{\"role\":\"user\",\"content\":\"Say hi in five words.\"}]}' | jq -r '.choices[0].message.content'\ngcloud compute tpus tpu-vm delete gemma4-v5e --zone=us-west4-a --quiet\n```\n\n| spec | v5e, one chip | source |\n|---|---|---|\n| HBM capacity | 16 GB nominal · 15.75 GiB visible to the runtime\n|\nvendor · measured |\nUsable for weights + KV at `0.92`\n|\n14.49 GiB |\nmeasured |\n| HBM bandwidth | 800 GiBps |\nvendor |\n| Peak bf16 | 197 TFLOPS | vendor |\n| Peak int8 | 393 TOPS (exactly 2x bf16) |\nvendor |\n| TensorCore | 1, with 4 MXUs (128x128) | vendor |\n| ICI | 400 GBps bidirectional, 4 ports | vendor |\n| Machine type | `ct5lp-hightpu-1t` |\n|\n| gcloud spelling |\n`v5litepod-1` , runtime `v2-alpha-tpuv5-lite`\n|\nmeasured |\n\nTwo unit traps worth knowing before you compare anything. Google quotes **v5e bandwidth in GiBps and v6e in GBps** — normalise before dividing, the real generational ratio is ~1.9x, not a clean 2x. And the \"16 GB\" capacity figure sits awkwardly next to the 15.75 GiB the runtime reports (15.75 GiB is 16.9 GB), so the vendor number is almost certainly 16 **GiB** loosely written. Size against the measured 15.75 GiB, never the marketing figure.\n\n**This single table decides every quantization question on this chip.**\n\n| format | native in the MXU? | what it buys on v5e |\n|---|---|---|\nbf16 |\n✅ | the baseline — everything here runs in it |\nint8 |\n✅ 2x bf16 throughput\n|\nthe only low-precision compute win |\nfp8 |\n❌ | storage and bandwidth only — values widen back to bf16 before the matmul |\nint4 / fp4 |\n❌ | footprint and bandwidth only, then unpack to bf16 |\n\nGoogle publishes bf16 and Int8 peaks for v5e and **no fp8 figure at all**, which is the tell. The practical consequence: *a benchmark showing no speedup from fp8 on this chip is the correct result, not a misconfiguration.* **v7/Ironwood is the first TPU with fp8 in the MXU** — do not carry any conclusion here forward to it.\n\nGemma 4 exists only as a JAX implementation in this stack, so anything in the torch path is unreachable no matter what the platform advertises. Measured state:\n\n| route | status on this build |\n|---|---|\nKV cache, bf16 (`auto` ) |\n✅ the only one worth running |\nKV cache `fp8_e4m3` / `fp8_e5m2`\n|\nreachable, 1.000x capacity — the block layout is word-aligned, so narrowing the element buys padding, not room. ~2% slower |\nKV cache `int8`\n|\n❌ rejected by the CLI enum — never reaches the engine |\nKV cache `int8_per_token_head` , `turboquant_*` , `nvfp4` , `fp8_inc` , `fp8_ds_mla`\n|\naccepted by the CLI, then kill the server at boot\n|\nWeights, compressed-tensors w4a16 (Google's QAT format) |\n❌ `NotImplementedError` on the JAX path |\nWeights, mxfp4\n|\n❌ MoE-only; E2B is dense, so there is nothing to attach to |\nWeights, qwix PTQ int8/int4 |\n❌ does not boot — the concrete path OOMs on quantization temporaries, the abstract path raises binding weights |\n| Weights, AWQ / GGUF / q4_0 | ❌ torch path or absent |\n\n**So bf16 is not a choice here, it is the only thing that runs** — and that is the single biggest constraint on the chip. Weights are **8.97 of the 14.49 GiB budget (62%)**, and none of it can be compressed today. Working int8 weights would roughly double the KV pool *and* buy real FLOPS, since int8 is the one format with a native MXU path. It is blocked upstream, not by configuration, so it is worth re-testing on every image bump.\n\nOne measured consequence of all this: decode moves ~3.15 GiB per step against a **3.94 ms** bandwidth floor, and measures **8.02 ms** — about **49% of peak bandwidth**. *(The 3.15 GiB is derived from the model's layer geometry; the 8.02 ms is measured.)* The chip is not the bottleneck at any point in this article.\n\nE2B is a strange checkpoint. Almost every intuition from a conventional decoder is wrong here, and the memory arithmetic later in this article only makes sense once these are on the table.\n\n| field | value |\n|---|---|\n`num_hidden_layers` |\n35 (28 sliding / 7 full attention, `i % 5 == 4` is full) |\n`num_kv_shared_layers` |\n20 — so only 15 layers own a cache\n|\n`num_attention_heads` / `num_key_value_heads`\n|\n8 / 1\n|\n`head_dim` / `global_head_dim`\n|\n256 / 512 |\n`hidden_size` / `intermediate_size`\n|\n1536 / 6144 |\n`vocab_size` |\n262,144 (tied embeddings) |\n`sliding_window` |\n512 |\n| resident at bf16 | 8.97 GiB |\n\n**1. \"E2B\" is not a 2B model.** It is ~2B *effective* against ~5B total, and lands at **8.97 GiB** resident. The `E`\n\nprefix is load-bearing — reading `E4B`\n\nas \"a 4B model\" understates its weights by roughly 2x, which is exactly the difference between fitting a 16 GB chip and not.\n\n**2. There are two attention geometries, not one.** Sliding layers run at `head_dim`\n\n256; the seven full-attention layers run at **512**, and that applies to K and V, not just Q. A single `head_dim`\n\ndoes not describe this model — anything that reads one value and applies it to all 35 layers under-counts the full layers by 2x. That is a 17% KV sizing error, and it is one people actually make.\n\n**3. Twenty of the thirty-five layers read someone else's cache.** `first_shared = 35 − 20 = 15`\n\n, so layers 0–14 own KV and layers 15–34 share. The mapping is \"last preceding layer of the same attention type\", and within 0–14 that means **all twenty shared layers resolve to just two source caches** — layer 13 for the sliding ones, layer 14 for the full ones. Twenty layers, two tensors.\n\n**4. KV costs 18 KiB/token, and the boot log will lie to you about why.**\n\n```\n12 sliding cached layers x 1 KV head x 2 (K,V) x 256 x 2 B = 12,288 B\n 3 full    cached layers x 1 KV head x 2 (K,V) x 512 x 2 B =  6,144 B\n                                                    total  = 18,432 B = 18 KiB/token\n```\n\nMultiply by the measured 321,344 resident tokens and you get **5.52 GiB** — exactly what the engine reports. But the log line describing the cache, `regular_attn_shape=(num_blocks, (64, 1, 2, 256))`\n\n, is a **first-wins sample taken from layer 0** — which is sliding, hence 256. It says nothing about layers 4, 9 and 14. On any hybrid model it under-reports. *Size KV from the config geometry and check it against total_hbm_avail_gb; never read it off that line.*\n\n**5. One KV head means more chips make things worse, not better.** `num_key_value_heads = 1`\n\nis full MQA, and a single head **cannot be sharded**. Runtimes pad `num_kv_heads`\n\nup to a multiple of the tensor-parallel size, so at TP=4 you pay **4x the KV memory to store the same head replicated**. On this model a larger topology multiplies the KV cost rather than dividing it. Check `num_key_value_heads`\n\nbefore assuming more chips solve a memory problem.\n\n**6. The heads do not tile the hidden size.** `8 x 256 = 2048`\n\nagainst `hidden_size = 1536`\n\n, so the Q projection is rectangular. Any code computing `head_dim = hidden_size / num_heads`\n\ngets **192** and is silently wrong.\n\nAnd one that explains the performance rather than the memory: **4.38 GiB of the 8.97 GiB resident is per-layer embedding tables** (262,144 x 256 x 35), which are *gathered per token, not streamed*. Only ~3.15 GiB actually moves per decode step — the dense transformer plus the 0.75 GiB tied embedding that `lm_head`\n\nreads in full. That is why an 8.97 GiB model decodes as fast as it does.\n\nTwo more traps worth knowing if you go poking at the checkpoint: the file also contains `audio_tower`\n\nand vision layers **with their own independent layer numbering**, so a regex matching `layers\\.(\\d+)\\.`\n\nsilently collides with them — always anchor on `model.language_model.`\n\n. And the QAT exports (`-qat-w4a16-ct`\n\n, `-qat-q4_0-unquantized`\n\n) **do not load on this stack at all**, partly because they legitimately ship no `k_norm`\n\nfor the KV-shared layers and the loader demands it anyway.\n\nEverything else follows from this:\n\n```\n 15.75 GiB   total HBM on the chip\n×  0.92      --gpu-memory-utilization\n─────────\n 14.49 GiB   the cap the engine will allocate inside\n−  8.97 GiB  model weights (bf16)\n─────────\n  5.52 GiB   KV cache  →  321,376 tokens at 18 KiB/token\n\n  1.26 GiB   what is left OUTSIDE the cap — compiled XLA programs live here,\n             and gpu_memory_utilization does not govern them\n```\n\nThat last line is the trap. I tested `--gpu-memory-utilization 0.95`\n\nexpecting a free +8% of KV. The KV pool sized *exactly* as arithmetic predicted — cap 14.96 GiB, KV 5.99 GiB, **348,864 tokens, +8.6%** — and then XLA died **691 s in**, loading `jit_structured_decode_fn`\n\n:\n\n```\nRuntimeProgramAllocationFailure: Attempting to reserve 384.11M at the bottom of memory.\nThat was not possible. There are 347.33M free, 0B reserved, and 347.33M reservable.\n```\n\nThe knob governs weights + KV only; compiled program images come out of the remainder it leaves behind. I have now measured this failure twice, months apart, at different page sizes (32 and 64) — 384.11 M wanted against 346.77 M and 347.33 M free. It is deterministic, not a race.\n\n**0.92 is a ceiling, not a conservative default.** And note the shape of that failure: it costs a full compile to discover — 691 s before it tells you.\n\n`--gpu-memory-utilization 0.92`\n\nFraction of total HBM the engine may allocate for **weights + KV cache**. Not activations, not compiled programs. 0.95 does not boot on this model/chip/build. 0.93 and 0.94 are untested; the estimated margins are ~291 MB and ~127 MB against a failure that was 37 MB short, so the risk/reward is poor.\n\nThere is also `--kv-cache-memory`\n\n(config field `kv_cache_memory_bytes`\n\n), which pins the pool in **bytes** instead of as a fraction, and skips memory profiling on later boots. Measured-good value on this setup: `5923602432`\n\n.\n\n`--max-model-len 32768`\n\nThe maximum context. The surprising part is that **it costs no KV capacity**:\n\n`max-model-len` |\n`block_size` |\nblocks/request | KV blocks | KV tokens |\n|---|---|---|---|---|\n| 16,384 | 32 | 512 | 10,043 | 321,376 |\n| 32,768 | 64 | 512 | 5,021 | 321,344 |\n\nThe Pallas backend scales `block_size`\n\n*with* `max_model_len`\n\nto hold blocks-per-request constant at\n\nAnd blocks-per-request turns out to predict decode speed:\n\n| blocks/request | c=1 TPOT |\n|---|---|\n512 (`max-model-len` 16384) |\n8.05 ms |\n512 (`max-model-len` 32768) |\n8.02 ms |\n| 2048 (a misconfigured arm) | 8.32 ms |\n\n**Corollary: do not set --block-size.** The backend picks it, and pinning 32 would fight the scaling that keeps long context free.\n\n`--max-num-batched-tokens 4096`\n\nThe token budget per scheduler step — the chunk size for chunked prefill. Smaller favours inter-token latency (a big prefill chunk stalls every in-flight decode); larger favours time-to-first- token. This is the single most misunderstood flag on TPU, for two reasons.\n\n**Reason 1: the buckets are powers of two.** TPU needs a compiled graph per tensor shape, so vLLM precompiles a ladder — `16, 32, 64, … 4096`\n\n— and rounds your value **up** to the next bucket. Setting 2496 compiles *exactly the same ladder* as 4096. Each chunk then costs a 4096-shaped kernel while carrying 2496 tokens of work. I tried it: **−24.7% throughput** at 8k/64 with the ITL tail completely unchanged (173.9 → 174.2 ms).\n\n**Reason 2: multimodal sets a hard floor.** With `--disable-chunked-mm-input`\n\n, one multimodal item must fit in a single batch. Go under it and the server refuses to start:\n\n```\nValueError: Chunked MM input disabled but max_tokens_per_mm_item (2496)\nis larger than max_num_batched_tokens (2048). Please increase max_num_batched_tokens.\n```\n\nSo for this model with `{\"image\":4,\"audio\":1}`\n\n: **floor 2496, next bucket 4096**. Every value in (2048, 4096] compiles identically, so the largest one wins. **4096 is optimal in that interval** — not by taste, by construction.\n\n`--max-num-seqs`\n\n(leave it alone)\nMaximum concurrent sequences. The default is **256**, and where that comes from is worth knowing, because it is not where you'd look:\n\n`SchedulerConfig.DEFAULT_MAX_NUM_SEQS = 128`\n\nis `EngineArgs.get_batch_defaults()`\n\noverrides it from a dict keyed by usage context and gated on device memory: `>= 70 GiB`\n\n→ 1024, otherwise → `max_num_batched_tokens`\n\n)… and it never fires, because `get_device_name()`\n\nreturns `'TPU V5E'`\n\nwhile the code tests `chip_name == \"V5E\"`\n\n. It also calls `get_device_total_memory()`\n\n, which raises `NotImplementedError`\n\non tpu-inference and is swallowed by a bare `except`\n\n, so the memory gate reads 0.So on a v5e you silently get **generic non-GPU defaults**. I tried capping it to 64 on the theory that 256 over-admits (256 × 16384 = 4.19M KV tokens against 321,376 resident). **It made things worse**, and in any case my benchmark's peak offered load was 64, so the cap never bound. Leave it at 256 unless you are actually seeing preemption in the logs.\n\n`--kv-cache-dtype auto`\n\nLeave this alone, and be suspicious of anyone who tells you otherwise. `--kv-cache-dtype fp8_e4m3`\n\ngives a **1.000x** capacity ratio on this stack. The KV block layout is word-aligned:\n\n```\nbf16: regular_attn_shape=(num_blocks, (32, 1, 2, 256))  →  32,768 bytes/block/layer\nfp8:  regular_attn_shape=(num_blocks, (32, 1, 4, 256))  →  32,768 bytes/block/layer\n```\n\nThe third dimension doubles exactly as the element width halves. Narrowing the element buys padding, not capacity — and costs ~2% throughput. The flag is accepted at the CLI, echoed in `non-default args`\n\n, praised by a log line, reported in `/metrics`\n\n, and allocates a genuinely `float8_e4m3fn`\n\ntensor. **Five independent signals that it worked, and it did nothing.**\n\nI expected `--kv-cache-dtype int8`\n\nto be worse still — reachable, and silently rounding the cache to whole integers because the model hardcodes its K/V scales to `1.0`\n\n. **I tested it and that is wrong on this build:** `int8`\n\nis not in vLLM's CLI enum and is rejected before it reaches any of that (`invalid choice: 'int8'`\n\n). The 16 accepted values are `auto, bfloat16, float16, fp8, fp8_ds_mla, fp8_e4m3, fp8_e5m2, fp8_inc, fp8_per_token_head, int4_per_token_head, int8_per_token_head, nvfp4`\n\nand four `turboquant_*`\n\n. The ones that *are* accepted but unsupported (`int8_per_token_head`\n\n, `turboquant_*`\n\n, `nvfp4`\n\n, `fp8_inc`\n\n, `fp8_ds_mla`\n\n) fail loudly at boot rather than quietly — so on this build the honest summary is: **fp8 is the trap, because it is the one that appears to work.**\n\n`auto`\n\nis not vague here — it means \"inherit the model dtype\", which `--dtype bfloat16`\n\npins.\n\n`--tensor-parallel-size 1`\n\nOne chip, so this is forced. But it's worth knowing *why* more chips wouldn't help this model: E2B has `num_key_value_heads=1`\n\n— full MQA. A single KV head **cannot be sharded**; runtimes pad it up to a multiple of the TP size, so at TP=4 you pay 4× the KV memory to store the same head replicated. For this model, more chips multiply KV cost rather than dividing it.\n\n`--disable-chunked-mm-input`\n\nand `--limit-mm-per-prompt`\n\nThese are the multimodal contract, and **together they set the 2496-token floor** discussed above. If you don't need images and audio, dropping them lets `max-num-batched-tokens`\n\nreach the 2048 bucket, which is the only route to a lower latency floor on this stack. That is a real trade I'd evaluate before copying this config into a text-only deployment.\n\n`--enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4`\n\nWhat makes this an *agent* backend rather than a text box: OpenAI-compatible tool calling, parsed natively. Worth knowing that these pull in structured-decoding machinery — the program that ran out of memory in the 0.95 experiment was `jit_structured_decode_fn`\n\n— so they are not free against that ungoverned 1.26 GiB remainder.\n\nSet with `-e`\n\non `docker run`\n\n; these come from tpu-inference, not vLLM:\n\n| var | default | what it does |\n|---|---|---|\n`ATTN_BUCKETIZED_NUM_REQS` |\n`False` |\nprecompile attention at power-of-two request buckets instead of a single shape at `max_num_seqs`\n|\n`SLICE_ROPE_CACHE` |\n`False` |\nslice the rotary cache to `max_model_len` at load — free HBM |\n`NUM_PRECOMPILE_WORKERS` |\n`1` |\nparallel XLA precompilation; compile is 685 s of an 857 s boot |\n`VLLM_TPU_BUCKET_PADDING_GAP` |\n`0` |\nlinear bucket increments (use 128) instead of powers of two — the fix for the 2496-pads-to-4096 problem |\n`VLLM_XLA_CHECK_RECOMPILATION` |\n`False` |\nerror on a runtime recompile; turn on for one validation boot |\n\nProvenance.All 12 cells were measured on the configuration above,3 repetitions each(36 runs). Throughput is stable — coefficient of variation ≤3.4%, most ≤1%. The one exception is marked: 128-ctx/16-clientTTFThas cv 56% across reps, so no point value should be trusted there. An earlier config was re-run as a control and reproduced a previous sweep to the digit (TPOT 8.05/8.08/8.43 ms, KV 5.52 GiB, 10,043 blocks), so the rig itself is stable.\n\nAggregate output tokens/sec:\n\n| context ↓ / clients → | 1 | 4 | 16 | 64 |\n|---|---|---|---|---|\n| 128 | 123.7 | 433.9 | 1,152.5 | 1,496.5 |\n| 1,024 | 120.4 | 415.4 | 991.0 | 1,258.7 |\n| 8,192 | 94.2 | 254.4 | 399.6 | 324.3 |\n\nMedian time-to-first-token (ms):\n\n| context ↓ / clients → | 1 | 4 | 16 | 64 |\n|---|---|---|---|---|\n| 128 | 15.9 | 31.8 | 98.8* | 247.3 |\n| 1,024 | 40.3 | 51.5 | 170.1 | 421.7 |\n| 8,192 | 289.4 | 304.2 | 594.4 | 11,734 |\n\nPer-stream tokens/sec (what one user feels):\n\n| context ↓ / clients → | 1 | 4 | 16 | 64 |\n|---|---|---|---|---|\n| 128 | 125 | 112 | 76 | 25 |\n| 1,024 | 124 | 109 | 69 | 21 |\n| 8,192 | 119 | 71 | 27 | 10 |\n\n**Short context (≤1K) scales cleanly to 64 clients.** 12.1× the single-stream throughput. This is the regime most agent traffic lives in.\n\n**Long context (8K) peaks at 16 clients and then goes backwards.** 400 → 324 tok/s from 16 to 64, with median TTFT blowing out to 11.7 seconds. The reason is arithmetic: 64 streams × 8,192 tokens = 524,288 KV tokens wanted against **321,376 resident**. Past the KV wall, more clients buy queueing, not throughput.\n\n**Single stream is bandwidth-limited at about half of peak.** Decode moves ~3.15 GiB per step (the model's 4.38 GiB of per-layer embedding tables are *gathered*, not streamed), which at the v5e's 800 GiBps is a 3.94 ms floor against 8.02 ms measured — **49% of peak**. *(That 3.15 GiB is derived from the model's layer geometry, not an instrument reading; the 8.02 ms is measured.)*. There is roughly 2× of headroom sitting in fixed per-step cost, not in memory bandwidth.\n\n| workload | context | clients | expected |\n|---|---|---|---|\n| Interactive chat / agent turns | ≤1K | 16 |\n991–1,152 tok/s, 99–170 ms TTFT, ~14 ms TPOT |\n| Max throughput, batch/offline | ≤1K | 64 |\n1,259–1,496 tok/s, 247–422 ms TTFT |\n| RAG / long documents | 8K | 16 |\n399.6 tok/s, 594 ms TTFT |\n| Long-context interactive | 8K | ≤4 |\n254 tok/s, ~304 ms TTFT |\n\n**Rule of thumb: keep clients × context < 250,000 tokens** — about 78% of the 321,344-token KV pool. Cross it and you're paying in tail latency for throughput you don't get.\n\nAgent workloads have a shape that suits this chip well:\n\n`--tool-call-parser gemma4`\n\ngives OpenAI-compatible function calling with no glue.What it is *not* good for: long-context batch summarization at high concurrency (the 8K/64 cell is the worst on the chart), or anything needing >321K tokens of live KV.\n\nLive rates from the Cloud Billing Catalog for `us-west4`\n\n, per chip-hour:\n\n| model | $/chip-hr | vs spot | stops itself? | preemptible? |\n|---|---|---|---|---|\n| 3-year commitment | 0.5400 | −6.6% | n/a | no |\nSpot |\n0.5779 |\n— | no |\nyes, ~30 s notice |\nFlex-start (DWS) |\n0.6000 |\n+3.8% |\nyes (`--max-run-duration` ) |\nno, once running |\n| 1-year commit / Reserved | 0.8400 | +45% | n/a | no |\nOn-demand |\n1.2000 |\n+108% |\nno | no |\n\nSpot is only **3.8% cheaper** than flex-start, and that discount is fragile. A preemption costs a full cold start — 857 s, of which 685 s is recompilation — which is **$0.138** of wasted spend at the spot rate. Flex-start's premium is **$0.0221/hour**. So:\n\nSpot beats flex-start only while preemptions are less frequent than every 6.2 h without the cache mount, or every 3.6 h with it.Measured rebuild cost: $0.1376 cold, $0.0798 warm.\n\nAnd flex-start *stops billing on its own* via `--max-run-duration`\n\n, while spot and on-demand run until you remember to delete them. One forgotten weekend on a spot node ($0.58 × 60 h ≈ $35) wipes out months of the 3.8% saving.\n\n**Mount the compile cache and the calculus changes again** — a warm cache turns a preemption from ~18 minutes into ~6, which is exactly why that one `-v`\n\nflag matters more than any tuning flag here.\n\n| workload | tok/s | spot | flex-start | on-demand |\n|---|---|---|---|---|\n| 128 ctx, 64 clients | 1,496.5 | $0.107 |\n$0.111 | $0.223 |\n| 1K ctx, 64 clients | 1,258.7 | $0.128 | $0.132 | $0.265 |\n| 1K ctx, 16 clients | 991.0 | $0.162 | $0.168 | $0.336 |\n| 8K ctx, 16 clients | 399.6 | $0.402 | $0.417 | $0.834 |\n| 1 client (any ctx) | 123.7 | $1.298 | $1.347 | $2.695 |\n\n**Running one chip 24/7:** $422/month on spot, $438 on flex-start, $876 on-demand.\n\nThe headline: **batching is worth more than any pricing decision.** Going from 1 client to 64 at short context is a **12.1× cost reduction per token** — far larger than the 2.08× between on-demand and spot. Tune your concurrency before you shop for discounts.\n\nThe falsified predictions were the most valuable output of this exercise.\n\n**1. \"fp8 KV cache will double capacity.\"** It gave 1.000x, because the layout is word-aligned. Five independent signals said the flag had worked. Only the block-shape line revealed the truth. *Lesson: verify quantization from the boot allocation log, never from the flag being accepted.*\n\n**2. \"0.95 memory utilization is a free 8% of KV.\"** The KV math was right to 0.03% and the engine died anyway, 13 minutes later, because compiled programs live outside the knob's control. *Lesson: a memory knob that doesn't govern all memory will lie to you.*\n\n**3. \"Lowering max-num-batched-tokens to the exact multimodal floor will cut latency.\"** It cost 24.7% throughput and moved the latency tail by 0.2%, because 2496 and 4096 round to the same compiled bucket.\n\n**4. \"Capping max-num-seqs at 64 will speed up decode.\"** It didn't, and the arm carrying it was worse on every cell.\n\nThere's a fifth I haven't been able to fix. The boot log reports `Hybrid KV cache layout: num_kv_cache_groups=1`\n\n— every one of the 15 cached layers gets full-length KV allocation, even though **12 of them are sliding-attention layers windowed at 512 tokens**. Windowing them would be worth **2.8× the KV capacity** at 16K context, at zero quality cost. It's unreachable: tpu-inference disables sliding windows for any model with more than one head dim, and Gemma 4 has two (256 on sliding layers, 512 on full). The source carries a `TODO: enable sliding windows once mixed dims support`\n\n. Worth re-checking on every image bump — it's the largest single win still on the table.\n\nThe final configuration was booted from scratch and exercised, not assembled from winning fragments:\n\n| check | result |\n|---|---|\n| cold boot |\n857 s (compile 685 s = 80%) |\n| warm boot, compile cache mounted |\n497 s (−42%) |\n| memory | 14.49 GiB cap / 8.97 weights / 5.52 KV — matches every other arm |\n| KV capacity |\n`block_size` 64 x 5,021 blocks = 321,344 tokens\n|\n| chat completion | ✅ |\ntool calling |\n✅ `{\"name\":\"get_weather\",\"arguments\":\"{\\\"city\\\": \\\"Paris\\\"}\"}`\n|\nmultimodal image |\n✅ correctly described a synthetic gradient PNG |\nlong context |\n✅ 26,016 prompt tokens accepted |\n| throughput | 12 cells x 3 reps, cv ≤3.4% |\n\nThe three flags that restate a default (`--dtype`\n\n, `--kv-cache-dtype`\n\n, `--gpu-memory-utilization`\n\n) are **confirmed no-ops by vLLM itself** — they do not appear in the engine's `non-default args`\n\nwhen passed at these values. They are in the command for auditability, since the real defaults are computed several layers from where they look like they are declared.\n\n**fp8 KV was re-verified on this exact build**, because the original result predated a container rebuild. Same 5,021 blocks, same 5.52 GiB, same 321,344 tokens as bf16 — while the shape goes `(64,1,2,256)`\n\n→ `(64,1,4,256)`\n\nand the dtype really is `float8_e4m3fn`\n\n. The first time I measured this the page size was 32; it reproduces at 64, so the word-alignment mechanism is not a block-size artifact.\n\n`/v1/completions`\n\nreturns an empty string on `-it`\n\nmodels.`/v1/chat/completions`\n\n. An empty benchmark result is expected there, not a broken deploy.`v5e`\n\nis `v5litepod`\n\nto gcloud.`v5litepod-1`\n\n, runtime `v2-alpha-tpuv5-lite`\n\n, `--type=v5litepod --topology=1x1`\n\n.`sha256:2a4a1f82…`\n\nfrom `docker images`\n\nis a config-blob ID, not a manifest digest; `docker pull`\n\nby it fails with `unexpected media type`\n\n. The version string is the better handle — `0.26.1rc1.dev125+g**a7a204cc6**`\n\nembeds vLLM's git SHA, so you can read the exact source your build shipped.`gemma-4-E2B-it`\n\n✅ unit / ❌ correctness / ❓ performance, while the 26B and 31B pass all three. My own quality probes were clean (8/9 byte-identical outputs, 3/3 needle retrievals at 2K/8K/14K), but you should know the flag exists.`docs.vllm.ai/en/v0.11.1/configuration/tpu/`\n\nare excellent and partly stale.`VLLM_TPU_MOST_MODEL_LEN`\n\nrecommendation no longer exists in either vLLM or tpu-inference. Check the version before you copy.Test the *whole* configuration, not a change to it. My final recommendation is a config that was never run end-to-end — it's the winning arm plus three pins that *should* be no-ops. Every time I assumed something was a no-op in this project, I was eventually proved wrong.\n\nUntested and plausibly better: `max-model-len 65536`\n\n(block_size would go to 128, blocks-per-request stays 512, so it may be free too), `max-num-batched-tokens 8192`\n\n, `VLLM_TPU_BUCKET_PADDING_GAP=128`\n\n, and **n-gram speculative decoding** — which needs no draft checkpoint and is marked fully passing on TPU, against a workload sitting at 49% of memory bandwidth.", "url": "https://wpnews.pro/news/self-hosting-a-lite-agent-backend-on-one-tpu-gemma-4-e2b-vllm-on-a-v5e-1", "canonical_source": "https://dev.to/gde/self-hosting-a-lite-agent-backend-on-one-tpu-gemma-4-e2b-vllm-on-a-v5e-1-fk1", "published_at": "2026-08-09 22:14:30+00:00", "updated_at": "2026-08-09 22:46:51.036126+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-agents", "developer-tools"], "entities": ["Google Cloud", "TPU v5e", "Gemma 4 E2B", "vLLM", "Hugging Face", "Secret Manager", "gcloud"], "alternates": {"html": "https://wpnews.pro/news/self-hosting-a-lite-agent-backend-on-one-tpu-gemma-4-e2b-vllm-on-a-v5e-1", "markdown": "https://wpnews.pro/news/self-hosting-a-lite-agent-backend-on-one-tpu-gemma-4-e2b-vllm-on-a-v5e-1.md", "text": "https://wpnews.pro/news/self-hosting-a-lite-agent-backend-on-one-tpu-gemma-4-e2b-vllm-on-a-v5e-1.txt", "jsonld": "https://wpnews.pro/news/self-hosting-a-lite-agent-backend-on-one-tpu-gemma-4-e2b-vllm-on-a-v5e-1.jsonld"}}