{"slug": "serving-gemma-4-on-an-amd-mi300x-what-1-99-an-hour-buys", "title": "Serving Gemma 4 on an AMD MI300X: What $1.99 an Hour Buys", "summary": "A developer published a step-by-step guide and open-source MCP toolkit for deploying Google's Gemma 4 E2B model to a single AMD Instinct MI300X GPU rented through AMD Developer Cloud at $1.99 per hour. The vLLM-based setup reached 305 tokens per second on a single stream and 10,284 tokens per second across 64 streams, or roughly 2,713 tokens per second per dollar-hour. The 21-tool MCP server runs on a workstation with no local AMD hardware, reaching the remote GPU over SSH and the DigitalOcean v2 API.", "body_md": "*This article provides a step by step deployment guide for **Gemma 4 E2B** to an **AMD Instinct MI300X** hosted GPU enabled system. A suite of Python MCP tools is built to simplify management of the vLLM hosted deployment. The card is reached through AMD Developer Cloud, which is DigitalOcean underneath, and the workstation driving it has no AMD GPU in it at all.*\n\n[github.com/xbill9/gemma4-dev/tree/main/gpu-vllm-mi300x-2b](https://github.com/xbill9/gemma4-dev/tree/main/gpu-vllm-mi300x-2b)\n\n| Model | `google/gemma-4-E2B-it` , reference bf16 release | \n| Hardware | 1x AMD Instinct MI300X, `gfx942` , CDNA 3, 191.7 GiB HBM3 | \n| Host | `gpu-mi300x1-192gb-devcloud` — 20 vCPU, 240 GB RAM, 720 GB disk, region`atl1` | \n| Control plane | AMD Developer Cloud / DigitalOcean v2 API — no `gcloud` , no EC2 | \n| Software | `vllm/vllm-openai-rocm:nightly-rocm100` , vLLM`0.3.1.dev3+g0bfc7a15d` | \n| Rate | **$1.99 per hour** , on-demand | \n| Result | **305 tok/s** single stream,**10,284 tok/s** at 64 streams,**2,713 tok/s per dollar-hour** | \n\nThe card is a DigitalOcean GPU droplet reached through AMD Developer Cloud (`devcloud.amd.com`) — same v2 API, same droplet ids, token from the My AMD Team account. Creating and destroying it are console actions, deliberately: both are dollar-per-hour decisions and neither belongs in a tool an agent can call.\n\nEverything after creation is scripted. The droplet carries a tag, every tool is scoped to that tag, and nothing in the toolkit can touch an instance that does not have it.\n\n`DIGITALOCEAN_ACCESS_TOKEN` in a mode 0600 `.env` beside the server`/dev/kfd` and `/dev/dri` present\n`rocm-smi`, `amd-smi`, `rocminfo` and `hipcc` do not exist locally and never will. A ROCm command in a local shell is a bug, not a check. Every reading in this article came back over SSH or through the DigitalOcean API.\n\nThat constraint shapes the whole toolkit. The MCP server runs on the workstation, holds no device, and reaches the hardware the same way you would by hand.\n\nOne file, `server.py`, exposes 21 tools over stdio: droplet lifecycle, image checks, deploy, logs, benchmark cells, and the SRE probes below. Every subprocess goes through one `run_command(cmd: list[str])` helper using `asyncio.create_subprocess_exec` — never a shell.\n\n``` bash\n$ grep -c \"^@mcp.tool\" server.py\n21\nbash\n$ droplet_status\n📡 debian-gpu-mi300x1-192gb-devcloud-atl1 (601418522)\n\n- status: `active`\n- size: `gpu-mi300x1-192gb-devcloud` — 20 vCPU, 240 GB RAM, 720 GB disk\n- region: atl1\n- cost: $1.99/hr — billed while powered off, too\n```\n\nThe rate comes from the DigitalOcean v2 API's own `droplet.size.price_hourly` field, read at the time of the run, not from a pricing page.\n\n``` bash\n$ gpu_status\n✅ GPU reporting on `debian-gpu-mi300x1-192gb-devcloud-atl1`.\n\n| Card | Product | GPU use % | VRAM used % |\n| card0 | Aqua Vanjaram [Instinct MI300X VF] | 0 | 0 |\n```\n\nIt presents as an SR-IOV virtual function, which looks like a partition and is not one: all 304 compute units and the whole 191.7 GiB are there. Neither `rocm-smi` nor `amd-smi` sets a useful exit code, so the tool parses the output and ignores the status code entirely.\n\n`vllm/vllm-openai-rocm:nightly-rocm100`. Read the version out of the running container rather than trusting the tag, because a nightly tag moves under you:\n\n``` python\n$ docker exec vllm python3 -c \"import vllm;print(vllm.__version__)\"\n0.3.1.dev3+g0bfc7a15d\n```\n\nNot every ROCm vLLM image can load this checkpoint — Gemma 4 runs 256-wide heads on its sliding-attention layers and 512 on its full-attention ones, and an image whose vLLM predates that split cannot parse the config. The measured comparison of three images is in the [companion article](https://github.com/xbill9/amd-gputools). Here the practical rule is enough: pin what you measured, and check a new image before you adopt it.\n\n``` bash\n$ deploy_vllm\n✅ Started `vllm` serving `google/gemma-4-E2B-it`.\n\n- image: `vllm/vllm-openai-rocm:nightly-rocm100`\n- context: 32768, gpu-memory-utilization 0.90\n- multimodal: `{\"image\": 4, \"audio\": 0}`\n```\n\nThe serve arguments, in full:\n\n```\nvllm serve google/gemma-4-E2B-it --host 0.0.0.0 --port 8000 \\\n  --max-model-len 32768 --gpu-memory-utilization 0.90 \\\n  --enable-auto-tool-choice --reasoning-parser gemma4 --tool-call-parser gemma4 \\\n  --chat-template /app/vllm/examples/tool_chat_template_gemma4.jinja \\\n  --limit-mm-per-prompt '{\"image\": 4, \"audio\": 0}' --async-scheduling\n```\n\n`audio` is `0` on purpose. E2B has a conformer audio encoder and no ROCm vLLM image ships the `vllm[audio]` extras, so a non-zero limit would allocate encoder memory for a path that cannot be used.\n\n`docker run` returns in about a second. The endpoint does not answer for another two and a half minutes, while weights load, `torch.compile` runs and graphs are captured.\n\n```\n# polling /v1/models every 10s from the moment docker run returned\nREADY after 160s\n{\"object\":\"list\",\"data\":[{\"id\":\"google/gemma-4-E2B-it\",\"max_model_len\":32768,...}]}\n```\n\nThis is why `serving_status` reports the container and the endpoint as two separate facts. A running container is not a serving model, and conflating them turns a normal boot into a phantom hang.\n\n``` bash\n$ serving_status\n✅ container: Up 3 minutes | endpoint: answering on `127.0.0.1:8000`\n   served: `google/gemma-4-E2B-it`\n```\n\nA flag being accepted is not evidence it did anything, so every modality is probed with a request whose correct answer is known in advance. The vision probe is a generated checkerboard rather than a fetched photo, so the expected answer is a fact about the image and not a caption.\n\n``` bash\n$ verify_capabilities\n✅ 4/4 capabilities verified.\n\n| text         | ✅ | The AMD MI300X is based on the CDNA 3 architecture. |\n| thinking     | ✅ | 1404 chars, 447 reasoning tokens |\n| tool calling | ✅ | tool_calls → get_weather{\"city\": \"Reykjavik\"} |\n| vision       | ✅ | alternating bright red and royal blue squares |\n\n📡 Audio is not probed — no ROCm vLLM image ships the `vllm[audio]` extras.\n```\n\nThe boot log is worth reading before sizing anything, because the engine states its own arithmetic:\n\n```\nAvailable KV cache memory: 155.04 GiB\nGPU KV cache size: 9,026,017 tokens\nMaximum concurrency for 32,768 tokens per request: 275.45x\n```\n\nNine million tokens of KV on one card. Hold that number — it decides which of the two ceilings below you actually hit.\n\nFour concurrencies against four context lengths, 128 output tokens throughout, three repeats per cell, load generated by vLLM's own bench client in a second container with no GPU device attached.\n\n``` bash\n$ python3 benchmarking_suite.py --droplet debian-gpu-mi300x1-192gb-devcloud-atl1 \\\n    --run-id 2026-09-17-vllm-sweep-mi300x-rerun --repeat 3 --seed-base 27000\n16 cells, 12 runnable, 4 infeasible at max_model_len 32768\n```\n\nThe 32,768 row is infeasible rather than missing: 32,768 input plus 128 output does not fit a 32,768 context, and a cell that cannot exist is recorded as such rather than dropped.\n\nOne detail is load-bearing. `vllm bench serve` derives its prompts from `--seed`, which defaults to 0, and this deployment caches prefixes — so two runs sharing a seed measure the cache, not the card. Every cell and every repeat here uses a seed no other run in the sweep uses.\n\nOutput tokens per second, median of three repeats, worst-cell coefficient of variation 7.96%:\n\n| context | 1 stream | 4 | 16 | 64 | \n|---|---|---|---|---|\n| 128 | 340.9 | 1,124.1 | 3,569.1 | 🥇 **10,284.0** | \n| 1,024 | 305.1 | 969.1 | 2,653.6 | 5,398.2 | \n| 8,192 | 192.8 | 445.5 | 688.2 | 702.7 | \n\nCounting prefill as well, the busiest cell moves **48,583 tokens a second** — 64 streams at 1,024 context. Single-stream latency at 1,024 context is 32.67 ms to first token and 3.04 ms per token after it.\n\nTwo things happen as context grows, and only one of them is the one people plan for.\n\nTime per output token barely moves: 2.85 ms at 128 context, 3.04 at 1,024, 3.9 at 8,192. Time to first token rises steeply: 12.97 ms, 32.67 ms, 169.61 ms. Decode stays cheap; prefill gets expensive.\n\nSo the 8,192 row flattens — 688 tok/s at 16 streams, 703 at 64 — while the 128 row keeps climbing to 10,284. **The constraint at long context is prefill, not memory.** The heaviest cell in the grid wants 64 x 8,192 = 524,288 KV tokens against the 9,026,017 the engine allocated — 5.8% of the pool, by arithmetic.\n\nThat inverts the sizing rule the TPU rigs in this monorepo run on, where KV capacity is the thing you run out of first. On a 192 GB card serving a 2B model, it never becomes the binding constraint.\n\nThroughput divided by hourly rate, in=1024, out=128. Every row is the same checkpoint under vLLM, each from its own schema-valid report:\n\n| rig | $/hr | basis | 1 | 4 | 16 | 64 | \n|---|---|---|---|---|---|---|\n| **MI300X** | 1.99 | on-demand | 153 | 487 | 1,333 | 🥇 **2,713** | \n| TPU v5e-1 | 0.5779 | spot | 🥇 208 | 🥇 711 | 🥇 1,572 | 1,972 | \n| TPU v6e-1 | 2.97 | on-demand | 67 | 233 | 587 | 719 | \n| NVIDIA L4 | 0.94 | spot | 49 | 187 | — | — | \n\nTokens per second per dollar-hour. The MI300X wins the busiest column outright and the v5e wins the other three — but those two rows are not priced the same way, which is the whole of the next section.\n\nA spot rate against an on-demand rate is a discount, not a hardware result. Put all three on on-demand list and the ranking is uniform:\n\n| rig | $/hr | 1 | 4 | 16 | 64 | \n|---|---|---|---|---|---|\n| 🥇 **MI300X** | 1.99 | **153** | **487** | **1,333** | **2,713** | \n| 🥈 TPU v5e-1 | 1.20 | 100 | 342 | 757 | 950 | \n| 🥉 TPU v6e-1 | 2.70 | 74 | 257 | 646 | 790 | \n\nAt matched pricing the MI300X returns **1.4x to 2.9x** the tokens per dollar of a v5e — 1.53x, 1.42x, 1.76x and 2.86x across the four concurrencies, by arithmetic over the two measured rows above.\n\nThat advantage has one real qualifier: DigitalOcean publishes no preemptible tier for GPU droplets, so $1.99 is a floor. The v5e spot rate of $0.5779 per chip-hour is a purchasable option, and at 16 streams or fewer it buys more tokens per dollar than this card does.\n\nSo the honest form of the finding is conditional. If you want a non-preemptible box, or you keep dozens of streams busy, this is the best tokens-per-dollar measured across these rigs. If your traffic is light and you tolerate preemption, a spot TPU is cheaper.\n\nPowering the droplet off does **not** stop DigitalOcean billing it. The resources stay reserved and the hourly rate keeps running; only destroying the droplet stops the meter.\n\nThat is why this toolkit ships no `create` and no `destroy` tool. Both are dollar-per-hour decisions, they stay a deliberate step in the console, and a rate quoted per hour means nothing until you know which hours you are paying for.\n\n``` bash\n$ stop_vllm\n✅ `docker rm -f vllm` exited 0.\n```\n\nThat frees the card's memory. It does not free your wallet.\n\nThe goal of this article was to deploy Gemma 4 E2B to a single AMD Instinct MI300X and measure what its hourly rate returns in tokens. The key to the solution was a tag-scoped MCP toolkit that reaches the card over SSH and the DigitalOcean API, because the machine running it has no GPU. The measured results were:\n\n`docker run` to an endpoint that answers\nOne droplet, one MI300X, region `atl1`, three repeats per cell with a unique prompt seed each, worst-cell coefficient of variation 7.96%. The comparison rows differ from this one in ways worth naming once: the TPU and L4 reports are single runs per cell on earlier vLLM builds, the v5e and L4 figures were measured on spot capacity while the MI300X and v6e were on-demand, and nothing else in this monorepo serves this checkpoint on AMD, so there is no twin to difference against.\n\nThe strategy for using MCP for AMD Instinct deployment and benchmarking was validated with an incremental step by step approach.", "url": "https://wpnews.pro/news/serving-gemma-4-on-an-amd-mi300x-what-1-99-an-hour-buys", "canonical_source": "https://dev.to/gde/serving-gemma-4-on-an-amd-mi300x-what-199-an-hour-buys-52h9", "published_at": "2026-09-17 21:36:22+00:00", "updated_at": "2026-09-17 21:52:54.792721+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-agents", "agent-protocols", "large-language-models", "ai-tools"], "entities": ["AMD", "AMD Instinct MI300X", "Gemma 4", "Google", "vLLM", "DigitalOcean", "AMD Developer Cloud", "ROCm"], "alternates": {"html": "https://wpnews.pro/news/serving-gemma-4-on-an-amd-mi300x-what-1-99-an-hour-buys", "markdown": "https://wpnews.pro/news/serving-gemma-4-on-an-amd-mi300x-what-1-99-an-hour-buys.md", "text": "https://wpnews.pro/news/serving-gemma-4-on-an-amd-mi300x-what-1-99-an-hour-buys.txt", "jsonld": "https://wpnews.pro/news/serving-gemma-4-on-an-amd-mi300x-what-1-99-an-hour-buys.jsonld"}}