{"slug": "same-gpu-same-model-different-cost-per-token-benchmark-qwen3-8-27b-on-your-own", "title": "Same GPU, Same Model, Different Cost per Token: Benchmark Qwen3.8-27B on Your Own Prompts", "summary": "A developer's analysis of EcoHash's speculative-decoding benchmark shows that prompt content alone shifts cost per output token by roughly 1.5× for Qwen3.8-27B served on a single RTX PRO 6000 Blackwell. Running the FP8 checkpoint on vLLM 0.27.1 with the model's multi-token-prediction head enabled, EcoHash found chat-style prompts stayed within a 1.5 s TTFT and 30 ms per-token latency budget only up to 32 concurrent requests ($0.41 per 1M output tokens), versus 48 for math prompts ($0.28) and 64 for a mixed pool through an API gateway ($0.27). The gap traces to the draft head: predictable arithmetic output accepts more speculative tokens, yielding 10.5 ms per token on math versus 15.0 ms on chat even at a single request.", "body_md": "When I'm deciding whether to rent a GPU for an open model, I work out the cost per million output tokens:\n\n```\ncost per 1M output tokens = hourly price ÷ (output tokens/s × 3,600) × 1,000,000\n```\n\nThe hourly price is easy to look up. Tokens per second is the number that's easy to get wrong, because it depends on what you send the model.\n\nFor Qwen3.8-27B on a single RTX PRO 6000 Blackwell (96 GB), the most detailed public numbers I found are in [EcoHash's speculative-decoding benchmark](https://github.com/ecohash-ai/ecohash-benchmarks/blob/main/llm/speculative-decoding-qwen3.8-27b.md). They served the FP8 checkpoint on vLLM 0.27.1 with the model's built-in multi-token-prediction (MTP) head switched on, which is a form of speculative decoding. Then they swept from 1 to 128 concurrent requests, with 150-token replies.\n\nThe headline number isn't the interesting part. What caught my eye is that they ran the same sweep with different prompt sets, and the results don't match.\n\nTheir latency budget was a 95th-percentile time to first token of 1.5 s or less, and a 95th-percentile time per output token of 30 ms or less. Here's the highest concurrency that stayed inside it:\n\n| Prompt set | Highest concurrency in budget | Output tokens/s there | Cost per 1M output tokens at $1.89/hr | \n|---|---|---|---|\n| Chat (MT-Bench-style) | 32 | 1,274 | $0.41 | \n| Math (GSM8K-style) | 48 | 1,862 | $0.28 | \n| Mixed pool, through their API gateway | 64 | 1,954 | $0.27 | \n\nLine chart of 95th-percentile time per token against concurrency for Qwen3.8-27B on one RTX PRO 6000. Chat prompts cross the 30 ms budget after 32 concurrent requests, math prompts after 48.\n\nThe card, the weights and the flags were the same in every run. The prompts alone moved the cost per token by about 1.5×.\n\nThe reason is the draft head. MTP guesses a few tokens ahead, and the model checks the guesses in one pass. Predictable output like arithmetic gets more guesses accepted, so each step yields more tokens. Open-ended chat gets fewer. The gap shows up even with a single request: 10.5 ms per token on math against 15.0 ms on chat.\n\nTwo more things will pull your numbers away from theirs. Qwen3.8-27B reasons at length, so real replies usually run well past 150 tokens. And long prompts add prefill time.\n\nSo before I trust anyone's tokens per second, these included, I'd measure my own.\n\nAny provider with the RTX PRO 6000 Blackwell will do. I priced it with EcoHash's [$1.89 per GPU-hour](https://ecohash.com/gpu/rtx-pro-6000), billed per second. Once the machine boots, run:\n\n```\nnvidia-smi --query-gpu=name,memory.total,compute_cap --format=csv\n```\n\nYou want to see \"RTX PRO 6000 Blackwell\", close to 96 GB of memory, and compute capability 12.0.\n\nThis is the config from EcoHash's [deployment guide for this model](https://ecohash.com/blog/deploy-qwen38-27b-rtx-pro-6000), and it's the setup behind the numbers above:\n\n```\nvllm serve /path/to/Qwen3.8-27B-FP8 \\\n  --served-model-name qwen3.8-27b \\\n  --dtype bfloat16 \\\n  --tensor-parallel-size 1 \\\n  --max-model-len 32768 \\\n  --gpu-memory-utilization 0.90 \\\n  --max-num-seqs 128 \\\n  --enable-prefix-caching \\\n  --enable-prompt-tokens-details \\\n  --reasoning-parser qwen3 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser qwen3_xml \\\n  --safetensors-load-strategy prefetch \\\n  --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}'\n```\n\nA few of these flags matter more than they look, and the guide explains why. In short: use `bfloat16`, not `float16`. Don't add `--quantization`, because the checkpoint is already FP8. And use the `qwen3_xml` tool parser. `--max-model-len 32768` is a choice rather than a limit of the model, so raise it if your prompts are longer. Longer requests take more cache each, so expect a lower ceiling. The benchmark will show you how much lower.\n\nPull 20 to 50 real prompts from your logs into `prompts.jsonl`, one per line. Each line can be `{\"prompt\": \"...\"}` or a full `{\"messages\": [...]}` conversation that includes your system prompt. Set `--max-tokens` close to your real reply length. For a reasoning model, that's probably well above 150.\n\n```\n\"\"\"Measure an OpenAI-compatible endpoint at several concurrency levels,\nthen turn the results into a cost per 1M output tokens.\n\n    python bench.py --base-url http://localhost:8000/v1 --model qwen3.8-27b \\\n        --prompts prompts.jsonl --levels 1,8,16,32,48,64 --max-tokens 512 --price 1.89\n\"\"\"\nimport argparse, asyncio, json, math, os, time\nfrom openai import AsyncOpenAI\n\ndef p95(xs):\n    xs = sorted(xs)\n    return xs[math.ceil(0.95 * len(xs)) - 1] if xs else float(\"nan\")\n\ndef has_text(delta):\n    # Reasoning models stream their thinking in a separate field before the answer.\n    extra = delta.model_extra or {}\n    return bool(delta.content or extra.get(\"reasoning_content\") or extra.get(\"reasoning\"))\n\nasync def one_request(client, model, messages, max_tokens):\n    t0 = time.perf_counter()\n    first, tokens = None, 0\n    stream = await client.chat.completions.create(\n        model=model, messages=messages, max_tokens=max_tokens, temperature=0.5,\n        stream=True, stream_options={\"include_usage\": True},\n    )\n    async for chunk in stream:\n        if first is None and chunk.choices and has_text(chunk.choices[0].delta):\n            first = time.perf_counter()\n        if chunk.usage:\n            tokens = chunk.usage.completion_tokens\n    end = time.perf_counter()\n    if first is None or tokens < 2:\n        return None\n    # Divide by tokens, not chunks: with speculative decoding one chunk can carry several.\n    return first - t0, (end - first) / (tokens - 1), tokens\n\nasync def run_level(client, args, prompts, conc):\n    for i in range(args.warmup):\n        await one_request(client, args.model, prompts[i % len(prompts)], args.max_tokens)\n    queue = asyncio.Queue()\n    for i in range(max(24, 3 * conc)):\n        queue.put_nowait(prompts[i % len(prompts)])\n    results, errors = [], 0\n\n    async def worker():\n        nonlocal errors\n        while True:\n            try:\n                messages = queue.get_nowait()\n            except asyncio.QueueEmpty:\n                return\n            try:\n                r = await one_request(client, args.model, messages, args.max_tokens)\n                if r:\n                    results.append(r)\n                else:\n                    errors += 1\n            except Exception as e:\n                errors += 1\n                print(f\"  request failed: {e!r}\")\n\n    t0 = time.perf_counter()\n    await asyncio.gather(*(worker() for _ in range(conc)))\n    wall = time.perf_counter() - t0\n    tok_s = sum(r[2] for r in results) / wall\n    return {\n        \"conc\": conc, \"ok\": len(results), \"errors\": errors, \"tok_s\": tok_s,\n        \"ttft_p95_ms\": p95([r[0] for r in results]) * 1000,\n        \"tpot_p95_ms\": p95([r[1] for r in results]) * 1000,\n        \"usd_per_1m\": args.price / (tok_s * 3600) * 1e6 if tok_s else float(\"nan\"),\n    }\n\ndef load_prompts(path):\n    prompts = []\n    with open(path) as f:\n        for line in f:\n            if line.strip():\n                row = json.loads(line)\n                prompts.append(row[\"messages\"] if \"messages\" in row\n                               else [{\"role\": \"user\", \"content\": row[\"prompt\"]}])\n    return prompts\n\nasync def main():\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--base-url\", default=\"http://localhost:8000/v1\")\n    ap.add_argument(\"--model\", default=\"qwen3.8-27b\")\n    ap.add_argument(\"--prompts\", required=True, help=\"JSONL: {\\\"prompt\\\": ...} or {\\\"messages\\\": [...]}\")\n    ap.add_argument(\"--levels\", default=\"1,8,16,32,48,64\")\n    ap.add_argument(\"--max-tokens\", type=int, default=512)\n    ap.add_argument(\"--warmup\", type=int, default=2)\n    ap.add_argument(\"--price\", type=float, default=1.89, help=\"GPU price, $/hour\")\n    ap.add_argument(\"--ttft-budget\", type=float, default=1500, help=\"p95 ms\")\n    ap.add_argument(\"--tpot-budget\", type=float, default=30, help=\"p95 ms per token\")\n    args = ap.parse_args()\n\n    client = AsyncOpenAI(base_url=args.base_url, api_key=os.environ.get(\"API_KEY\", \"none\"),\n                         timeout=600, max_retries=0)\n    prompts = load_prompts(args.prompts)\n    rows = []\n    for conc in map(int, args.levels.split(\",\")):\n        row = await run_level(client, args, prompts, conc)\n        row[\"fits\"] = (row[\"errors\"] == 0 and row[\"ttft_p95_ms\"] <= args.ttft_budget\n                       and row[\"tpot_p95_ms\"] <= args.tpot_budget)\n        rows.append(row)\n        print(f\"C={conc}: {row['tok_s']:.0f} tok/s, TPOT p95 {row['tpot_p95_ms']:.1f} ms\")\n\n    print(\"\\n| conc | ok | errors | output tok/s | TTFT p95 ms | TPOT p95 ms | $ per 1M out | in budget |\")\n    print(\"|---|---|---|---|---|---|---|---|\")\n    for r in rows:\n        print(f\"| {r['conc']} | {r['ok']} | {r['errors']} | {r['tok_s']:.0f} | {r['ttft_p95_ms']:.0f} \"\n              f\"| {r['tpot_p95_ms']:.1f} | ${r['usd_per_1m']:.2f} | {'yes' if r['fits'] else 'no'} |\")\n    best = max((r for r in rows if r[\"fits\"]), key=lambda r: r[\"tok_s\"], default=None)\n    if best:\n        print(f\"\\nBest level inside budget: C={best['conc']}, {best['tok_s']:.0f} tok/s, \"\n              f\"${best['usd_per_1m']:.2f} per 1M output tokens at ${args.price}/hr\")\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\nRun it on the GPU instance itself, against `localhost:8000`, so you're measuring the card and not your network. If you have to go through a public endpoint, put its key in the `API_KEY` environment variable.\n\n```\npip install openai\npython bench.py --prompts prompts.jsonl --levels 1,8,16,32,48,64 --max-tokens 1024 --price 1.89\n```\n\nA few choices in the script are deliberate:\n\nThe script prints a markdown table and names the best level inside your budget. If your budget is different, change `--tpot-budget` and `--ttft-budget`. For scale, 30 ms per token is about 33 tokens per second for each stream.\n\nThen compare that cost per 1M with what an API would charge you for the same model. EcoHash's own API lists Qwen3.8-27B at $0.60 per 1M output tokens, and the 21 providers on OpenRouter ranged from $1.25 to $3.20 when I checked on November 1. If the card can't beat the API you'd otherwise use at a realistic load, renting it is about control, such as a longer context or your own serving flags, not about price.\n\nKeep in mind that the cost at the best level assumes the card stays that busy every hour you pay for it. Real traffic doesn't. Scale the number by your average load, not your peak.\n\nAt $1.89 an hour, billed per second, a 30-minute session costs about 95 cents, and EcoHash's $1 signup credit covers roughly that. Terminate the instance when you're done, because the clock keeps running while it sits idle.", "url": "https://wpnews.pro/news/same-gpu-same-model-different-cost-per-token-benchmark-qwen3-8-27b-on-your-own", "canonical_source": "https://dev.to/yyyysa4/same-gpu-same-model-different-cost-per-token-benchmark-qwen38-27b-on-your-own-prompts-5hm0", "published_at": "2026-11-02 02:14:31+00:00", "updated_at": "2026-11-02 02:22:11.057854+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-chips", "mlops"], "entities": ["EcoHash", "Qwen3.8-27B", "RTX PRO 6000 Blackwell", "vLLM", "NVIDIA"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/same-gpu-same-model-different-cost-per-token-benchmark-qwen3-8-27b-on-your-own", "markdown": "https://wpnews.pro/news/same-gpu-same-model-different-cost-per-token-benchmark-qwen3-8-27b-on-your-own.md", "text": "https://wpnews.pro/news/same-gpu-same-model-different-cost-per-token-benchmark-qwen3-8-27b-on-your-own.txt", "jsonld": "https://wpnews.pro/news/same-gpu-same-model-different-cost-per-token-benchmark-qwen3-8-27b-on-your-own.jsonld"}}