Same GPU, Same Model, Different Cost per Token: Benchmark Qwen3.8-27B on Your Own Prompts

A developer's analysis of EcoHash's speculative-decoding benchmark shows that prompt content alone shifts cost per output token by roughly 1.5× for Qwen3.8-27B served on a single RTX PRO 6000 Blackwell. Running the FP8 checkpoint on vLLM 0.27.1 with the model's multi-token-prediction head enabled, EcoHash found chat-style prompts stayed within a 1.5 s TTFT and 30 ms per-token latency budget only up to 32 concurrent requests ($0.41 per 1M output tokens), versus 48 for math prompts ($0.28) and 64 for a mixed pool through an API gateway ($0.27). The gap traces to the draft head: predictable arithmetic output accepts more speculative tokens, yielding 10.5 ms per token on math versus 15.0 ms on chat even at a single request.

When I'm deciding whether to rent a GPU for an open model, I work out the cost per million output tokens: cost per 1M output tokens = hourly price ÷ output tokens/s × 3,600 × 1,000,000 The hourly price is easy to look up. Tokens per second is the number that's easy to get wrong, because it depends on what you send the model. For Qwen3.8-27B on a single RTX PRO 6000 Blackwell 96 GB , the most detailed public numbers I found are in EcoHash's speculative-decoding benchmark https://github.com/ecohash-ai/ecohash-benchmarks/blob/main/llm/speculative-decoding-qwen3.8-27b.md . They served the FP8 checkpoint on vLLM 0.27.1 with the model's built-in multi-token-prediction MTP head switched on, which is a form of speculative decoding. Then they swept from 1 to 128 concurrent requests, with 150-token replies. The headline number isn't the interesting part. What caught my eye is that they ran the same sweep with different prompt sets, and the results don't match. Their latency budget was a 95th-percentile time to first token of 1.5 s or less, and a 95th-percentile time per output token of 30 ms or less. Here's the highest concurrency that stayed inside it: | Prompt set | Highest concurrency in budget | Output tokens/s there | Cost per 1M output tokens at $1.89/hr | |---|---|---|---| | Chat MT-Bench-style | 32 | 1,274 | $0.41 | | Math GSM8K-style | 48 | 1,862 | $0.28 | | Mixed pool, through their API gateway | 64 | 1,954 | $0.27 | Line chart of 95th-percentile time per token against concurrency for Qwen3.8-27B on one RTX PRO 6000. Chat prompts cross the 30 ms budget after 32 concurrent requests, math prompts after 48. The card, the weights and the flags were the same in every run. The prompts alone moved the cost per token by about 1.5×. The reason is the draft head. MTP guesses a few tokens ahead, and the model checks the guesses in one pass. Predictable output like arithmetic gets more guesses accepted, so each step yields more tokens. Open-ended chat gets fewer. The gap shows up even with a single request: 10.5 ms per token on math against 15.0 ms on chat. Two more things will pull your numbers away from theirs. Qwen3.8-27B reasons at length, so real replies usually run well past 150 tokens. And long prompts add prefill time. So before I trust anyone's tokens per second, these included, I'd measure my own. Any provider with the RTX PRO 6000 Blackwell will do. I priced it with EcoHash's $1.89 per GPU-hour https://ecohash.com/gpu/rtx-pro-6000 , billed per second. Once the machine boots, run: nvidia-smi --query-gpu=name,memory.total,compute cap --format=csv You want to see "RTX PRO 6000 Blackwell", close to 96 GB of memory, and compute capability 12.0. This is the config from EcoHash's deployment guide for this model https://ecohash.com/blog/deploy-qwen38-27b-rtx-pro-6000 , and it's the setup behind the numbers above: vllm serve /path/to/Qwen3.8-27B-FP8 \ --served-model-name qwen3.8-27b \ --dtype bfloat16 \ --tensor-parallel-size 1 \ --max-model-len 32768 \ --gpu-memory-utilization 0.90 \ --max-num-seqs 128 \ --enable-prefix-caching \ --enable-prompt-tokens-details \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser qwen3 xml \ --safetensors-load-strategy prefetch \ --speculative-config '{"method":"mtp","num speculative tokens":3}' A few of these flags matter more than they look, and the guide explains why. In short: use bfloat16 , not float16 . Don't add --quantization , because the checkpoint is already FP8. And use the qwen3 xml tool parser. --max-model-len 32768 is a choice rather than a limit of the model, so raise it if your prompts are longer. Longer requests take more cache each, so expect a lower ceiling. The benchmark will show you how much lower. Pull 20 to 50 real prompts from your logs into prompts.jsonl , one per line. Each line can be {"prompt": "..."} or a full {"messages": ... } conversation that includes your system prompt. Set --max-tokens close to your real reply length. For a reasoning model, that's probably well above 150. """Measure an OpenAI-compatible endpoint at several concurrency levels, then turn the results into a cost per 1M output tokens. python bench.py --base-url http://localhost:8000/v1 --model qwen3.8-27b \ --prompts prompts.jsonl --levels 1,8,16,32,48,64 --max-tokens 512 --price 1.89 """ import argparse, asyncio, json, math, os, time from openai import AsyncOpenAI def p95 xs : xs = sorted xs return xs math.ceil 0.95 len xs - 1 if xs else float "nan" def has text delta : Reasoning models stream their thinking in a separate field before the answer. extra = delta.model extra or {} return bool delta.content or extra.get "reasoning content" or extra.get "reasoning" async def one request client, model, messages, max tokens : t0 = time.perf counter first, tokens = None, 0 stream = await client.chat.completions.create model=model, messages=messages, max tokens=max tokens, temperature=0.5, stream=True, stream options={"include usage": True}, async for chunk in stream: if first is None and chunk.choices and has text chunk.choices 0 .delta : first = time.perf counter if chunk.usage: tokens = chunk.usage.completion tokens end = time.perf counter if first is None or tokens < 2: return None Divide by tokens, not chunks: with speculative decoding one chunk can carry several. return first - t0, end - first / tokens - 1 , tokens async def run level client, args, prompts, conc : for i in range args.warmup : await one request client, args.model, prompts i % len prompts , args.max tokens queue = asyncio.Queue for i in range max 24, 3 conc : queue.put nowait prompts i % len prompts results, errors = , 0 async def worker : nonlocal errors while True: try: messages = queue.get nowait except asyncio.QueueEmpty: return try: r = await one request client, args.model, messages, args.max tokens if r: results.append r else: errors += 1 except Exception as e: errors += 1 print f" request failed: {e r}" t0 = time.perf counter await asyncio.gather worker for in range conc wall = time.perf counter - t0 tok s = sum r 2 for r in results / wall return { "conc": conc, "ok": len results , "errors": errors, "tok s": tok s, "ttft p95 ms": p95 r 0 for r in results 1000, "tpot p95 ms": p95 r 1 for r in results 1000, "usd per 1m": args.price / tok s 3600 1e6 if tok s else float "nan" , } def load prompts path : prompts = with open path as f: for line in f: if line.strip : row = json.loads line prompts.append row "messages" if "messages" in row else {"role": "user", "content": row "prompt" } return prompts async def main : ap = argparse.ArgumentParser ap.add argument "--base-url", default="http://localhost:8000/v1" ap.add argument "--model", default="qwen3.8-27b" ap.add argument "--prompts", required=True, help="JSONL: {\"prompt\": ...} or {\"messages\": ... }" ap.add argument "--levels", default="1,8,16,32,48,64" ap.add argument "--max-tokens", type=int, default=512 ap.add argument "--warmup", type=int, default=2 ap.add argument "--price", type=float, default=1.89, help="GPU price, $/hour" ap.add argument "--ttft-budget", type=float, default=1500, help="p95 ms" ap.add argument "--tpot-budget", type=float, default=30, help="p95 ms per token" args = ap.parse args client = AsyncOpenAI base url=args.base url, api key=os.environ.get "API KEY", "none" , timeout=600, max retries=0 prompts = load prompts args.prompts rows = for conc in map int, args.levels.split "," : row = await run level client, args, prompts, conc row "fits" = row "errors" == 0 and row "ttft p95 ms" <= args.ttft budget and row "tpot p95 ms" <= args.tpot budget rows.append row print f"C={conc}: {row 'tok s' :.0f} tok/s, TPOT p95 {row 'tpot p95 ms' :.1f} ms" print "\n| conc | ok | errors | output tok/s | TTFT p95 ms | TPOT p95 ms | $ per 1M out | in budget |" print "|---|---|---|---|---|---|---|---|" for r in rows: print f"| {r 'conc' } | {r 'ok' } | {r 'errors' } | {r 'tok s' :.0f} | {r 'ttft p95 ms' :.0f} " f"| {r 'tpot p95 ms' :.1f} | ${r 'usd per 1m' :.2f} | {'yes' if r 'fits' else 'no'} |" best = max r for r in rows if r "fits" , key=lambda r: r "tok s" , default=None if best: print f"\nBest level inside budget: C={best 'conc' }, {best 'tok s' :.0f} tok/s, " f"${best 'usd per 1m' :.2f} per 1M output tokens at ${args.price}/hr" if name == " main ": asyncio.run main Run it on the GPU instance itself, against localhost:8000 , so you're measuring the card and not your network. If you have to go through a public endpoint, put its key in the API KEY environment variable. pip install openai python bench.py --prompts prompts.jsonl --levels 1,8,16,32,48,64 --max-tokens 1024 --price 1.89 A few choices in the script are deliberate: The script prints a markdown table and names the best level inside your budget. If your budget is different, change --tpot-budget and --ttft-budget . For scale, 30 ms per token is about 33 tokens per second for each stream. Then compare that cost per 1M with what an API would charge you for the same model. EcoHash's own API lists Qwen3.8-27B at $0.60 per 1M output tokens, and the 21 providers on OpenRouter ranged from $1.25 to $3.20 when I checked on November 1. If the card can't beat the API you'd otherwise use at a realistic load, renting it is about control, such as a longer context or your own serving flags, not about price. Keep in mind that the cost at the best level assumes the card stays that busy every hour you pay for it. Real traffic doesn't. Scale the number by your average load, not your peak. At $1.89 an hour, billed per second, a 30-minute session costs about 95 cents, and EcoHash's $1 signup credit covers roughly that. Terminate the instance when you're done, because the clock keeps running while it sits idle.