{"slug": "qwen3-8-27b-nvfp4-mtp-benchmark-on-rtx-5090-tier-comparison-64k-long-context-n", "title": "Qwen3.8-27B NVFP4-MTP benchmark on RTX 5090 (tier comparison, 64K long-context, spec-draft-n-max sweep) + mtp-bench.py tool", "summary": "A developer benchmarked the Qwen3.8-27B model with NVFP4 quantization and multi-token prediction (MTP) on an RTX 5090, comparing tier configurations and sweeping spec-draft-n values at 64K context. They also released mtp-bench.py, a tool for running these benchmarks.", "body_md": "|\n#!/usr/bin/env python3 |\n|\nimport argparse, json, os, sys, time |\n|\nfrom urllib import request |\n|\n|\n|\nPROMPTS = [ |\n|\n{\"name\": \"code_python\", \"prompt\": \"Write a Python function that returns the n-th Fibonacci number using memoization. Include a docstring.\"}, |\n|\n{\"name\": \"code_cpp\", \"prompt\": \"Write a C++ template function `clamp(x, lo, hi)` that returns x clamped to [lo, hi]. No std::clamp.\"}, |\n|\n{\"name\": \"explain_concept\", \"prompt\": \"Explain how speculative decoding works in large language model inference, in three short paragraphs.\"}, |\n|\n{\"name\": \"summarize\", \"prompt\": \"Summarize in two sentences: The Industrial Revolution began in Britain in the late 18th century, transforming manufacturing through mechanization, steam power, and the factory system. It spread to continental Europe and North America during the 19th century.\"}, |\n|\n{\"name\": \"qa_factual\", \"prompt\": \"Q: What are the four fundamental forces of physics?\\nA:\"}, |\n|\n{\"name\": \"translation\", \"prompt\": \"Translate to French: 'The quick brown fox jumps over the lazy dog.'\"}, |\n|\n{\"name\": \"creative_short\", \"prompt\": \"Write a four-line poem about an old lighthouse.\"}, |\n|\n{\"name\": \"stepwise_math\", \"prompt\": \"Solve step by step: A train leaves station A at 60 km/h. Two hours later, a second train leaves the same station on the same track at 90 km/h. How long until the second train catches the first?\"}, |\n|\n{\"name\": \"long_code_review\", \"prompt\": ( |\n|\n\"You are reviewing a backend service that has been suffering intermittent latency spikes \" |\n|\n\"in production. Below is the relevant code and a description of the system. After reading \" |\n|\n\"carefully, produce a structured review with three sections: (1) likely root causes ranked \" |\n|\n\"by probability, (2) concrete code or configuration changes you would make first, \" |\n|\n\"(3) what telemetry you would add to confirm the diagnosis.\\n\\n\" |\n|\n\"System description: a Python FastAPI service in front of a Postgres 15 database, deployed \" |\n|\n\"as four replicas behind an nginx load balancer. Each request reads a user record, fetches \" |\n|\n\"their last 50 events from a partitioned events table, computes an aggregate score, writes \" |\n|\n\"the score back to the user row, and returns a JSON response. Average payload is 4 KB. \" |\n|\n\"p50 latency is 35 ms; p99 spikes to 1.8 seconds approximately every 90 seconds in a \" |\n|\n\"regular pattern. The spikes correlate with elevated Postgres CPU but not with elevated \" |\n|\n\"Postgres connection count. The application pool is sized at 20 connections per replica. \" |\n|\n\"PgBouncer is in front of Postgres in transaction pooling mode with a pool size of 50.\\n\\n\" |\n|\n\"Code excerpt — the hot endpoint:\\n\" |\n|\n\"``` python\\n@app.post('/score/{user_id}')\\nasync def score(user_id: int, payload: ScoreRequest):\\n\" |\n|\n\" async with db.transaction() as tx:\\n user = await tx.fetchrow(\\n\" |\n|\n\" 'SELECT id, tier, last_score FROM users WHERE id = $1 FOR UPDATE',\\n user_id,\\n )\\n\" |\n|\n\" if user is None:\\n raise HTTPException(404)\\n events = await tx.fetch(\\n\" |\n|\n\" 'SELECT type, weight, ts FROM events '\\n 'WHERE user_id = $1 ORDER BY ts DESC LIMIT 50',\\n user_id,\\n )\\n\" |\n|\n\" new_score = compute_score(user['tier'], events, payload.signals)\\n\" |\n|\n\" await tx.execute(\\n 'UPDATE users SET last_score = $1, updated_at = now() WHERE id = $2',\\n new_score, user_id,\\n )\\n\" |\n|\n\" await tx.execute(\\n 'INSERT INTO score_history (user_id, score, ts) VALUES ($1, $2, now())',\\n user_id, new_score,\\n )\\n\" |\n|\n\" await cache.set(f'score:{user_id}', new_score, ex=300)\\n\" |\n|\n\" metrics.histogram('score.latency_ms').observe((time.time() - start) * 1000)\\n\" |\n|\n\" return {'user_id': user_id, 'score': new_score}\\n```\\n\\n\" |\n|\n\"Schema notes: `users` is ~50M rows, `events` is partitioned by month with ~2B rows total \" |\n|\n\"and a btree index on `(user_id, ts DESC)`. `score_history` is unpartitioned, ~800M rows, \" |\n|\n\"with a single index on `user_id`. Postgres autovacuum is at default settings. There is \" |\n|\n\"a nightly batch job that rebuilds materialized views starting at 02:00 UTC; spikes occur \" |\n|\n\"throughout the day, not just during the batch window. Connection pooling metrics show \" |\n|\n\"PgBouncer waiting connections occasionally hit 8-12 during spikes but never saturate. \" |\n|\n\"CPU on the FastAPI replicas stays below 30% even during spikes. Network round-trip time \" |\n|\n\"between the application and Postgres is consistently 0.4 ms.\\n\\nBegin your review now.\" |\n|\n)}, |\n|\n] |\n|\n|\n|\ndef estimate_tokens(text): |\n|\n\"\"\"Very rough token estimate: ~4 chars per English token.\"\"\" |\n|\nreturn len(text) // 4 |\n|\n|\n|\ndef generate_long_prompt(target_tokens=64000): |\n|\n\"\"\"Generate a long context prompt by repeating a passage until we hit target_tokens.\"\"\" |\n|\npassage = ( |\n|\n\"The Industrial Revolution began in Britain in the late 18th century, transforming manufacturing \" |\n|\n\"through mechanization, steam power, and the factory system. It spread to continental Europe and \" |\n|\n\"North America during the 19th century. Key innovations included the spinning jenny, the water frame, \" |\n|\n\"and James Watt's improved steam engine. These inventions dramatically increased production capacity \" |\n|\n\"and led to urbanization as workers moved from rural areas to factory towns. Social changes included \" |\n|\n\"the rise of the middle class, labor movements, and new political ideologies such as liberalism, \" |\n|\n\"conservatism, and socialism. Economic shifts included the decline of feudalism, the growth of global \" |\n|\n\"trade, and the establishment of modern banking systems. The transportation revolution brought \" |\n|\n\"canals, turnpikes, railroads, and steamships, reducing costs and connecting markets across vast distances. \" |\n|\n\"These transformations laid the foundation for the modern world economy.\\n\" |\n|\n) |\n|\npassage_tokens = estimate_tokens(passage) |\n|\nif passage_tokens == 0: |\n|\nreturn \"Empty prompt\" |\n|\nrepeats_needed = max(1, target_tokens // passage_tokens) |\n|\nlong_text = passage * repeats_needed |\n|\n# Trim to approximate target (±10%) |\n|\napprox_tokens = estimate_tokens(long_text) |\n|\nif approx_tokens > target_tokens * 1.1: |\n|\nword_limit = int(target_tokens * 2) # ~2 chars per word |\n|\nlong_text = \" \".join(long_text.split()[:word_limit]) |\n|\nreturn ( |\n|\nf\"You will be given a very long document about the Industrial Revolution. \" |\n|\nf\"Read it carefully and answer questions at the end.\\n\" |\n|\nf\"Estimated tokens: ~{estimate_tokens(long_text)}\\n\\n\" |\n|\nf\"{long_text}\\n\\n\" |\n|\nf\"Q1: What were the key technological innovations mentioned? \" |\n|\nf\"Q2: What social changes occurred during this period? \" |\n|\nf\"Q3: How did transportation evolve?\" |\n|\n) |\n|\n|\n|\ndef load_long_prompt(file_path, target_tokens=64000): |\n|\n\"\"\"Load a text file and use it as context, trimming to approximately target_tokens.\"\"\" |\n|\nif not os.path.exists(file_path): |\n|\nprint(f\"ERROR: File not found: {file_path}\"); sys.exit(1) |\n|\nwith open(file_path, \"r\", encoding=\"utf-8\") as f: |\n|\ntext = f.read() |\n|\nfile_tokens = estimate_tokens(text) |\n|\nif file_tokens > target_tokens * 2: |\n|\nword_limit = int(target_tokens * 2) |\n|\ntext = \" \".join(text.split()[:word_limit]) |\n|\nreturn ( |\n|\nf\"You will be given a long document. Read it carefully and answer the questions below.\\n\" |\n|\nf\"File: {file_path}\\nEstimated tokens: ~{estimate_tokens(text)}\\n\\n\" |\n|\nf\"{text}\\n\\n\" |\n|\nf\"Q1: What were the main topics discussed? \" |\n|\nf\"Q2: Summarize the key points in three sentences.\" |\n|\n) |\n|\n|\n|\ndef post(url, payload): |\n|\nreq = request.Request(url, data=json.dumps(payload).encode(), headers={\"Content-Type\":\"application/json\"}, method=\"POST\") |\n|\nwith request.urlopen(req, timeout=300) as r: |\n|\nreturn json.loads(r.read()) |\n|\n|\n|\ndef run(args): |\n|\nout = {\"results\": []} |\n|\n|\n|\n# Build the full prompt list, adding long-context variant if requested |\n|\nprompts_to_run = list(PROMPTS) |\n|\nif args.long_context: |\n|\nif args.context_file: |\n|\ncontent = load_long_prompt(args.context_file, args.context_size) |\n|\nelse: |\n|\ncontent = generate_long_prompt(args.context_size) |\n|\ntoken_est = estimate_tokens(content) |\n|\nprompts_to_run.append({ |\n|\n\"name\": f\"long_ctx_{args.context_size // 1000}k\", |\n|\n\"prompt\": content, |\n|\n\"_token_estimate\": token_est, |\n|\n}) |\n|\nprint(f\"\\n[+] Long context prompt: ~{token_est:,} tokens (target: {args.context_size:,})\") |\n|\n|\n|\nfor p in prompts_to_run: |\n|\nt0 = time.time() |\n|\nr = post(f\"{args.url}/v1/chat/completions\", { |\n|\n\"model\": \"qwen-3.8-reasoning\", |\n|\n\"messages\": [{\"role\": \"user\", \"content\": p[\"prompt\"]}], |\n|\n\"max_tokens\": 256 if \"long_ctx\" in (p.get(\"name\", \"\")) else 192, |\n|\n\"seed\": 42, |\n|\n}) |\n|\nwall = time.time() - t0 |\n|\n# OpenAI-compatible endpoint: timings are in usage or top-level |\n|\nusage = r.get(\"usage\", {}) or {} |\n|\nt = r.get(\"timings\", {}) or {} |\n|\npredicted_n = usage.get(\"completion_tokens\") or t.get(\"predicted_n\") |\n|\npredicted_per_second = t.get(\"predicted_per_second\") or (predicted_n / wall if wall > 0 else 0) |\n|\nrec = {\"name\": p[\"name\"], \"wall_s\": round(wall,3), |\n|\n\"predicted_n\": predicted_n, \"predicted_per_second\": round(predicted_per_second, 2), |\n|\n\"draft_n\": t.get(\"draft_n\",0), \"draft_n_accepted\": t.get(\"draft_n_accepted\",0)} |\n|\nrec[\"accept_rate\"] = round(rec[\"draft_n_accepted\"]/rec[\"draft_n\"],4) if rec[\"draft_n\"] else None |\n|\nout[\"results\"].append(rec) |\n|\nar = f\"{rec['accept_rate']:.3f}\" if rec[\"accept_rate\"] is not None else \"n/a\" |\n|\ntok_est = p.get(\"_token_estimate\", 0) |\n|\nest_str = f\" ctx={tok_est:,}t\" if tok_est else \"\" |\n|\nprint(f\" {rec['name']:<18} pred={rec['predicted_n']:>4} draft={rec['draft_n']:>4} acc={rec['draft_n_accepted']:>4} rate={ar} tok/s={rec['predicted_per_second']:.1f}{est_str}\") |\n|\ntd = sum(x[\"draft_n\"] or 0 for x in out[\"results\"]) |\n|\nta = sum(x[\"draft_n_accepted\"] or 0 for x in out[\"results\"]) |\n|\ntp = sum(x[\"predicted_n\"] or 0 for x in out[\"results\"]) |\n|\ntw = sum(x[\"wall_s\"] for x in out[\"results\"]) |\n|\nout[\"aggregate\"] = {\"n_requests\": len(out[\"results\"]), \"total_predicted\": tp, \"total_draft\": td, \"total_draft_accepted\": ta, |\n|\n\"aggregate_accept_rate\": round(ta/td,4) if td else None, \"wall_s_total\": round(tw,2)} |\n|\nprint(\"\\nAggregate:\", json.dumps(out[\"aggregate\"], indent=2)) |\n|\nif args.out: |\n|\njson.dump(out, open(args.out,\"w\"), indent=2); print(\"Wrote\", args.out) |\n|\n|\n|\ndef diff(a, b): |\n|\nA, B = json.load(open(a)), json.load(open(b)) |\n|\nprint(f\"{'metric':<24} {'A':>14} {'B':>14} {'delta':>10}\") |\n|\nfor k in (\"aggregate_accept_rate\",\"total_predicted\",\"total_draft\",\"total_draft_accepted\",\"wall_s_total\"): |\n|\nva, vb = A[\"aggregate\"].get(k), B[\"aggregate\"].get(k) |\n|\nif va is None or vb is None: print(f\"{k:<24} {str(va):>14} {str(vb):>14}\"); continue |\n|\nd = vb - va |\n|\ns = f\"{d:>+10.4f}\" if isinstance(d,float) else f\"{d:>+10}\" |\n|\nprint(f\"{k:<24} {va:>14} {vb:>14} {s}\") |\n|\nby_a = {x[\"name\"]: x for x in A[\"results\"]} |\n|\nprint(\"\\n{:<20} {:>8} {:>8} {:>8}\".format(\"prompt\",\"A\",\"B\",\"delta\")) |\n|\nfor rb in B[\"results\"]: |\n|\nra = by_a.get(rb[\"name\"]) or {} |\n|\nar = ra.get(\"accept_rate\") or 0; br = rb.get(\"accept_rate\") or 0 |\n|\nprint(f\"{rb['name']:<20} {ar:>8.3f} {br:>8.3f} {br-ar:>+8.3f}\") |\n|\n|\n|\nap = argparse.ArgumentParser(description=\"MTP Bench — test speculative decoding performance\") |\n|\nap.add_argument(\"--url\", default=\"http://127.0.0.1:8080\") |\n|\nap.add_argument(\"--out\") |\n|\nap.add_argument(\"--diff\", nargs=2) |\n|\n# Long context arguments |\n|\nap.add_argument(\"--long-context\", action=\"store_true\", help=\"Add a large-context prompt to the benchmark\") |\n|\nap.add_argument(\"--context-size\", type=int, default=64000, help=\"Target context size in tokens (default: 64000)\") |\n|\nap.add_argument(\"--context-file\", type=str, default=None, help=\"Path to a text file to use as long context\") |\n|\na = ap.parse_args() |\n|\nif a.diff: diff(*a.diff) |\n|\nelse: run(a) |", "url": "https://wpnews.pro/news/qwen3-8-27b-nvfp4-mtp-benchmark-on-rtx-5090-tier-comparison-64k-long-context-n", "canonical_source": "https://gist.github.com/PierpaoloPernici/2e6f6f42965d531b364f39ab0e8a52ad", "published_at": "2026-08-14 22:04:52+00:00", "updated_at": "2026-08-14 22:28:24.468007+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Qwen3.8-27B", "NVFP4", "MTP", "RTX 5090", "mtp-bench.py"], "alternates": {"html": "https://wpnews.pro/news/qwen3-8-27b-nvfp4-mtp-benchmark-on-rtx-5090-tier-comparison-64k-long-context-n", "markdown": "https://wpnews.pro/news/qwen3-8-27b-nvfp4-mtp-benchmark-on-rtx-5090-tier-comparison-64k-long-context-n.md", "text": "https://wpnews.pro/news/qwen3-8-27b-nvfp4-mtp-benchmark-on-rtx-5090-tier-comparison-64k-long-context-n.txt", "jsonld": "https://wpnews.pro/news/qwen3-8-27b-nvfp4-mtp-benchmark-on-rtx-5090-tier-comparison-64k-long-context-n.jsonld"}}