{"slug": "giraffebench-1-0-the-quality-of-local-and-cloud-llms-in-a-way-you-can-see", "title": "Giraffebench 1.0 – The quality of local and cloud LLMs in a way you can see", "summary": "Giraffebench 1.0 published benchmark runs of Qwen4 Max, a 2400B mixture-of-experts model released November 16, 2026 and served closed-weight through OpenRouter by Alibaba, showing the model completing a coding task in 25 minutes at 26.4 tokens per second for $0.371 under the pi 1.5.0 harness and in 20 minutes at 24.7 tokens per second for $0.549 under OpenCode 2.2.5. The pi run used 39 requests and 1,411,522 tokens (75,942 new input, 1,295,616 cached, 39,964 output) with 19,240 reasoning tokens billed, while the OpenCode run used 49 requests and 2,358,792 tokens (100,580 new input, 2,228,736 cached, 29,476 output) with 28,161 reasoning tokens billed. Both runs were billed at OpenRouter list prices of $1.50 per million input tokens, $0.075 per million cached tokens and $4.50 per million output tokens, with no thinking setting sent so the model default applied.", "body_md": "[Run the giraffe](g/qwen--qwen4-max-pi/index.html)\n\n### Qwen4 Max\n\nQwen · OpenRouter · 2400 B MoE\n\nReleased November 16, 2026\n\n**25 min**\n\n**26.4 tok/s**\n\n**$0.371**\n\n## Details\n\n- Model on OpenRouter\n- [qwen/qwen4-max](https://openrouter.ai/qwen/qwen4-max)\n- Weights\n- Closed model, weights not published\n- Released\n- November 16, 2026 (date the model was added to OpenRouter)\n- Hardware\n- Cloud via OpenRouter, served by Alibaba\n- Harness\n- pi 1.5.0\n- Thinking\n- Model default (pi and OpenCode sent no thinking setting to OpenRouter, so the model default applied)\n19,240 reasoning tokens billed\n- Coding time\n- 25 min, 39 requests\n- Tokens\n- 1,411,522 (75,942 new input, 1,295,616 cached, 39,964 output)\n- OpenRouter cost (real, via API)\n- $0.371 (list price per million tokens: $1.50 input, $0.075 cached, $4.50 output)\n\n[Run the giraffe](g/qwen--qwen4-max-oc/index.html)\n\n### Qwen4 Max\n\nQwen · OpenRouter · 2400 B MoE\n\nReleased November 16, 2026\n\n**20 min**\n\n**24.7 tok/s**\n\n**$0.549**\n\n## Details\n\n- Model on OpenRouter\n- [qwen/qwen4-max](https://openrouter.ai/qwen/qwen4-max)\n- Weights\n- Closed model, weights not published\n- Released\n- November 16, 2026 (date the model was added to OpenRouter)\n- Hardware\n- Cloud via OpenRouter, served by Alibaba\n- Harness\n- OpenCode 2.2.5\n- Thinking\n- Model default (pi and OpenCode sent no thinking setting to OpenRouter, so the model default applied)\n28,161 reasoning tokens billed\n- Coding time\n- 20 min, 49 requests\n- Tokens\n- 2,358,792 (100,580 new input, 2,228,736 cached, 29,476 output)\n- OpenRouter cost (real, via API)\n- $0.549 (list price per million tokens: $1.50 input, $0.075 cached, $4.50 output)", "url": "https://wpnews.pro/news/giraffebench-1-0-the-quality-of-local-and-cloud-llms-in-a-way-you-can-see", "canonical_source": "https://keno303.github.io/giraffebench/index.html", "published_at": "2026-11-20 04:09:50+00:00", "updated_at": "2026-11-20 04:45:08.985535+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools"], "entities": ["Giraffebench", "Qwen4 Max", "Qwen", "OpenRouter", "Alibaba", "pi 1.5.0", "OpenCode 2.2.5"], "also_reported_by": [], "takeaway": "Qwen4 Max cost $0.371 for 39 requests under pi 1.5.0 versus $0.549 for 49 requests under OpenCode 2.2.5, with cached input billed at $0.075 per million tokens against $1.50 for new input.", "recommendation": {"advice": "Route heavy agent runs through harnesses that maximize cached input: Qwen4 Max billed cached tokens at $0.075 versus $1.50 per million new input, and the pi run's 1,295,616 cached tokens held total cost to $0.371.", "playbook": "llm-cost", "playbook_url": "https://wpnews.pro/playbooks/llm-cost"}, "alternates": {"html": "https://wpnews.pro/news/giraffebench-1-0-the-quality-of-local-and-cloud-llms-in-a-way-you-can-see", "markdown": "https://wpnews.pro/news/giraffebench-1-0-the-quality-of-local-and-cloud-llms-in-a-way-you-can-see.md", "text": "https://wpnews.pro/news/giraffebench-1-0-the-quality-of-local-and-cloud-llms-in-a-way-you-can-see.txt", "jsonld": "https://wpnews.pro/news/giraffebench-1-0-the-quality-of-local-and-cloud-llms-in-a-way-you-can-see.jsonld"}}