Giraffebench 1.0 – The quality of local and cloud LLMs in a way you can see Giraffebench 1.0 published benchmark runs of Qwen4 Max, a 2400B mixture-of-experts model released November 16, 2026 and served closed-weight through OpenRouter by Alibaba, showing the model completing a coding task in 25 minutes at 26.4 tokens per second for $0.371 under the pi 1.5.0 harness and in 20 minutes at 24.7 tokens per second for $0.549 under OpenCode 2.2.5. The pi run used 39 requests and 1,411,522 tokens (75,942 new input, 1,295,616 cached, 39,964 output) with 19,240 reasoning tokens billed, while the OpenCode run used 49 requests and 2,358,792 tokens (100,580 new input, 2,228,736 cached, 29,476 output) with 28,161 reasoning tokens billed. Both runs were billed at OpenRouter list prices of $1.50 per million input tokens, $0.075 per million cached tokens and $4.50 per million output tokens, with no thinking setting sent so the model default applied. Run the giraffe g/qwen--qwen4-max-pi/index.html Qwen4 Max Qwen · OpenRouter · 2400 B MoE Released November 16, 2026 25 min 26.4 tok/s $0.371 Details - Model on OpenRouter - qwen/qwen4-max https://openrouter.ai/qwen/qwen4-max - Weights - Closed model, weights not published - Released - November 16, 2026 date the model was added to OpenRouter - Hardware - Cloud via OpenRouter, served by Alibaba - Harness - pi 1.5.0 - Thinking - Model default pi and OpenCode sent no thinking setting to OpenRouter, so the model default applied 19,240 reasoning tokens billed - Coding time - 25 min, 39 requests - Tokens - 1,411,522 75,942 new input, 1,295,616 cached, 39,964 output - OpenRouter cost real, via API - $0.371 list price per million tokens: $1.50 input, $0.075 cached, $4.50 output Run the giraffe g/qwen--qwen4-max-oc/index.html Qwen4 Max Qwen · OpenRouter · 2400 B MoE Released November 16, 2026 20 min 24.7 tok/s $0.549 Details - Model on OpenRouter - qwen/qwen4-max https://openrouter.ai/qwen/qwen4-max - Weights - Closed model, weights not published - Released - November 16, 2026 date the model was added to OpenRouter - Hardware - Cloud via OpenRouter, served by Alibaba - Harness - OpenCode 2.2.5 - Thinking - Model default pi and OpenCode sent no thinking setting to OpenRouter, so the model default applied 28,161 reasoning tokens billed - Coding time - 20 min, 49 requests - Tokens - 2,358,792 100,580 new input, 2,228,736 cached, 29,476 output - OpenRouter cost real, via API - $0.549 list price per million tokens: $1.50 input, $0.075 cached, $4.50 output