| | | | | +----------------------------------+--------------------------------------------------------------------------------+-----------------+-----------------+-----------------+ | | | \| Model / route \| Experience / outcome \| Total tokens \| Output tokens \| Cost \| | | | +----------------------------------+--------------------------------------------------------------------------------+-----------------+-----------------+-----------------+ | | | \| Xiaomi MiMo v2.6 Flash \| Incredibly fast and intelligent; solved. \| 225,291 \| 3,281 \| $0.005 (0.5c) \| | | | \| GLM 5.3 Flash / OpenRouter \| Least intelligent group but cheapest; solved. \| 340,006 \| 11,205 \| $0.01 \| | | | \| ChatGPT Luna Latest \| Very fast, concise, token-efficient, intelligent, and cheap; solved. \| 401,837 \| 5,119 \| $0.01 \| | | | \| Qwen 3.8 Flash \| Initially confused and failed, then eventually solved. \| ~418,000 \| 16,676 \| $0.02 \| | | | \| Qwen 3.8 27B \| Fast; one wrong answer, then quickly self-corrected. \| ~774,784 \| 12,658 \| $0.02 \| | | | \| DeepSeek Flash / direct API \| Fast, cheap, and accurate; solved. \| 369,334 \| 9,089 \| $0.03 \| | | | \| DeepSeek Flash / OpenRouter \| Fast and intelligent; solved in 76 seconds. \| 371,689 \| 13,888 \| $0.03 \| | | | \| Claude Haiku 5.5 \| Fast and surprisingly intelligent, but overly thorough and token-heavy. \| 662,346 \| 26,244 \| $0.03 \| | | | \| Thinking Machines Inkling \| Low intelligence; good speed and relatively token-efficient; may loop. \| N/A \| N/A \| $0.03 \| | | | \| Xiaomi MiMo v2.6 Pro \| Overthought the task and wandered into tangents. \| ~420,644 \| 13,638 \| $0.05 \| | | | \| GLM 5.3 / OpenRouter \| Cheap and intelligent but behind DeepSeek; solved in about 2m 20s. \| ~702,473* \| 11,879 \| $0.08 \| | | | \| GPT Sol Latest (Sol 6.1) \| Slow, but highly intelligent and extremely token-efficient; solved in ~89s. \| 294,957 \| 1,406 \| $0.15 \| | | | \| MiniMax M3 \| Least intelligent comparison; many wrong turns and repeated small mistakes. \| 4,546,552* \| 116,804 \| $0.20 \| | | | \| GPT Terra Latest \| Very fast, very smart, and token-efficient; solved. \| 229,888 \| 2,342 \| $0.20 \| | | | \| Mistral Large 4.0 \| Medium speed; small mistakes but reached the solution. \| 303,687 \| 4,041 \| $0.21 \| | | | \| Kimi K2.7 Code \| Less intelligent, slow, expensive; not worth using. \| 100,740 \| 8,938 \| $0.22 \| | | | \| Claude Opus 5.5 \| Highest intelligence and token efficiency; pretty fast; cheaper than Sonnet. \| 312,837 \| 1,862 \| $0.24 \| | | | \| Qwen 3.8 Max 0902 \| Fast and intelligent, better than Kimi K3, but token-hungry. \| 1,305,241 \| 26,390 \| $0.25 \| | | | \| Kimi K3 / OpenRouter \| Faster and cheaper than direct K3, but still very expensive. \| 750,064 \| 5,922 \| ~$0.32 \| | | | \| Meta Muse Spark Open \| Intelligent and medium speed, but among the more expensive runs. \| 872,850 \| 10,284 \| $0.35 \| | | | \| Gemini 4 Argon \| Extremely fast and intelligent; strong value for capability. \| 384,046 \| 2,870 \| $0.35 \| | | | \| Grok 4.7 \| Very fast and intelligent, but expensive. \| 655,648 \| 4,217 \| $0.46 \| | | | \| Meta Muse Glimmer 30B \| Fast but low intelligence and token-inefficient; cancelled unsolved. \| N/A \| N/A \| ~$0.50 wasted \| | | | \| Claude Sonnet 5.5 \| Smart and fast but very token-inefficient and expensive. \| 1,243,285* \| 20,381 \| $0.57 \| | | | \| Kimi K3 / direct API \| Intelligent but not notable; extremely slow and far too expensive. \| 205,710 \| 3,215 \| $0.72 \| | | | \| ChatGPT Astra Latest \| Medium speed, token-efficient, highly intelligent; cost similar to Fable. \| 278,603 \| 1,536 \| $0.83 \| | | | \| Claude Fable 5.1 \| OK speed, very expensive, medium token efficiency; solved in ~53s. \| 397,806* \| 2,857 \| $0.85 \| | | | \| Gemini Flash \| Refused the reverse-engineering task. \| N/A \| N/A \| Not reported \| | | | \| Thinking Machines Inkling Small \| Got stuck immediately and was terminated. \| N/A \| N/A \| Not reported \| | | | +----------------------------------+--------------------------------------------------------------------------------+-----------------+-----------------+-----------------+ | | | |
How LLM Evaluation Produces Comparable Numbers: Inside the LFORLA RL Game Agents Benchmark