{"slug": "i-ran-19-tasks-through-kimi-k3-1-pro-and-claude-fable-5-2-the-gap-wasn-t", "title": "I ran 19 tasks through Kimi K3.1 Pro and Claude Fable 5.2. The gap wasn't capability.", "summary": "A developer benchmarked Kimi K3.1 Pro and Claude Fable 5.2 across 19 script-graded tasks run through the same OpenAI-compatible endpoint, finding the two models one task apart (17/19 vs 18/19) and both misses attributable to slips rather than capability limits. The runs showed 84% of Kimi's output tokens were thinking tokens, making it roughly $63.3 per million visible tokens versus $40.0 for Fable despite Fable's higher list price, and surfaced portability pitfalls including non-standard temperature handling, 200 responses with empty bodies, and divergent structured-output and tool-calling behavior.", "body_md": "*AI disclosure: parts of this post were drafted and edited with AI assistance. The tests, numbers and charts come from my own runs on 12 November 2026.*\n\nShort version: for a user-facing chat product I'd pick Claude Fable 5.2. For batch work on a budget I'd pick Kimi K3.1 Pro. On capability the two are one task apart out of nineteen, and both misses were slips rather than limits.\n\nI ran all 19 tasks on the same day, through the same OpenAI-compatible endpoint, on the same machine. Only the model string changed. Code tasks got up to three repair rounds; everything else was one shot. Every answer was script-graded, so the model's explanation never earned points.\n\n```\nbase_url = \"https://api.crazyrouter.com/v1\"\n```\n\nSo what actually separated them?\n\n| Dimension | Tasks | Kimi K3.1 Pro | Claude Fable 5.2 | \n|---|---|---|---|\n| Code repair | 3 | 3/3 | 3/3 | \n| From scratch | 3 | 3/3 | 3/3 | \n| Tool use | 3 | 2/3 | 2/3 | \n| Repo QA | 3 | 3/3 | 3/3 | \n| Instruction following | 3 | 2/3 | 3/3 | \n| Resistance to misleading | 2 | 2/2 | 2/2 | \n| Long context | 2 | 2/2 | 2/2 | \n| Total | 19 | 17/19 | 18/19 | \n\nAll six code tasks passed in the first round for both models. Repo QA and long context were clean. On the fake-model question, neither one invented a context window or a price.\n\n**The SQL task.** \"Total order amount last month.\" Kimi called `search_docs` and searched documentation. Fable called `run_sql`, but the query hit `information_schema` instead of the orders table. Both failed, differently.\n\n**The writing task.** Kimi overshot the 140–160 character range this time. The same task passed in an earlier dry run at 155 characters, so I treat it as output instability, not a capability gap.\n\nNeither failure produces an error in production. Both produce rework.\n\nWhere does the money actually go?\n\n| Model | Output tokens | Of which thinking | Visible text | \n|---|---|---|---|\n| Kimi K3.1 Pro | 10,896 | 9,175 | 1,721 | \n| Claude Fable 5.2 | 6,386 | 0 | 6,386 | \n\n84% of Kimi's output is thinking. That is where its cost lives. Fable has no thinking tokens, so everything you pay for is readable.\n\nRecompute per visible token and the ranking flips: roughly $63.3 per million visible tokens for Kimi, $40.0 for Fable. The model with the 4x list price is cheaper per readable token.\n\nThe log task hid a key at line 287; the module task put `LAST_RETRY = 43` near the end. Both models answered both, with a handful of output tokens.\n\nBut the inputs were about 14K tokens. Kimi publishes a 1,048,576-token window; Anthropic's product page for Fable 5.2 does not publish a number. So this proves only that at 14K tokens neither model loses the needle. It says nothing about million-token capacity. To test that, build an input near the limit — I did not.\n\n**Temperature is not portable.** Kimi returns 400 for `temperature: 0.7`, and the error does not name the field. gpt-6.1-terra requires exactly 1. Claude accepts 2 and silently ignores it. Gemini returns 429 for 5, which looks like a rate limit. DeepSeek returns a clear 400. Six models, five behaviors.\n\n**A 200 can carry an empty body.** With a 1,200-token cap, Kimi returned 200, `finish_reason: length`, zero characters, and 1,197 thinking tokens. Validate content length and finish reason, not the status code.\n\n**Structured output has dialects.** Claude routes reject `json_object` with a 400 and point to `json_schema`; that works. The others accept `json_object`.\n\n**Tool calling is not universal.** gpt-6.1-terra returned \"service temporarily unavailable\" for tool calls three times. Gemini needs `tool_choice: required`; with `auto`, four of five runs came back empty with `content_filter`.\n\nSo what would I pick?\n\n| Workload | Pick | Why | \n|---|---|---|\n| Customer-facing chat | Fable 5.2 | 130.1s vs 401.6s wall clock across the suite | \n| Batch / async | Kimi K3.1 Pro | Batch inference bills at 60% of list | \n| Output parsed by code | Either, plus a validator | JSON and tables passed for both; Kimi missed a character range once | \n| Long-context retrieval | Either at 14K tokens | Ceiling untested; measure with your own material | \n| Tool orchestration | Either, with a schema check | Both missed the same routing question | \n\nKimi: 27,131 input and 10,896 output, $0.163. Fable: 36,074 input and 6,386 output, $0.544. That is 3.3x, not 4x, because the token shapes pull in opposite directions.\n\nThree workloads at list price: a 100k-round support bot around $460 vs $1,840; a thousand 8k/1.5k documents around $31 vs $124; a coding assistant at 2M in / 400k out per developer around $8 vs $32.\n\nCurrent availability is on the [model catalog](https://crazyrouter.com/en/models), which is what I checked before the run. If your input/output ratio differs from mine, the [pricing page](https://crazyrouter.com/en/pricing?utm_source=devto&utm_medium=article&utm_campaign=offsite_article&utm_content=cr_1e98c0ee00ea_mid_text) is where to redo the arithmetic.\n\n19 tasks, two models, one day. Enough for a direction, not a verdict. Not tested: longer contexts, multi-turn tool use, or pricing beyond the $8/$40 list and the $0.20 cache read.\n\nThree steps for your own numbers: ask your most common question ten times and record the median and the worst case; have the model write one real draft under your platform's rules and count your edits; run two `max_tokens` settings three times each and count the `length` finishes.\n\nTwenty minutes. Your numbers will beat this post.", "url": "https://wpnews.pro/news/i-ran-19-tasks-through-kimi-k3-1-pro-and-claude-fable-5-2-the-gap-wasn-t", "canonical_source": "https://dev.to/buluo/i-ran-19-tasks-through-kimi-k31-pro-and-claude-fable-52-the-gap-wasnt-capability-818", "published_at": "2026-11-12 10:42:39+00:00", "updated_at": "2026-11-12 10:51:43.294086+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-products", "developer-tools"], "entities": ["Kimi K3.1 Pro", "Claude Fable 5.2", "Anthropic", "Moonshot AI", "Gemini", "DeepSeek", "gpt-6.1-terra", "crazyrouter"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-ran-19-tasks-through-kimi-k3-1-pro-and-claude-fable-5-2-the-gap-wasn-t", "markdown": "https://wpnews.pro/news/i-ran-19-tasks-through-kimi-k3-1-pro-and-claude-fable-5-2-the-gap-wasn-t.md", "text": "https://wpnews.pro/news/i-ran-19-tasks-through-kimi-k3-1-pro-and-claude-fable-5-2-the-gap-wasn-t.txt", "jsonld": "https://wpnews.pro/news/i-ran-19-tasks-through-kimi-k3-1-pro-and-claude-fable-5-2-the-gap-wasn-t.jsonld"}}