{"slug": "carbon-is-that-slow", "title": "Carbon is that slow", "summary": "Gemini 4 Carbon took roughly 3x as long as opponents to complete tasks in leaderboard battle data, averaging 6m 08s versus 2m 04s, according to the analysis. Carbon produced 16.4k output tokens versus 6.8k for opponents and made 21 tool calls versus 11, while inference speed (52 tps vs 73 tps), latency (12s vs 8s) and agent turns (13 vs 8) did not appear to drive the slowdown. Carbon won 83 of 135 battles when voters waited for both solutions to finish but lost 40 of 42 early votes, indicating its results are strong only when users are willing to wait.", "body_md": "# Carbon is that slow\n\nIn reviewing our leaderboard [1], I noticed an outlier: Gemini 4 Carbon was taking more than twice as long to complete its tasks than any other model. This looked weird, so I pulled up our battle data where Carbon was a contestant.\n\nI found that Carbon really is that slow. In fact, it took about 3x as long to complete tasks compared to opponents (6m 08s vs 2m 04s).\n\nI wanted to look into what might be the cause. The biggest outliers were:\n\n- Output tokens 16.4k for Carbon vs 6.8k for opponents.\n- Tool calls: 21 for Carbon vs 11 for opponents.\n\nIn terms of output tokens, Carbon is unique:\n\nIn terms of completion time, Carbon has only one comparable model, Qwen 3.8 Max, not exactly a great neighborhood to be in.\n\nIn terms of tool calls, there’s an outlier even worse than Gemini 4: GPT 6 Astra. We don’t have enough data from the infamous 6.2 Astra, but this points to one thing that may be wrong with it.\n\nFeatures I looked into that don’t seem to be driving poor Carbon performance are inference speed (52 tps vs opponent’s 73 tps), latency (12s vs opponent’s 8s), and agent turns (13 vs opponent’s 8).\n\nInterestingly enough, Carbon has a high win rate when the voter waits for both solutions to finish, but a *terrible* win rate from early voting.\n\n- Both finished: 83 carbon wins, 52 losses.\n- Early votes: 2 carbon wins, 40 losses!\n\nHere’s early voting against all votes.\n\n(We allow early voting contrary to Arena.ai because I think it makes sense for agents, where it doesn’t for chat. This means our data takes speed into account.)\n\nBasically, Carbon is good when you’re willing to wait.", "url": "https://wpnews.pro/news/carbon-is-that-slow", "canonical_source": "http://micjm.com/blog/carbon-is-that-slow/", "published_at": "2026-11-18 06:00:00+00:00", "updated_at": "2026-11-18 18:12:28.824621+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "large-language-models"], "entities": ["Gemini 4 Carbon", "Qwen 3.8 Max", "GPT 6 Astra", "Arena.ai"], "also_reported_by": [], "takeaway": "Gemini 4 Carbon averaged 6m 08s per task versus 2m 04s for opponents, driven by 16.4k output tokens versus 6.8k and 21 tool calls versus 11, not by inference speed (52 tps vs 73 tps) or latency (12s vs 8s).", "recommendation": {"advice": "When benchmarking agent models, log output tokens and tool calls alongside wall-clock time and separate early votes from completed comparisons, since Carbon won 83 of 135 finished battles but lost 40 of 42 early votes.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/carbon-is-that-slow", "markdown": "https://wpnews.pro/news/carbon-is-that-slow.md", "text": "https://wpnews.pro/news/carbon-is-that-slow.txt", "jsonld": "https://wpnews.pro/news/carbon-is-that-slow.jsonld"}}