{"slug": "composio-s-kimi-k3-test-finds-a-6x-token-gap-between-agent-harnesses", "title": "Composio's Kimi K3 test finds a 6x token gap between agent harnesses", "summary": "Composio, an AI-agent infrastructure company, reported that Moonshot AI's Kimi K3 consumed nearly six times as many tokens in Claude Code (340,000 median) as in Kimi Code (61,000 median) while completing similar numbers of tasks on a 28-task benchmark. The test held the model constant and varied the harness—the system prompt, tool loop, and context management—showing that orchestration choices can produce a larger economic difference than success-rate gaps. Moonshot AI warns that Kimi K3's performance can become unstable when a harness fails to preserve its thinking history, recommending a verified compatible harness such as Kimi Code.", "body_md": "Composio, the AI-agent infrastructure company founded by [Soham Ganatra (@GanatraSoham)](https://x.com/GanatraSoham) and [Karan Vaidya (@KaranVaidya6)](https://x.com/KaranVaidya6), said Wednesday that Moonshot AI's Kimi K3 consumed nearly six times as many tokens in Claude Code as it did in Kimi Code while completing the same set of agent tasks at a similar success rate.\n\nThe [company-run test](https://x.com/composio/status/2082452269522378858?s=46) sent Kimi K3 through Claude Code, Nous Research's Hermes and Kimi Code on 28 identical tasks. Kimi Code completed 22, Hermes completed 21 and Claude Code completed 20. The narrow spread in completions contrasted with a wide gap in token consumption: the median task used 61,000 tokens in Kimi Code, 67,000 in Hermes and 340,000 in Claude Code. Composio also reported that individual tasks could consume up to 30 times as many tokens in one harness as another. ([x.com](https://x.com/composio/status/2082452269522378858?s=46))\n\nHermes recorded the shortest median completion time at 179 seconds. Kimi Code took 297 seconds, while Claude Code took 348 seconds, according to Composio. Those figures make Hermes the speed leader in this small test and Kimi Code the token-efficiency leader.\n\nThe results are consequential because developers usually compare agent models using completion rates and benchmark scores. Composio's test held the model constant and changed the software around it: the system prompt, tool loop, context management and other orchestration choices collectively described as the harness. In this run, those choices produced a larger economic difference than the success-rate gap.\n\n### Kimi warns that harness compatibility matters\n\nMoonshot's own [Kimi K3 technical blog](https://www.kimi.com/fr/blog/kimi-k3) provides an important qualification. K3 was trained to preserve its thinking history, and Moonshot warns that performance can become unstable when a harness fails to return that history correctly. Moonshot recommends a verified compatible harness such as Kimi Code and cautions against switching models during a session. ([kimi.com](https://www.kimi.com/fr-fr/blog/kimi-k3))\n\nThat makes Composio's result a test of model-harness pairing rather than a general verdict on Claude Code. A harness optimized around one model family's context and reasoning behavior can impose substantial overhead when used with another. Moonshot itself uses different harnesses across the benchmarks in K3's release materials, including Kimi Code, Claude Code and Codex.\n\nThe token gap still carries a direct cost. Moonshot prices K3 API input at $3 per million tokens on a cache miss and $0.30 per million on a cache hit, with output priced at $15 per million tokens. Actual bills depend on the input-output mix and caching, but an agent that repeatedly reloads context or takes unnecessary tool-loop steps can erase savings from choosing a cheaper model. ([kimi.com](https://www.kimi.com/resources/kimi-k3-pricing?utm_source=openai))\n\nComposio's 28-task sample is directional, and the benchmark comes from a vendor with a commercial stake in the orchestration layer. [Composio](https://composio.dev/) sells infrastructure for connecting agents to more than 1,000 tools, including authentication, tool execution and context management. Its conclusion that developers should inspect the harness before replacing the model reinforces the product category Ganatra and Vaidya are building. ([composio.dev](https://composio.dev/blog/series-a))\n\n### Composio is betting on the layer around the model\n\nGanatra's interest in that layer began before the current coding-agent market formed. He said in a [2025 interview](https://www.pmf.show/episodes/ep-73-he-quit-his-job-went-all-in-on-ai-agents-then-grew-to-100k-users-a-30m-series-a-) that he encountered the cost of maintaining software integrations while working as the founding engineer and later an engineering and product manager at fraud and risk startup Bureau. He left around April or May 2023 to build Composio after concluding that improving code-generation models could automate part of that integration work. ([pmf.show](https://www.pmf.show/episodes/ep-73-he-quit-his-job-went-all-in-on-ai-agents-then-grew-to-100k-users-a-30m-series-a-))\n\nVaidya, Composio's CTO, lists Google and Rubrik among his prior stops on his [personal site](https://kvaidya.com/). Both founders graduated from IIT Bombay's computer science program in 2017, according to the university's [alumni office](https://acr.iitbombay.org/alumni-startup-spotlight-2/). ([acr.iitbombay.org](https://acr.iitbombay.org/alumni-startup-spotlight-2/))\n\nOn July 22nd, 2025, Composio announced a $25 million Series A led by Lightspeed Venture Partners, bringing its reported total funding to $29 million. The round included Vercel CEO Guillermo Rauch, HubSpot co-founder Dharmesh Shah, SV Angel, Operator Partners, Elevation Capital and Together Fund. Composio said at the time that its platform served more than 100,000 developers and 200 startups and enterprises; those figures remain company-reported. ([prnewswire.com](https://www.prnewswire.com/news-releases/composio-raises-29m-to-solve-ais-learning-problem-building-skills-that-actually-improve-over-time-302510684.html))\n\nKimi K3 gives Composio a timely demonstration of the bet behind that funding. Frontier models increasingly arrive with specific requirements for thinking history, context compaction, tool formats and reasoning controls. The harness decides whether those requirements become efficient execution or an expensive loop. Composio's test puts a number on that implementation risk: similar task completion can conceal a sixfold median difference in token use.", "url": "https://wpnews.pro/news/composio-s-kimi-k3-test-finds-a-6x-token-gap-between-agent-harnesses", "canonical_source": "https://runtimewire.com/article/composio-kimi-k3-agent-harness-token-efficiency-test", "published_at": "2026-07-29 16:53:36+00:00", "updated_at": "2026-07-29 16:57:07.793412+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-infrastructure", "large-language-models", "developer-tools"], "entities": ["Composio", "Moonshot AI", "Kimi K3", "Claude Code", "Kimi Code", "Nous Research", "Hermes", "Soham Ganatra"], "alternates": {"html": "https://wpnews.pro/news/composio-s-kimi-k3-test-finds-a-6x-token-gap-between-agent-harnesses", "markdown": "https://wpnews.pro/news/composio-s-kimi-k3-test-finds-a-6x-token-gap-between-agent-harnesses.md", "text": "https://wpnews.pro/news/composio-s-kimi-k3-test-finds-a-6x-token-gap-between-agent-harnesses.txt", "jsonld": "https://wpnews.pro/news/composio-s-kimi-k3-test-finds-a-6x-token-gap-between-agent-harnesses.jsonld"}}