Together Compiler: AI Agents Rewrite GPU Inference Code

Together AI announced Together Compiler (TKCC) on November 4, a production system in which AI agents autonomously rewrite the low-level GPU kernel code that runs model inference, reporting 1.3x to 3x tokens-per-second gains across Kimi K3, GLM 5.3, MiniMax M3.1-Flash and DeepSeek v4.1-Flash, and up to 3.7x higher throughput in traffic-shift scenarios. Together AI says TKCC is already serving trillions of tokens daily across its top inference workloads, with a persistent engine that hot-reloads candidate GPU code and cut initialization time from 467 seconds to 9 seconds per experiment cycle, enabling 252x more optimization experiments than baseline agentic systems. Together AI kept humans on the final deploy step, requiring engineer review and approval before optimized code reaches production, and the company's figures have no independent benchmark verification yet.

Together AI shipped something that got buried under louder announcements last week: a production system where AI agents autonomously rewrite the GPU kernel code that runs AI models at scale. Not application code. Not configuration files. The low-level forward pass code that determines how fast a model serves tokens — continuously, overnight, while the engine keeps running. The system is called Together Compiler TKCC , announced November 4. It is the first product from Together AI’s Polymorphic Inference https://www.together.ai/blog research agenda — inference infrastructure that adapts itself rather than waiting for engineers to retune it. Why Static Compilation Breaks in Production The gap between launch-day performance and production reality is a known problem. Together AI cites a telling statistic: 80% of OpenRouter traffic goes to models released within the past three months. Add to that prompt length drift one deployment saw an 80% increase post-launch and regional traffic differences 97,000-token prompts in one region versus 70,000 in another , and a compiler optimized at launch is out of date almost immediately. Traditional GPU kernel tuning is expensive, slow, and requires specialized expertise. The standard approach is to tune once, deploy, and accept gradual degradation. Together Compiler replaces that with a continuous search process. How It Works The architecture is straightforward in concept, technically interesting in execution. Multiple agents work in parallel, generating kernel optimization candidates and sharing proposals through a shared message board. The key innovation is a persistent inference engine that hot-reloads candidate GPU code without restarting — cutting initialization time from 467 seconds to 9 seconds per experiment cycle. That matters because optimization is a search problem. Faster iterations mean more experiments: Together Compiler runs 252x more optimization experiments than baseline agentic systems, according to the company. Winning candidates go through a two-tier validation system — fast engines for rapid iteration, slower engines for realistic distribution testing — plus a final KV-cache contamination check before anything reaches production. The Numbers Performance results against highly optimized open-source baselines: 1.3x to 3x tokens per second improvement across tested models Kimi K3, GLM 5.3, MiniMax M3.1-Flash, DeepSeek v4.1-Flash . In traffic-shift scenarios, Together reports up to 3.7x higher throughput. SemiAnalysis InferenceX records 1,111 TPS on MiniMax M3.1-Flash and 1,205 TPS on DeepSeek v4.1-Flash. The system is already in production. Together AI says TKCC is serving trillions of tokens daily https://thenewstack.io/together-compiler-gpu-optimization/ across its top inference workloads and popular serverless endpoints. That is not a preview — it is already running. One caveat worth noting: these figures come from Together AI’s own reporting. No independent benchmark verification exists yet, and the “up to 3x” claim lacks a breakdown by model, hardware, or typical case. The production scale is real; the headline numbers should be treated as the high end of observed gains. Humans Stay on the Deploy Button The design choice that stands out most: Together deliberately kept humans in the loop for final deployment. The agents run the full optimization cycle autonomously. But the final step — pushing optimized code to production — requires a human to review and approve the changes. This is not a technical limitation. It is an explicit trust architecture decision. In a moment where AI agents are being handed increasing autonomy over infrastructure, Together’s answer is: agents find the gains, engineers decide what ships. That stance is worth taking seriously, especially as the same pattern — agents rewriting their own serving infrastructure — is about to appear across more platforms. What You Can Do Now If you are already using Together AI’s serverless endpoints, you are benefiting from this automatically — no code changes required. The compiler is active on many of the platform’s popular endpoints. For dedicated deployments, Together is running a limited beta https://www.unite.ai/together-ai-brings-continuous-compilation-to-production-inference/ — application-based, with eligibility criteria not publicly specified. A self-serve one-click optimization feature is planned for dedicated endpoints but has no announced release date. Together Compiler is not open-source. If you are running your own serving infrastructure, this is not something you can drop in today. The technology remains within Together AI’s platform for now — but the architecture it demonstrates is replicable, and other inference providers will not be far behind.