# Together Compiler > Source: > Published: 2026-11-06 13:34:27+00:00 Submitted 04 Nov 2026 # Together Compiler: Continual Compilation for AI Inference [#4 on Trending](https://www.alphaxiv.org/?sort=Hot)alphaXiv Exclusive ## Abstract Today’s AI workloads are rapidly changing: new models are released more often, model architectures are increasingly diverse, and traffic distributions change frequently. The challenge we face at Together AI is continual optimization for these changing workloads. Compiling a given deployment into an optimized forward pass is a challenge that is well-suited for agent hill-climbing. Existing systems struggle with continual compilation due to a tradeoff between measurement throughput and fidelity. Many systems use agents to optimize individual kernels from the inference forward pass using offline benchmarks, but timings are low-fidelity because of data-dependent behavior in primitives like mixture-of-experts (MoE). On the other hand, systems use agents to directly optimize end-to-end engines but suffer from slow engine launch times. To address these challenges, we build Together Compiler (TKCC), the first truly continual agent loop designed for AI inference optimization. To address the throughput-fidelity tradeoff, we propose: (1) a Tiered Persistent Engine that decouples engine launch overheads from forward pass computation; (2) a cost model to select the parallelism configurations that seed the optimization process. We also conduct extensive ablations on agent context management and present our insights on deploying TKCC in production. TKCC outperforms highly-optimized hand-written Megakernels on the Llama 1B workhorse workload by up to 1.7× and the best open source engines on the latest frontier models (Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash) across hardware platforms by up to 2.6× higher tokens per second (TPS) on SemiAnalysis’ InferenceX benchmark. TKCC also sets the state-of-the-art performance on low-latency, batch size 1 inference, generating 1111 TPS on MiniMax M3.1 Flash and 1205 TPS on DeepSeek v4.1 Flash. Finally, TKCC generalizes and provides speedup beyond frontier language model inference. TKCC serves trillions of tokens across our production fleet. ## AI Overview Inference deployments go stale. One lab's model saw its typical context grow from 10,000 to 18,000 tokens over two weeks, while its cache-hit rate rose by eight percentage points. That example illustrates why a setup tuned once can drift away from the best choice as requests change; it is not a benchmark condition for the results below. Together Compiler (TKCC) is built around that problem. Agents propose replacement forward-pass code, which is tested inside an engine that is already loaded. Quick screening weeds out poor candidates, and promising ones get a slower, more representative check. A cost model picks a starting arrangement of work across GPUs. In one batch-size-one MiniMax example, the process is associated with higher per-user interactivity than a published configuration, though the GPU-parallelism configuration changed as well, so that comparison is not code-only. ## Restarting an inference engine for every candidate is slow. TKCC instead starts a persistent engine once and loads the model weights once. When an agent submits a candidate, the server replaces the model's forward-function pointer on hot reload and recaptures CUDA/HIP graphs; teardown restores the baseline. Several agents work in parallel and communicate, so GPU time is less likely to sit idle while one agent is thinking. Measurement comes in two tiers. The fast tier serves fewer layers and rejects slow, invalid, or hanging candidates cheaply. The slow tier serves more layers and gives a more representative signal for data-dependent behavior. On B200 GLM 5.3 optimization, the authors compared this two-tier setup against a single slow tier: it evaluated 1,354 candidates rather than 168, about 8.1 times as many, and found a solution 7.2% faster. This tests the measurement design, not any particular candidate-generation strategy. The authors also compared against Apple's RSI-Inference Bench, an end-to-end agent-optimization harness whose higher-fidelity testing costs experiment throughput because the engine must be launched for each measurement. Their comparison covered the two-tier setup, a single slow tier, no tiers, and Apple's harness. The Apple harness had a 10-hour budget and an eight-hour verifier timeout, and used the same GLM 5.3 serving image, forward-pass edit surface without quantization, and AgentX workload as TKCC, measuring output tokens per second. The slow tier matters because of mixture-of-experts (MoE) models, which route each token to only some experts rather than applying the full weights to every token. Input distribution can therefore change what work is done. The authors observed that randomly initialized data activated 13–38% more experts than real data, and that this caused a 1.3× end-to-end slowdown in the engine. This is the paper's motivation for not trusting isolated, random-input kernel measurements. Systems such as GEAK tune kernels with agents, but the paper does not run a head-to-head comparison against them. ## Parallelism here means how a model's work and weights are divided among GPUs. TKCC's cost model narrows the candidate configurations using analytical estimates and measured compute, memory, and communication performance, and it accounts for production workload characteristics such as expert imbalance and uneven request lengths. The aim is to avoid both an exhaustive search and reliance on hardware specifications alone. The headline MiniMax result uses "P90 TPS per user," the reciprocal of 90th-percentile inter-token latency. This is a user-interactivity measure, not aggregate tokens per second per GPU. In the MiniMax M3.1 Flash B200 batch-size-one example, TKCC records 1,111 against 633 for the best published InferenceX result, which the paper describes as 1.8 times higher. The qualification is essential: the published image used TP4, while TKCC's cost model chose TP8. The authors say that running the same InferenceX image with TP8 achieved performance similar to its chosen TP4 result, so the 1.8× comparison does not isolate the effect of agent-written code. The InferenceX experiments optimized for 24 hours per setting. Not every comparison necessarily uses an identical parallelism configuration. ## The pieces fit the opening problem. A persistent engine lowers the cost of testing candidates, real-workload measurement keeps the signal faithful to production behavior, and the cost model keeps the search aimed at promising layouts as request patterns shift. The authors frame TKCC as a continual production system, arguing that static inference is over and that agent-driven optimization must be continual to be useful. That is their claim, not an independent verification of universal optimality. The evidence here is a faster search on one GLM 5.3 B200 setup and a measured but confounded MiniMax case, not a guarantee for every model or traffic shape.