Sarvam AI Launches Token Factory, Bringing Open-Model Inference to Indian GPUs Sarvam AI launched Token Factory, a general-availability inference service for open-weight models running on GPU clusters in India and billed in rupees through an OpenAI-compatible API, with a catalogue of eight models including Sarvam 105B, GLM 5.3, DeepSeek V4.1 Flash 1017, Kimi K3.1 Pro and Gemma 4 31B. Sarvam reports Sarvam 105B at 330 tokens per second with 0.2-second first-token latency, Gemma 4 31B at 308 tokens per second versus 193 for the only other provider, and a 95.7% prompt cache hit rate across more than 450 billion tokens processed daily on its DeepSeek V4.1 Flash deployment, alongside a 99.95% uptime SLA and optional zero data retention. The company said it modified vLLM kernels, cache handling and routing rather than replacing its serving stack, citing a vLLM fused decode kernel on GLM 5.3 Flash that skipped a clamping step on mixture-of-experts activations and produced deviations from the reference implementation of up to 165%. Sarvam AI Launches Token Factory, Bringing Open-Model Inference to Indian GPUs Sarvam AI opens its India-hosted inference stack to everyone, undercutting global providers on price while matching them on speed across frontier open-weight models. - Sarvam launches Token Factory https://www.sarvam.ai/blogs/sarvam-token-factory?utm source=alphasignal , an India-hosted inference service for frontier open-weight models. - Sarvam 105B hits 330 tokens/sec with 200ms first-token latency at ₹10/₹25 per million tokens. - Gemma 4 31B serves first token 5x faster than the only other provider, at 308 tok/sec. - Three-tier cache pushes DeepSeek V4.1 Flash to 95.7% prompt cache hit rate across 450B daily tokens. - Custom kernels and MXFP8 quantization on vLLM double GLM 5.3 uncached prefill throughput per GPU. - OpenAI-compatible API, 99.95% uptime SLA, optional zero data retention, and on-prem deployment available. Sarvam AI has launched Token Factory https://www.sarvam.ai/blogs/sarvam-token-factory?utm source=alphasignal , a general-availability inference service for open-weight models. The service runs on GPU clusters in India, charges in rupees, and exposes every model through an OpenAI-compatible API. Open-weight models give developers access to model parameters, while a hosted inference provider supplies the GPUs and serving software needed to run them. Sarvam’s offer combines that flexibility with domestic data processing, aggressive token pricing, and on-premises deployment for regulated institutions. One API, eight models The launch model catalogue https://www.sarvam.ai/token-factory/models?utm source=alphasignal includes Sarvam 105B, GLM 5.3, GLM 5.3 Flash, DeepSeek V4.1 Flash 1017, DeepSeek V4 Pro, Kimi K3, Kimi K3.1 Pro, and Gemma 4 31B. Sarvam’s published benchmarks report latency, throughput, and pricing across several of those models. The figures are vendor-reported and should be reproduced under each application’s prompt lengths, concurrency, output lengths, and quality requirements. | Vendor-reported launch metrics | | |---|---| | Model | Reported result | |---|---| | Sarvam 105B | First token in 0.2 seconds and single-user throughput of 330 tokens per second. DeepSeek V4.1 Flash launches with introductory pricing of ₹14 per million input tokens and ₹55 per million output tokens. | | Gemma 4 31B | First token in 0.40 seconds for a 10,000-token prompt, which Sarvam says is five times faster than the only other provider serving the model. Reported streaming throughput is 308 tokens per second, compared with 193. | | DeepSeek V4.1 Flash 1017 | An Interactive endpoint targets coding agents at about 200 tokens per second per user. A Standard endpoint costs roughly $0.092 per million tokens on a blended input-output basis. | | Kimi K3.1 Pro | Up to 1.9 times more tokens per GPU at the same per-user speed as InferenceX’s published coding benchmark. | Sarvam says its DeepSeek V4.1 Flash deployment processes more than 450 billion tokens daily and serves 95.7% of prompt tokens from cache. The company also claims to operate India’s largest Nvidia B200 cluster, with additional capacity planned for more models. Kernel bugs with production consequences Token Factory builds on vLLM, SGLang, Dynamo, and Moonshot’s Mooncake cache. Sarvam modified kernels, cache handling, routing, and cross-machine state transfer rather than replacing those serving systems wholesale. Model inference has two main stages. Prefill processes the input prompt and creates the model’s internal state; decode then produces the response one token at a time. Prefill consumes substantial compute for long prompts, while decode is often constrained by memory bandwidth. On GLM 5.3 Flash, vLLM’s fused decode kernel skipped a clamping operation on expert activations, the intermediate values produced by the model’s mixture-of-experts layers. Deviations from the reference implementation reached 165%, sometimes replacing code indentation with stray characters. Sarvam fused clamping, activation, multiplication, and quantisation into one corrected kernel, which ran about 40% faster than the previous path. A separate race condition in the indexer cache corrupted output and reduced the effectiveness of speculative decoding, which uses a smaller draft model to propose tokens for the main model to verify. After both fixes, the share of accepted draft tokens rose from 38% to 84%. Sarvam also moved GLM 5.3 from FP8 weights on SGLang to MXFP8 weights on vLLM. MXFP8 stores values in eight bits and shares a scale across each block of 32 values, matching hardware support in B200 tensor cores. Uncached prefill throughput per GPU increased from 3,700 to 7,600 tokens per second, decode throughput rose 25%, and 95th-percentile time to first token on production chat traffic fell by half. A cache built for 131,000-token prompts Agent traffic places different demands on inference infrastructure because prompts grow across tool calls and repeated turns. Sarvam reports an average GLM 5.3 agent prompt of 131,000 tokens, with agents often returning seconds or minutes after a tool call carrying nearly the same context. GPU memory retained that context for about four seconds under normal conditions and roughly two seconds under load. Most returning turns therefore required the servers to process the full prompt again. Sarvam added a shared G3 cache, built on Mooncake, behind every prefill server. The cache retains prompts for hours or days and works with cache-aware routing to send requests toward reusable context. On DeepSeek, the combined hit rate rose from 91.4% to 95.7%, halving the number of tokens requiring prefill. During peak GLM 5.3 traffic, the number of required prefill nodes fell from 58 to 34. DeepSeek V4.1 Flash 1017 exposed another cache problem because its sliding-window attention retains only a moving portion of the context. Cache reads and writes used different positions, causing 74% of prompt lengths to miss reusable state and trigger reprocessing. Aligning those positions raised the cold-cache hit rate from 37.1% to 99.8% and reduced latency from 2.80 seconds to 0.64 seconds. State has to survive the GPU handoff Disaggregated serving assigns prefill and decode to separate GPU pools so each stage can use hardware configured for its workload. Moving a request between those pools also requires every server to agree on cache positions and model state. Kimi K3.1 Pro’s prefill GPU, router, and decode GPU each used different checkpoint positions for its linear-attention layers. Aligning them increased multi-turn cache hits from 39% to 73%. GLM 5.3 Flash showed the same class of failure in its draft model. Accepted tokens per speculative-decoding step fell from 4.92 to 1.05 when the split servers used mismatched cache layouts. Matching the layouts restored the figure to 4.69. The buying case, with caveats Sarvam compares its performance with named providers including Together, Fireworks, Baseten, DeepInfra, and InferenceX, and publishes methodology for the cited tests. Independent workload testing remains necessary because prompt shape, concurrency, caching, and output length can materially change inference results. Data residency is the clearest distinction for Indian enterprises and public-sector organisations. Requests run on Sarvam-operated GPUs in India instead of being relayed to an overseas model developer. Zero data retention is available on request, and the managed service carries a 99.95% uptime service-level agreement. Regulated customers can deploy the same stack on premises. The OpenAI-compatible interface lowers integration and migration work across the catalogue. Model behaviour, tokenisation, tool calling, context limits, and rate limits can still vary, so applications need model-specific tests before switching production traffic. Long-context agents make Sarvam’s serving work economically relevant because every avoided prefill reduces latency and GPU demand. Its examples show roughly twofold to threefold efficiency gains in selected parts of the stack. The savings available to a production application will depend on how closely its traffic resembles Sarvam’s benchmark and cache patterns.