ZenteiQ scales batch inference 4x with GKE Inference Gateway powered by llm-d

ZenteiQ, an AI startup spun out of the Indian Institute of Science and selected under the IndiaAI Mission, scaled aggregate inference throughput 4.07x and raised daily output from 1.47 billion to 5.97 billion generated tokens per 1,000 accelerators by deploying Google Kubernetes Engine Inference Gateway, powered by the open-source llm-d distributed inference framework, in front of its vLLM accelerator pools. The startup had been bottlenecked by standard ClusterIP round-robin load balancing, where uneven queues and redundant prefill work swung generation throughput by as much as half within a single run and delayed scheduled pre-training of its 9B- and 35B-parameter scientific foundation models. Google Cloud product manager Fisayo Feyisetan said GKE Inference Gateway gave ZenteiQ the routing layer to route on live vLLM queue depths and prompt caches without building a custom inference scheduler.

Today we hear from ZenteiQ , an artificial intelligence startup spun out of the Indian Institute of Science IISc and selected under the IndiaAI Mission to build sovereign scientific foundation models from the ground up. To pre-train models that understand physical laws, governing equations, and numerical solvers, ZenteiQ designed and built a large-scale batch synthetic data-generation pipeline that runs across multiple model families and feeds verified datasets into training runs. Google Kubernetes Engine GKE and GKE Inference Gateway gave ZenteiQ the inference-routing layer within its pipeline to scale aggregate inference throughput 4.07x and grow daily output from 1.47 billion to 5.97 billion generated tokens for every 1,000 accelerators. - Fisayo Feyisetan, Product Manager, Google Cloud Unlike general-purpose language models trained primarily on web text and code, scientific foundation models must understand complex physical systems, mathematical governing equations, and numerical solvers across domains such as thermodynamics, computational fluid dynamics, and electromagnetics. At ZenteiQ, we are building sovereign scientific foundation models from scratch that interpret simulation outputs, evaluate equation formulations, and work through step-by-step mathematical derivations that obey physical laws. Pre-training models on large datasets of governing equations takes hundreds of billions of reasoning tokens. Alongside curated scientific literature and simulation datasets, we also rely on a batch synthetic data-generation pipeline that we independently designed and implemented to generate, evaluate, and filter candidate derivations and solver validation problems before feeding them into our MaxText https://github.com/AI-Hypercomputer/maxtext pre-training jobs. Because we operate on a fixed compute allocation, every hour our accelerators spend generating and verifying training data is an hour taken away from pre-training. When data generation falls behind, multi-host accelerator clusters reserved for pre-training sit idle, waiting on datasets. To maximize data generation speed, our pipeline separates prompt preparation on worker pods from model serving on vLLM across multi-node accelerator pools. Each batch moves through four stages: expanding scientific seed concepts into prompt batches, generating candidate reasoning traces with generator models, scoring responses using evaluator models, and running guardrail checks with a separate model family before packaging verified datasets for pre-training. We originally ran these model pools on Kubernetes with standard ClusterIP round-robin load balancing across inference endpoints. As we scaled our pipeline to feed our 9B- and 35B-parameter model schedules, two routing bottlenecks began slowing down dataset creation. Together, uneven queues and redundant prefill work caused our generation throughput to swing by as much as half within a single run, stretching out dataset creation and delaying the start of scheduled pre-training runs. We needed routing that understood live vLLM queue depths and prompt caches, without pulling our researchers off model development to build and maintain a custom inference scheduler. We already run our data preparation and MaxText pre-training jobs on GKE, which lets our team manage data generation and model training on the same Kubernetes platform. Following a recommendation from Google Cloud to address our routing bottlenecks without changing our underlying scientific methodology or pipeline design, we deployed GKE Inference Gateway https://cloud.google.com/kubernetes-engine/docs/concepts/about-gke-inference-gateway , powered by the open-source llm-d https://llm-d.ai/ distributed inference framework, in front of our vLLM accelerator pools. In our pipeline, worker pods pull scientific seed documents and domain taxonomies from Cloud Storage and expand them into batches of multi-step prompts. Those batches flow through a single GKE Inference Gateway deployment that manages traffic across all of our model families and serving nodes. Rather than maintaining isolated routing for each model tier, we route generation requests, evaluation queries, and guardrail checks across their respective model pools through the same gateway. Once candidate responses pass evaluation and guardrail checks, worker pods deduplicate and package the verified datasets for direct consumption by downstream MaxText pre-training jobs on GKE. With GKE Inference Gateway and llm-d, routing decisions use live serving signals from our vLLM instances rather than static round-robin rotation. GKE Inference Gateway continuously monitors real-time queue depth and key-value KV cache utilization across our vLLM endpoints. When a serving node is busy decoding a long equation derivation, the gateway routes incoming requests to nodes with open capacity, so heavy prompts no longer pile up behind busy workers while neighboring nodes sit idle. At the same time, the gateway routes requests with matching scientific instructions to the serving node where that prefix’s KV cache is already warm in accelerator memory. Because 70% to 80% of each prompt is shared instructional context while the generated reasoning trajectories vary from request to request, combining prefix-cache locality with live queue balancing lets our pods skip repeated prefill work without overloading any single replica. Adopting GKE Inference Gateway powered by llm-d transformed the speed and predictability of our data generation pipeline. Across our production clusters, intelligent routing delivered: For our research and infrastructure teams, the most important outcome was schedule predictability. Quadrupling daily output meant verified datasets were ready on time, so our multi-host pre-training runs started on schedule instead of waiting on data generation. As we scale our scientific foundation models from 9B to 35B parameters and beyond, we plan to use the same GKE Inference Gateway and llm-d routing setup for post-training reinforcement learning RL and automated numerical solver evaluation loops. Because each turn in those multi-turn rollouts builds on the context of the turns before it, they depend even more heavily on live queue balancing and warm prefix caches. To try these capabilities on your own workloads, explore the GKE Inference Gateway documentation https://cloud.google.com/kubernetes-engine/docs/concepts/about-gke-inference-gateway and check out the open-source llm-d project https://llm-d.ai/ . You can also read how the Gemini platform team cut latency by 35% with GKE Inference Gateway https://cloud.google.com/blog/products/containers-kubernetes/how-gke-inference-gateway-improved-latency-for-vertex-ai .