Not All LLM Workloads Are Equal: Benchmarking TPU Performance on Classification vs. Generation

Google Cloud TPU v6e benchmarks show Gemma 3 27B hits a performance wall past 64 concurrent users in generation tasks, plateauing at a 4.12x normalized throughput multiplier at 128 users, while Gemma 3 12B scales to 8.19x, according to a deployment analysis by the vllm-project/tpu-inference team. For prefill-heavy classification tasks, both models achieve similar peak scaling (6.0x to 6.4x at 128 users) without saturating TPUs, suggesting larger models can be deployed for such workloads without throughput penalties.

Moving Large Language Models LLMs from experimental prototypes into enterprise production exposes a critical truth: your infrastructure dictates both your performance ceilings and your unit economics. Standard hardware benchmarks often ignore a fundamental reality—not all LLM requests stress the silicon in the same way. In this post, we dive into a comprehensive benchmarking exercise comparing Gemma 3 12B and Gemma 3 27B on Google Cloud TPU v6e to answer a crucial architectural question: How does TPU infrastructure actually perform when tasked with structurally distinct workloads at scale? Before diving into the methodology, here are the critical takeaways for architects deploying Gemma 3 on TPU v6e: For decode-heavy generation tasks, the Gemma 3 27B model hits a strict performance wall past 64 concurrent users, plateauing at a 4.12x normalized throughput multiplier at 128 users. In contrast, the 12B model scales up to an 8.19x multiplier. Suggestion : If your workload requires high-concurrency generation, downsize to the 12B model, or set strict pod-autoscaling limits capping concurrent requests at 64 per replica for the 27B model. For prefill-heavy classification tasks, model parameter size matters significantly less. Both the 12B and 27B models achieve similar peak scaling around 6.0x to 6.4x normalized throughput at 128 users without saturating the TPUs. Suggestion : You can safely deploy larger, more capable models for summarization or classification workflows without paying a throughput penalty. The average --max-num-seqs or --max-model-len should be kept judiciously based on the average user load and average tokens per request, without which there might be request drops. Hardware saturation manifests as severe latency spikes and silent request dropouts. To mitigate this, do not rely on standard CPU/Memory scaling triggers. Instead, scale based on End-to-End E2E latency metrics, and implement aggressive vLLM bucket padding optimizations VLLM TPU BUCKET PADDING GAP to conserve memory. The inference stack can be divided into three core pillars: 1. Infrastructure: GKE & TPU The foundation of our deployment is a Google Kubernetes Engine GKE Autopilot cluster. Connected to this is a single-host TPU v6e node pool configured with a 2x2 chip topology. 2. Software & Tools: vllm For the serving framework, we leveraged vllm via vllm-project/tpu-inference https://github.com/vllm-project/tpu-inference . 3. Models: Gemma 3 12B and 27B We evaluated two highly capable open-weights models: Gemma 3 12B and Gemma 3 27B. These models were accessed via HuggingFace. Not all LLM requests stress the system equally. We benchmarked two distinct scenarios: Classification and Generation, across 16, 32, 64, and 128 concurrent users: We measured metrics like Throughput requests/sec , End-to-End Latency and the results provided some fascinating insights into how parameter size and hardware bandwidth interact. To ensure architectural consistency, every benchmark was executed using the vllm-project/tpu-inference https://github.com/vllm-project/tpu-inference hardware plugin, leveraging a standardized global serving configuration of max-model-len=128000 , max-num-batched-tokens=8192 , and max-num-seqs=512 . In Generation tasks, both models perform similarly up to 64 concurrent users. However, at 128 concurrent users, the Gemma 3 12B model shows significantly better scaling, achieving an 8.19x normalized throughput multiplier compared to a 4.12x plateau for the Gemma 3 27B model normalized against the Gemma 3 12B baseline at 16 users . This suggests that the larger 27B model hits memory or compute limits much earlier under high generation loads. | Concurrent Users | Gemma 3 12B Throughput req/s | Gemma 3 27B Throughput req/s | | 16 users | 1.00 x | 1.05 x | | 32 users | 1.98 x | 1.97 x | | 64 users | 2.96 x | 4.00 x | | 128 users | 8.19 x | 4.12 x | In Classification tasks, there is negligible difference in scaling behavior between the Gemma 3 12B and Gemma 3 27B models. Both models operate efficiently within the hardware's capacity and scale well, reaching peak normalized throughputs of approximately 6.04x to 6.37x at 128 concurrent users normalized against the Gemma 3 12B baseline at 16 users . | Concurrent Users | Gemma 3 12B Throughput req/s | Gemma 3 27B Throughput req/s | | 16 users | 1.00 x | 0.76x | | 32 users | 1.18x | 1.53x | | 64 users | 2.04x | 3.15x | | 128 users | 6.37x | 6.04x | End-to-End E2E latency exhibits different scaling behaviors depending on the model size and task. When using identical serving hyperparameters --max-num-seqs=512 , the Gemma 3 12B model's Classification latency roughly doubles when moving from 32 users to 64 users, indicating resource contention. However, for the larger Gemma 3 27B model, Classification latency remains relatively flat between 32 and 64 users before doubling at the 128-user mark. | Model | Task | 16 Users | 32 Users | 64 Users | 128 Users | | Gemma 3 12B | Generation | 1.00x | 1.13x | 1.40x | 1.70x | | Gemma 3 12B | Classification | 1.00x | 0.99x | 1.79x | 2.90x | | Gemma 3 27B | Generation | 1.20x | 1.68x | 2.93x | 3.33x | | Gemma 3 27B | Classification | 1.20x | 1.95x | 1.95x | 3.88x | Benchmarking Gemma 3 12B and 27B models on Google Cloud TPU v6e architecture reveals that raw parameter count is not the sole predictor of inference performance; rather, the interaction between the serving framework, hardware topology, and workload token ratios dictates efficiency. For generation tasks low input, high output , the 12B model proves superior at high concurrency, sustaining an 8.19x relative throughput multiplier where the 27B model saturates at 4.12x. Conversely, for prefill-heavy classification tasks, both models perform similarly, allowing organizations to deploy larger models without a severe scaling penalty. Our evaluation also mapped exact hardware saturation thresholds—such as End-to-End latency doubling at 64 users for classification and hitting a cliff at 128 users for generation—enabling precise, data-driven auto-scaling triggers rather than costly over-provisioning. Ultimately, achieving these peak metrics requires aggressive tuning of vllm parameters, such as adjusting batched tokens and configuring TPU-specific bucket padding to prevent compute waste, proving that cost-effective AI infrastructure must strictly align model selection and serving configurations to the unique input/output profiles of production workloads. Don't let unoptimized infrastructure bottleneck your enterprise AI rollouts. Now that you know how different workload shapes impact hardware saturation, it's time to put these insights into practice: Have you encountered similar performance walls in your own production deployments? Share your scaling strategies, ask questions, and join the discussion in the Google Cloud Community forums https://www.googlecloudcommunity.com/ .