Google Cloud has released a blueprint for securing AI workloads on Google Kubernetes Engine (GKE), consolidating controls across multiple services to address infrastructure, model supply chain, and ap…
Google's GKE Inference Gateway introduces prefix-aware load balancing for LLMs, routing requests to replicas that already hold cached context to avoid reprocessing shared prompt prefixes. This approac…
Google Cloud's GKE Inference Gateway delivers up to 92.8% shorter wait times and 62.6% lower inter-token latency compared to the next leading managed Kubernetes service, according to an independent be…
Google Cloud demonstrated a multi-cluster inference setup deploying an LLM across two regional GKE clusters using TPU v6e accelerators and GKE managed DRANET for networking. The experiment, documented…