LLM Inference Cost Optimization: Run AI Inference for Less

Cast AI benchmark testing shows that continuous batching at batch size 8 reduces Llama 3.1 70B inference cost on a single H100 from approximately $0.60-$0.80 per million tokens to $0.15-$0.25 per million tokens, a 3-4x reduction with no additional hardware. The figures are based on Llama 3.1 70B FP8 on an H100 80GB SXM5 with vLLM 0.5+, 512-token prompt plus 256-token completion, and H100 spot pricing at approximately $2-4/hr on AWS eu-west-1 (2025). Cast AI's 2026 State of Kubernetes Optimization Report finds average GPU utilization across production Kubernetes fleets is only 5%, with the best-performing cluster (a 136-node H200 deployment) reaching 49%.

Running Llama 3.1 70B on a single H100 in single-stream mode costs approximately $0.60-$0.80 per million tokens . Continuous batching at batch size 8 reduces that to $0.15-$0.25 per million tokens – a 3-4x reduction with no additional hardware. These figures come from Cast AI benchmark testing: Llama 3.1 70B FP8 on an H100 80GB SXM5, vLLM 0.5+, 512-token prompt + 256-token completion, H100 spot pricing at approximately $2-4/hr on AWS eu-west-1 2025 . The GPU did not get faster. You stopped leaving it idle between requests. One caveat: this cost reduction applies to shorter context lengths, up to around 2K tokens. For longer contexts, the KV cache competes with weight memory and reduces the effective batch size you can sustain. That gap captures the core llm inference cost problem. According to Cast AI’s 2026 State of Kubernetes Optimization Report https://cast.ai/reports/state-of-kubernetes-optimization/ , average GPU utilization across production Kubernetes fleets sits at 5% . The best-performing cluster in that dataset, a 136-node H200 deployment, reached 49%. Most teams are nowhere near it. The short answer: LLM inference cost optimization reduces the cost of serving large language models on Kubernetes by raising GPU efficiency through sharing and batching, right-sizing GPU requests, and autoscaling inference endpoints to match demand, including scale-to-zero when idle. Most inference spend funds idle capacity, not useful work. This post covers five concrete levers you can apply today, with specific tools, metrics, and configuration patterns for each. Lever 1: GPU Sharing with MIG and Time-Slicing A single A100 80GB GPU can host multiple independent inference workloads simultaneously. NVIDIA Multi-Instance GPU MIG carves one physical GPU into hardware-isolated partitions, each with dedicated memory, compute, and L2 cache. This is not virtualization. Each partition behaves like an independent physical device. A100 80GB MIG profiles include: 1g.10gb : 1 slice, 10GB memory. Fits INT4/GPTQ 7B models ~4-5GB or small embedding models. 2g.20gb : 2 slices, 20GB memory. Fits 7B and 13B INT8 models 7B INT8 ~7GB, 13B INT8 ~13GB with KV cache headroom. 3g.40gb : 3 slices, 40GB memory. Fits INT4/AWQ models up to ~30GB with normal KV cache headroom. INT4 70B weights ~35GB approach the partition limit, leaving minimal KV cache space. 7g.80gb : Full GPU. Required for 70B models at AWQ precision ~35GB weights plus KV cache or any FP16/BF16 model above 34B parameters. One critical constraint: each MIG partition runs an independent model instance. Hardware isolation means NVLink and peer-to-peer CUDA communication are disabled between partitions. Tensor parallelism across two MIG slices on the same A100 is architecturally impossible. If your 70B model needs to be split across partitions, use full physical GPUs connected by NVLink instead. For Kubernetes deployments, the nvidia-device-plugin exposes MIG partitions as schedulable resources. A deployment targeting a 3g.40gb partition requests nvidia.com/mig-3g.40gb: 1 in its resource spec. Cast AI’s GPU automation layer reads these labels and provisions the correct node type automatically, without manual instance selection. resources: limits: nvidia.com/mig-3g.40gb: "1" Time-slicing is the alternative when MIG is not available T4, A10G, older Ampere cards without MIG support . Time-slicing multiplexes GPU access at the scheduling level rather than hardware partitioning it. Multiple pods share the same GPU context. Unlike MIG, there is no memory isolation between tenants. Noisy neighbor effects are real. Use time-slicing for development workloads or low-priority batch jobs, not latency-sensitive production inference. Lever 2: Continuous Batching, Speculative Decoding, and Prefix Caching Static batching holds a batch open until it is full, then processes all requests together. This approach made sense for offline workloads. For online inference, it forces short requests to wait for long ones to finish. Throughput suffers and tail latency spikes. Continuous batching solves this by treating generation as a stream of individual iterations. New requests join the batch as soon as a slot opens, rather than waiting for the entire batch to complete. vLLM uses continuous batching by default. The result is 3-5x higher throughput versus static batching at the same hardware budget, with lower P99 latency under mixed-length workloads. Two additional vLLM features compound the gains further. Speculative Decoding Speculative decoding uses a small draft model to propose multiple tokens ahead, which the main model verifies in a single forward pass. Token generation in autoregressive LLMs is memory-bandwidth-bound, not compute-bound. Verifying a batch of draft tokens costs almost the same compute as verifying one. When the draft model’s proposals are correct, you get several tokens per forward pass instead of one. In vLLM, enable it with: --speculative-model <draft-model-name --num-speculative-tokens 5 Speculative decoding is most effective for batch size 1-2 workloads: chat interfaces and copilot tools where per-request latency matters more than aggregate throughput. At those concurrency levels, it can reduce generation latency by 30-50% when a suitable small draft model is available. At higher concurrency or large batch sizes, running the draft model in parallel reduces the net benefit. Measure its effect at your typical request concurrency before committing to it in production. Prefix Caching Most chat and RAG workloads send the same system prompt with every request. Without prefix caching, the model recomputes the KV state for that prefix on every call. For a 512-token system prompt on a 70B model, that is significant wasted prefill compute. Prefix caching stores the KV state of common prefixes in GPU memory and reuses it across requests. Enable it in vLLM with: --enable-prefix-caching For chat or RAG deployments with long shared system prompts, prefix caching can eliminate 60-80% of prefill compute for repeat queries. This translates directly to lower time-to-first-token and higher effective throughput on the same hardware. If you run any RAG pipeline or chatbot with a fixed system prompt, this is a one-flag win. Lever 3: Quantization and llm inference cost Per Token Quantization reduces the numerical precision used to store model weights. Fewer bits per parameter means smaller VRAM footprint, higher memory throughput, and in some cases hardware-accelerated compute on the right GPU. The catch: lower precision introduces rounding error, and that error accumulates across layers. The right quantization format depends on your GPU architecture. Not all GPUs support all formats natively. | GPU | Architecture | Recommended Format | Notes | |---|---|---|---| | H100 80GB, H200 | Hopper | FP8 | Native FP8 Tensor Cores. 2x VRAM reduction vs BF16. 1.5-1.8x throughput gain. Use for production 70B+ deployments. | | A100 80GB production 70B | Ampere | AWQ / INT4 | AWQ 4-bit cuts a 70B model to ~35-37GB, leaving ~40GB for KV cache on an 80GB device. 1-3% MMLU drop. Recommended for most 70B production workloads on A100. | | A100 80GB smaller models or high accuracy | Ampere | INT8 | INT8 keeps each parameter at 1 byte. A 70B INT8 model weighs ~70GB, fitting an A100 80GB with minimal KV cache headroom. Better suited to 7B/13B models or when AWQ accuracy is unacceptable. | | T4, L4, A10G | Turing / Ampere | INT4 / GPTQ / GGUF | Limited VRAM 16-24GB . INT4 fits smaller models. Accuracy trade-off is more pronounced. | The VRAM math matters. A 70B model at FP16/BF16 requires approximately 140GB for weights alone. AWQ 4-bit on an A100 80GB cuts that to roughly 35-37GB, leaving around 40GB for KV cache and activations on a single 80GB device. INT8 keeps each parameter at 1 byte, so a 70B INT8 model still weighs approximately 70GB. It fits an A100 80GB but leaves minimal KV cache headroom. For most production 70B workloads on A100, AWQ is the practical choice. For FP8 inference, 70B models require native FP8 hardware support: the H100 80GB or H200. Running FP8 on an A100 via software emulation wastes cycles and degrades output quality. Accuracy Trade-offs by Format Quantization always involves a quality trade-off. The magnitude depends on format, model family, and task type. Benchmark your specific model before deploying quantized weights in production. FP8 on H100/H200: Typically less than 1% MMLU accuracy drop versus BF16. Recommended for production 70B+ deployments on Hopper hardware. AWQ on A100: Typically 1-3% accuracy drop versus BF16. AWQ preserves accuracy better than naive INT8 by identifying and protecting the most sensitive weight channels. INT8 on A100: Similar accuracy range to AWQ but at a ~70GB weight footprint for 70B models. More appropriate for 7B and 13B models where full VRAM is available and maximum accuracy is required. INT4 GPTQ/GGUF : Typically 3-6% accuracy drop. Use only when the model must fit a smaller GPU. Evaluate carefully on your benchmark tasks before production deployment. For most production 70B deployments on A100 80GB hardware, AWQ is the right call. You get a single-GPU deployment that works today, on widely available hardware, with a quantified and acceptable accuracy cost. Lever 4: Right-Sizing GPU Requests to Reduce Inference Cost Over-provisioning is the default behavior in Kubernetes GPU deployments. Teams request an H100 80GB for a workload that uses 9GB. The GPU runs at 8% utilization and bills at 100%. This pattern repeats across fleets at scale, as Cast AI’s 2026 State of Kubernetes Optimization Report documents in detail. Right-sizing GPU requests means matching resource declarations to actual peak usage, not worst-case assumptions. For inference workloads, the key metrics are: GPU memory utilization : Measured per-container via nvidia-smi or DCGM exporter. Compare peak memory usage against requested memory. A pod requesting 40GB but peaking at 18GB should move to a smaller partition or instance type. SM utilization : Streaming multiprocessor utilization indicates whether the model is compute-saturated or memory-bandwidth-bound. Most inference workloads are memory-bandwidth-bound, not compute-bound. High memory bandwidth utilization at low SM utilization is expected behavior, not a sign of wasted compute. Token throughput vs. SLO : Define a minimum tokens-per-second target per replica. Use this to set horizontal pod autoscaler thresholds. If throughput drops below the SLO, add replicas. If it is well above the SLO, reduce the instance size. Cast AI’s GPU right-sizing automation reads DCGM metrics continuously and surfaces recommendations based on observed workload behavior, not static rules. For teams running mixed model sizes across a shared cluster, automated right-sizing closes the gap between provisioned and consumed capacity without requiring manual profiling cycles. One common mistake: requesting nvidia.com/gpu: 1 without specifying MIG profiles or instance types. This allows the scheduler to land the pod on any available GPU, including an H100 80GB serving a 7B INT4 model. Explicit resource labels prevent this. Use node selectors or node affinity rules to constrain workloads to the correct GPU tier. Lever 5: Autoscaling and Scale-to-Zero Idle GPU replicas are the most expensive line item in any inference budget. A static deployment that never scales down keeps billing for capacity you are not using. H100 spot pricing ranges from $2-6/hr depending on cloud and region. Even at $3/hr spot, one idle H100 replica burns $2,160/month. At on-demand H100 pricing of $8-12/hr on major cloud providers, an idle replica costs $5,760-$8,640/month. Kubernetes Horizontal Pod Autoscaler HPA scales on CPU and memory by default. Neither metric is meaningful for GPU inference workloads. Effective autoscaling for LLM inference uses inference-specific signals: Queue depth : Number of pending inference requests waiting for a slot. A rising queue depth signals under-provisioning before latency degrades. Tokens per second per replica : A sustained drop below your SLO target triggers scale-out. A sustained rate above your target triggers scale-in. GPU memory pressure : KV cache utilization above 80% is a leading indicator of upcoming OOM evictions or quality degradation from KV cache eviction policies. KEDA Kubernetes Event-Driven Autoscaler supports custom metrics from Prometheus, making it the standard choice for inference-aware autoscaling. Below is a complete ScaledObject targeting vLLM’s queue-depth metric: Scale-to-zero is the most aggressive form of cost control. For development endpoints, internal tools, or low-traffic models, scaling to zero replicas when idle eliminates holding costs entirely. The trade-off is cold-start latency. A 70B AWQ model on an A100 80GB takes 30-90 seconds to load from object storage into GPU memory, depending on storage bandwidth. For latency-sensitive workloads, maintain a minimum of one warm replica. For batch or async workloads, scale-to-zero is often the right default. Cast AI’s workload autoscaling https://cast.ai/gpu-workload-scaling-policies/ integrates directly with GPU node lifecycle management. When the last inference replica scales down, Cast AI deprovisions the underlying GPU node and stops the billing clock. When demand returns, it provisions a new node and schedules the pod in parallel. This is the operational difference between HPA alone and full-stack automation: HPA scales pods, but the node keeps billing. Cast AI terminates the node. How Cast AI Reduces LLM Inference Cost at Scale Implementing these five levers manually means maintaining five separate tools, configurations, and escalation paths. Cast AI’s platform automates all five simultaneously through a single control plane, without requiring manual configuration of each component. The Cast AI GPU automation layer reads workload resource requests, matches them to optimal instance types across spot and on-demand pools, enforces right-sizing recommendations, and manages node lifecycle end-to-end. For inference workloads specifically, it handles MIG partition scheduling, GPU node bin-packing, and scale-to-zero node termination as a unified control plane. Fairgen, a generative AI company running production LLM inference on Kubernetes, reduced GPU infrastructure costs by 70% after deploying Cast AI, without changing their model stack or serving framework. The full breakdown is in the Cast AI LLM workloads case study https://cast.ai/blog/why-cast-ai-is-best-for-llm-workloads/ . The gains came from node right-sizing, autoscaling-triggered node termination, and spot instance coverage on non-latency-sensitive inference jobs. For teams at earlier stages of optimization, Cast AI’s GPU cost monitoring https://cast.ai/kubernetes-cost-monitoring/ dashboard gives per-workload GPU utilization, memory usage, and cost allocation in real time. Most teams discover that 2-3 workloads account for the majority of GPU spend. Fixing those first generates enough savings to fund the rest of the optimization program. The benchmark: the best-performing cluster in Cast AI’s 2026 fleet data achieved 49% GPU utilization across 136 H200 nodes. That is a production AI workload running with Cast AI autonomous optimization. The 5% fleet average shows how far most teams have to go. The 49% shows what is achievable without replacing hardware. Frequently Asked Questions What is LLM inference cost optimization? LLM inference cost optimization is the practice of reducing the GPU compute and memory expenses required to serve large language model responses at production scale. It includes techniques for raising GPU utilization batching, sharing , reducing per-token compute requirements quantization , matching allocated resources to actual demand right-sizing , and eliminating idle capacity costs autoscaling, scale-to-zero . What is the biggest driver of high LLM inference cost? Idle GPU capacity. Most production Kubernetes clusters run GPUs at 5% average utilization, according to Cast AI’s 2026 State of Kubernetes Optimization Report. The GPU bills at 100% of its hourly rate regardless of actual utilization. Closing that gap through batching, sharing, and autoscaling produces the largest cost reductions. Does quantization affect model output quality? Yes. Quantization reduces numerical precision and introduces rounding error. FP8 on an H100 typically causes less than 1% MMLU accuracy drop versus BF16, making it suitable for most production use cases. AWQ on an A100 causes 1-3% accuracy drop. INT4 GPTQ/GGUF causes 3-6% accuracy drop. Always benchmark your specific model and task before deploying quantized weights in production. Can I run a 70B model on an A100 80GB? Yes, with AWQ 4-bit quantization. A 70B model quantized with AWQ requires approximately 35-37GB for weights, leaving around 40GB for KV cache and activations on a single 80GB device. INT8 on A100 keeps each parameter at 1 byte, so a 70B INT8 model weighs approximately 70GB. It fits an A100 80GB but leaves minimal KV cache headroom. For most production 70B workloads on A100, AWQ is the practical choice. For FP8 inference with native hardware acceleration, you need an H100 80GB or H200. The A100 uses the Ampere architecture and does not have native FP8 Tensor Cores. Running FP8 on an A100 via software emulation provides no throughput benefit and risks accuracy degradation. Can I use MIG partitions for tensor parallelism on a 70B model? No. MIG partitions are hardware-isolated. NVLink and peer-to-peer CUDA communication are disabled between MIG instances on the same physical GPU. Tensor parallelism requires fast inter-GPU communication. If you need TP-2 or higher for a 70B model, use full physical GPUs connected by NVLink, not MIG partitions. When should I use speculative decoding? Speculative decoding is most valuable for low-concurrency, latency-sensitive chat workloads. At batch size 1-2, it can reduce generation latency by 30-50% when a suitable small draft model is available. At high concurrency or large batch sizes, the draft model overhead reduces its benefit. Measure the effect at your typical request concurrency before enabling it in production. What is prefix caching and when does it help? Prefix caching stores the KV attention state of repeated prompt prefixes, such as system prompts, in GPU memory. Subsequent requests that share that prefix skip the prefill computation for the cached portion. For chat or RAG workloads with long shared system prompts, prefix caching eliminates 60-80% of prefill compute on repeat queries. Enable it in vLLM with the --enable-prefix-caching flag.