Cut coding agent costs by 90%: Secure LLM serving with Ray + vLLM on Anyscale Anyscale published a cost model and code repository showing that self-hosting coding-agent LLM inference with Ray Serve LLM and vLLM on its platform can cut per-developer costs to roughly $58 per month for always-on serving ($2,920 per month for 50 registered developers sharing one GPU) or about $17 per developer ($840 per month) for work-hours-only serving, versus an $800-per-month equivalent Claude Code API baseline. The figures assume about $4 per GPU-node hour for an RTX PRO 6000. Anyscale also reported empirical gains: 8.0x faster compilation with compile-cache restoration (48.5 to 6.0 seconds), 3.79x higher request throughput with a 512 GiB CPU KV-cache tier that raised cached prompt tokens from 6.1% to 74.9% and cut median time-to-first-token from 48.4 to 9.2 seconds, 1.86x higher output throughput from multi-token prediction (65 to 121 tokens/s), and 2.87x faster generation with CUDA graphs (15.9 to 45.6 tokens/s). Cut coding agent costs by 90%: Secure LLM serving with Ray + vLLM on Anyscale Kunling Geng https://anyscale.com/blog?author=kunling-geng , Aydin Abiar https://anyscale.com/blog?author=aydin-abiar and Xing Fang https://anyscale.com/blog?author=xing-fang | October 8, 2026 Coding agents repeatedly send repository context, instructions, tool results, and conversation history to generate actions. That work is also some of the most sensitive you do: your source, your internal docs, your architecture decisions, leaving your perimeter on every keystroke. Owning the runtime https://www.anyscale.com/blog/learning-loops is how that stays yours: the code, the traces, and the usage data that eventually could help you post-train those models to make them more tailored to your team. Beyond data privacy, paying for closed-source LLMs by the token also means a linearly growing bill per developer, per request, with no way to derive efficiencies of scale as the team and number of agents grows. Owning the runtime also shifts that by enabling a team to share the underlying GPUs. BMW found the cost-benefit https://youtu.be/yKt-CpfrOIE?si=mFz8jRYluEfBFqpg&t=46 of moving from API to in-house inference with as few as 8 developers. But building and managing your in-house LLM serving platform for coding comes with its own operational challenges when running GPU clusters. As far as costs, key findings from our cost model show that self-hosting costs a fraction of per-token API pricing per developer: - Always-on serving: About $2,920 per month, or about $58 per registered developer. - Work-hours-only serving: About $840 per month, or about $17 per registered developer. - API baseline: An active developer routinely runs up about $800 per month in equivalent Claude Code API costs. These figures assume roughly $4 per GPU-node hour for an RTX PRO 6000 and 50 registered developers sharing one GPU. They are planning projections rather than a TCO; the full assumptions and caveats are in the cost comparison section below. In addition to cost savings, optimizing the serving infrastructure yields measurable performance gains. Key findings from our empirical evaluations demonstrate significant performance improvements: - 8.0x faster compilation with compile-cache restoration. In a no-MTP NVFP4 setup, compilation time dropped from 48.5 to 6.0 seconds. - 3.79x higher request throughput with CPU KV-cache tiering. Adding a 512 GiB CPU tier in a single-replica GLM-5.2 evaluation increased the cached share of prompt tokens from 6.1% to 74.9% across 58 common prompts, reducing median time-to-first-token TTFT from 48.4 to 9.2 seconds. - 1.86x higher output throughput with speculative decoding. Multi-token prediction MTP increased output throughput from 65 to 121 tokens/s. - 2.87x faster output generation with CUDA graphs enabled. Client-observed token throughput increased from 15.9 to 45.6 tokens/s compared with running in eager mode. This guide provides a step-by-step roadmap covering model and hardware selection, service deployment, single-replica tuning, autoscaling with cache-aware routing, and multi-model operations. How the LLM serving stack operates: - vLLM : the inference engine that loads the model on GPUs and generates tokens. - Ray Serve LLM : the orchestration layer that scales, routes and load-balances requests across many vLLM replicas behind one OpenAI- and Anthropic-compatible endpoint, including advanced patterns like prefill/decode disaggregation. - Anyscale Platform : the managed production runtime that provisions GPUs, autoscales nodes including scale-to-zero , and adds reliability and observability such as zero-downtime rollouts, fault tolerance, logs, tracing and alerting. To make this path reproducible, we created the LLM serving for coding agents code repository https://github.com/anyscale/llm-serving-for-coding-agents . The repository is organized into five runnable parts: - Part 1: Deploy https://github.com/anyscale/llm-serving-for-coding-agents/tree/95af323db8951d1a7e5ea588a4e8c706ae15c115/part1-deploy-naive : Launch and validate a baseline Anyscale service. - Part 2: Connect https://github.com/anyscale/llm-serving-for-coding-agents/tree/95af323db8951d1a7e5ea588a4e8c706ae15c115/part2-connect-clients-production : Connect Claude Code, Codex, and Cursor through their native APIs. - Part 3: Optimize https://github.com/anyscale/llm-serving-for-coding-agents/tree/95af323db8951d1a7e5ea588a4e8c706ae15c115/part3-optimize : Tune GPU usage, caching, decoding, and autoscaling. - Part 4: Route https://github.com/anyscale/llm-serving-for-coding-agents/tree/95af323db8951d1a7e5ea588a4e8c706ae15c115/part4-litellm-router : Route between the open model and Claude with LiteLLM. - Part 5: Roll out https://github.com/anyscale/llm-serving-for-coding-agents/tree/95af323db8951d1a7e5ea588a4e8c706ae15c115/part5-enterprise-rollout : Configure zero-touch deployment for a team. LinkHow coding agents shift the serving workload LinkCoding agents produce far more than chat traffic Coding-agent prompts combine system instructions, repository context, tool definitions, diagnostics, and conversation history—often to produce only a short tool call or edit. This makes the workload prefill-heavy , with large inputs dominating time to first token TTFT . Most of that context is repeated across turns, creating strong opportunities for automatic prefix KV-cache reuse https://docs.vllm.ai/en/stable/features/automatic prefix caching/ . At team scale, correlated bursts add another challenge. A high-performance platform must therefore balance four goals: low TTFT, smooth token streaming, high cache reuse, and elastic capacity. Coding-agent harnesses act as API clients. When configured with custom endpoints, the local harness—or, for hosted clients such as Cursor, its cloud backend—sends requests to a compatible model server instead of relying on the default hosted model. Because these clients use different API schemas and endpoints, the serving infrastructure must expose the corresponding routes. The repository's client integration guide https://github.com/anyscale/llm-serving-for-coding-agents/blob/95af323db8951d1a7e5ea588a4e8c706ae15c115/part2-connect-clients-production/README.md configures these three paths: | Coding agent | API path | |---|---| | Cursor | /v1/chat/completions | | Claude Code | /v1/messages | | Codex | /v1/responses | Hosting an open-weight model on Anyscale can provide several benefits: - Reduce cost: Lower marginal token cost when utilization is high enough to offset fixed infrastructure expense. - Control capacity: Reduce dependence on a single provider's rate limits while managing your own capacity and queues. - Security and data control: Keep more of the inference path inside your network. - Reduce model-provider lock-in: Gain control over model and serving upgrades. - Evolve the LLM with your own data: Create a data flywheel by capturing approved agent traces, curating and evaluating them with Ray Data https://docs.ray.io/en/latest/data/data.html , and post-training with Ray Train https://docs.ray.io/en/latest/train/train.html or an RL framework. Community and transparency: Leverage the open ecosystem to understand and customize model behavior. LinkRay Serve LLM scales it for production While vLLM handles core inference optimizations—such as prefix caching https://docs.vllm.ai/en/stable/features/automatic prefix caching/ , chunked prefill https://docs.vllm.ai/en/stable/configuration/optimization/ chunked-prefill , quantization https://docs.vllm.ai/en/stable/features/quantization/quantized kvcache/ , speculative decoding https://docs.vllm.ai/en/stable/features/speculative decoding/ , and model parallelism https://docs.vllm.ai/en/stable/serving/parallelism scaling/ —a robust deployment needs more. Ray Serve LLM adds a production layer to: - Scale replicas across multi-GPU environments. - Route requests intelligently based on load and cache state. - Expose OpenAI- and Anthropic-compatible APIs. - Stream tokens from replicas through HAProxy while bypassing the legacy Python ingress hop. - Centralize metrics for services, models, and GPUs. - Ergonomic builders for complex, multi-node deployments like Wide-EP and prefill disaggregation A published recent Ray Serve LLM benchmark https://www.anyscale.com/blog/high-performance-distributed-inference-ray-serve-llm-vllm-google-kubernetes-gke measured cumulative throughput gains of up to 4.4x on prefill-heavy workloads and 24.8x on decode-heavy ones relative to an older, unbatched Ray Serve LLM baseline after introducing HAProxy, direct streaming, and RayExecutorV2. On Anyscale, Ray Serve LLM adds managed autoscaling, observability, and fault tolerance around the inference engine. LinkDeploy an LLM for coding agents A practical deployment has four steps: 1. Select a model based on task quality and serving footprint. 2. Select hardware and a parallelism strategy. 3. Configure the vLLM engine and Ray Serve deployment. 4. Deploy the service and validate each coding client. LinkStep 1: Select an LLM for task success, not leaderboard rank alone Open-weight model quality is moving quickly. Agent leaderboards such as the Agent Arena ranking for open-source models https://arena.ai/leaderboard/agent?license=open-source are fine places to start, and model families including Qwen, DeepSeek, Kimi, and GLM increasingly target coding and tool use. However, public rankings are only a starting point, and Anyscale LLM experts can help you select the best model for your specific workloads. Common benchmarks include SWE-bench https://www.swebench.com/ for resolving real repository issues, Terminal-Bench https://www.tbench.ai/benchmarks for terminal tasks, Toolathlon https://toolathlon.xyz/introduction for long-horizon multi-tool workflows, τ-bench https://taubench.com/ for conversational tool use, CyberGym https://www.cybergym.io/cybergym/ for vulnerability analysis, and GDPval-AA v2 https://artificialanalysis.ai/evaluations/gdpval-aa for professional knowledge work. Scores often reflect the full agent system—not only the model—so compare results only when benchmark versions and evaluation setups match. Evaluate candidate models on your own work: - Multi-file edits and repository navigation. - Structured tool calls over several turns. - Long-context instruction following. - Debugging and test repair. - Latency and number of turns required to finish a task. A smaller model that completes common work in one reliable turn can be cheaper than a larger model that needs more GPUs. A stronger model may be cheaper for difficult tasks if it avoids retries. Measure cost per successful task , not only cost per million tokens. LinkStep 2: Fit weights, runtime memory, and KV cache to the hardware GPU memory holds model weights, the KV cache, temporary activations, CUDA graphs, and runtime overhead. Weight memory is mostly fixed and scales with parameter count and precision, so weight quantization can be essential when unquantized weights constrain the deployment. The KV cache stores attention state for active requests and grows with context length and concurrency. For long-context coding-agent workloads, KV cache quantization can likewise reduce per-token memory. Quantizing both, when supported and quality-validated, leaves more room for concurrent long requests. When selecting a quantization strategy, consider the following formats: - FP8 https://docs.anyscale.com/llm/serving/performance-optimization : A strong option for both weights and the KV cache https://docs.vllm.ai/en/stable/features/quantization/quantized kvcache/ when supported by the hardware and runtime. It roughly halves raw storage compared with BF16; quality remains model-, task-, and calibration-dependent. - NVFP4 https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/ : On NVIDIA Blackwell GPUs, this format halves raw storage for quantized tensors compared with FP8. Whole-checkpoint savings are smaller because some modules and metadata remain at higher precision; the tested Qwen checkpoint used approximately 22 GB versus 27 GB for FP8, a reduction of about 19%. - MxFP4 https://huggingface.co/docs/transformers/quantization/mxfp4 : An open alternative for 4-bit precision, used by models such as gpt-oss . Note: Runtime setup and GPU architecture dictate the extent of hardware acceleration. While NVIDIA's Qwen3.6-27B-NVFP4 https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4 weights are deployed here, the pinned vLLM release relies on a Marlin fallback instead of a native kernel for dense NVFP4 on RTX PRO 6000 hardware. Furthermore, because NVFP4 KV caching is not supported on this device, the system defaults to an FP8 KV cache. LinkStep 3: Configure the engine for the model and workload A correct serving configuration is model-specific. In addition to memory and parallelism, it must use the right chat template, reasoning parser, tool-call parser, multimodal limits, and context length. The repository's Qwen serving implementation https://github.com/anyscale/llm-serving-for-coding-agents/blob/95af323db8951d1a7e5ea588a4e8c706ae15c115/part3-optimize/serve qwen3 6 27b optimized.py L138-L182 configures one GPU per replica, a 256K context limit, FP8 KV cache, local prefix caching, chunked prefill, and Qwen-specific reasoning and tool parsers. The following excerpt shows the relevant engine arguments; the implementation configures speculative decoding and compile-cache behavior separately: engine kwargs = { "tensor parallel size": 1, "max model len": 262144, "gpu memory utilization": 0.9, "max num seqs": 32, "max num batched tokens": 8192, "enable prefix caching": True, "kv cache dtype": "fp8", "quantization": "modelopt", "reasoning parser": "qwen3", "tool call parser": "qwen3 coder", "enable auto tool choice": True, "limit mm per prompt": {"image": 4, "video": 0}, } max model len caps a single request's context. max num batched tokens serves as a scheduler budget for chunked prefill. Meanwhile, max num seqs restricts engine concurrency. LinkStep 4: Enable direct streaming and connect the clients The repository's always-on service configuration https://github.com/anyscale/llm-serving-for-coding-agents/blob/95af323db8951d1a7e5ea588a4e8c706ae15c115/part3-optimize/service-always-on.yaml L12-L30 enables HAProxy and direct streaming at the service level: env vars: RAY SERVE ENABLE HA PROXY: "1" RAY SERVE LLM ENABLE DIRECT STREAMING: "1" Follow the repository's coding-agent connection tutorial https://github.com/anyscale/llm-serving-for-coding-agents/blob/95af323db8951d1a7e5ea588a4e8c706ae15c115/part2-connect-clients-production/README.md . First, deploy the optimized always-on service https://github.com/anyscale/llm-serving-for-coding-agents/tree/95af323db8951d1a7e5ea588a4e8c706ae15c115/part3-optimize : anyscale service deploy -f service-always-on.yaml --working-dir . Then copy its public base url and bearer token from Anyscale Console → Services → Query . The agent executes tools locally, but model requests—including instructions, repository context, conversation history, and tool results—and streamed model responses pass through the service. Claude Code Export the service URL and token, then run the repository's claude-service.sh https://github.com/anyscale/llm-serving-for-coding-agents/blob/95af323db8951d1a7e5ea588a4e8c706ae15c115/part2-connect-clients-production/claude-service.sh : export ANYSCALE BASE URL=