{"slug": "cut-coding-agent-costs-by-90-secure-llm-serving-with-ray-vllm-on-anyscale", "title": "Cut coding agent costs by 90%: Secure LLM serving with Ray + vLLM on Anyscale", "summary": "Anyscale published a cost model and code repository showing that self-hosting coding-agent LLM inference with Ray Serve LLM and vLLM on its platform can cut per-developer costs to roughly $58 per month for always-on serving ($2,920 per month for 50 registered developers sharing one GPU) or about $17 per developer ($840 per month) for work-hours-only serving, versus an $800-per-month equivalent Claude Code API baseline. The figures assume about $4 per GPU-node hour for an RTX PRO 6000. Anyscale also reported empirical gains: 8.0x faster compilation with compile-cache restoration (48.5 to 6.0 seconds), 3.79x higher request throughput with a 512 GiB CPU KV-cache tier that raised cached prompt tokens from 6.1% to 74.9% and cut median time-to-first-token from 48.4 to 9.2 seconds, 1.86x higher output throughput from multi-token prediction (65 to 121 tokens/s), and 2.87x faster generation with CUDA graphs (15.9 to 45.6 tokens/s).", "body_md": "# Cut coding agent costs by 90%: Secure LLM serving with Ray + vLLM on Anyscale\n\n[Kunling Geng](https://anyscale.com/blog?author=kunling-geng),\n\n[Aydin Abiar](https://anyscale.com/blog?author=aydin-abiar)and\n\n[Xing Fang](https://anyscale.com/blog?author=xing-fang)| October 8, 2026\n\nCoding agents repeatedly send repository context, instructions, tool results, and conversation history to generate actions. That work is also some of the most sensitive you do: your source, your internal docs, your architecture decisions, leaving your perimeter on every keystroke. [__Owning the runtime__](https://www.anyscale.com/blog/learning-loops) is how that stays yours: the code, the traces, and the usage data that eventually could help you post-train those models to make them more tailored to your team.\n\nBeyond data privacy, paying for closed-source LLMs by the token also means a linearly growing bill per developer, per request, with no way to derive efficiencies of scale as the team and number of agents grows. Owning the runtime also shifts that by enabling a team to share the underlying GPUs. [__BMW found the cost-benefit__](https://youtu.be/yKt-CpfrOIE?si=mFz8jRYluEfBFqpg&t=46) of moving from API to in-house inference with as few as 8 developers. But building and managing your in-house LLM serving platform for coding comes with its own operational challenges when running GPU clusters. \n\nAs far as costs, key findings from our cost model show that self-hosting costs a fraction of per-token API pricing per developer:\n\n- **Always-on serving:** About $2,920 per month, or about $58 per registered developer.\n- **Work-hours-only serving:** About $840 per month, or about $17 per registered developer.\n- **API baseline:** An active developer routinely runs up about $800 per month in equivalent Claude Code API costs.\n\nThese figures assume roughly $4 per GPU-node hour for an RTX PRO 6000 and 50 registered developers sharing one GPU. They are planning projections rather than a TCO; the full assumptions and caveats are in the cost comparison section below.\n\nIn addition to cost savings, optimizing the serving infrastructure yields measurable performance gains.\n\nKey findings from our empirical evaluations demonstrate significant performance improvements:\n\n- **8.0x faster compilation with compile-cache restoration.** In a no-MTP NVFP4 setup, compilation time dropped from 48.5 to 6.0 seconds.\n- **3.79x higher request throughput with CPU KV-cache tiering.** Adding a 512 GiB CPU tier in a single-replica GLM-5.2 evaluation increased the cached share of prompt tokens from 6.1% to 74.9% across 58 common prompts, reducing median time-to-first-token (TTFT) from 48.4 to 9.2 seconds.\n- **1.86x higher output throughput with speculative decoding.** Multi-token prediction (MTP) increased output throughput from 65 to 121 tokens/s.\n- **2.87x faster output generation with CUDA graphs enabled.** Client-observed token throughput increased from 15.9 to 45.6 tokens/s compared with running in eager mode.\n\nThis guide provides a step-by-step roadmap covering model and hardware selection, service deployment, single-replica tuning, autoscaling with cache-aware routing, and multi-model operations.\n\n**How the LLM serving stack operates:**\n\n- **vLLM** : the inference engine that loads the model on GPUs and generates tokens.\n- **Ray Serve LLM** : the orchestration layer that scales, routes and load-balances requests across many vLLM replicas behind one OpenAI- and Anthropic-compatible endpoint, including advanced patterns like prefill/decode disaggregation.\n- **Anyscale Platform** : the managed production runtime that provisions GPUs, autoscales nodes (including scale-to-zero), and adds reliability and observability such as zero-downtime rollouts, fault tolerance, logs, tracing and alerting.\n\nTo make this path reproducible, we created the [__LLM serving for coding agents code repository__](https://github.com/anyscale/llm-serving-for-coding-agents).\n\nThe repository is organized into five runnable parts:\n\n- [**__Part 1: Deploy__**](https://github.com/anyscale/llm-serving-for-coding-agents/tree/95af323db8951d1a7e5ea588a4e8c706ae15c115/part1-deploy-naive)**:** Launch and validate a baseline Anyscale service.\n- [**__Part 2: Connect__**](https://github.com/anyscale/llm-serving-for-coding-agents/tree/95af323db8951d1a7e5ea588a4e8c706ae15c115/part2-connect-clients-production)**:** Connect Claude Code, Codex, and Cursor through their native APIs.\n- [**__Part 3: Optimize__**](https://github.com/anyscale/llm-serving-for-coding-agents/tree/95af323db8951d1a7e5ea588a4e8c706ae15c115/part3-optimize)**:** Tune GPU usage, caching, decoding, and autoscaling.\n- [**__Part 4: Route__**](https://github.com/anyscale/llm-serving-for-coding-agents/tree/95af323db8951d1a7e5ea588a4e8c706ae15c115/part4-litellm-router)**:** Route between the open model and Claude with LiteLLM.\n- [**__Part 5: Roll out__**](https://github.com/anyscale/llm-serving-for-coding-agents/tree/95af323db8951d1a7e5ea588a4e8c706ae15c115/part5-enterprise-rollout)**:** Configure zero-touch deployment for a team.\n\n## LinkHow coding agents shift the serving workload\n\n### LinkCoding agents produce far more than chat traffic\n\nCoding-agent prompts combine system instructions, repository context, tool definitions, diagnostics, and conversation history—often to produce only a short tool call or edit. This makes the workload **prefill-heavy**, with large inputs dominating time to first token (TTFT).\n\nMost of that context is repeated across turns, creating strong opportunities for [__automatic prefix KV-cache reuse__](https://docs.vllm.ai/en/stable/features/automatic_prefix_caching/). At team scale, correlated bursts add another challenge. A high-performance platform must therefore balance four goals: low TTFT, smooth token streaming, high cache reuse, and elastic capacity.\n\nCoding-agent harnesses act as API clients. When configured with custom endpoints, the local harness—or, for hosted clients such as Cursor, its cloud backend—sends requests to a compatible model server instead of relying on the default hosted model. Because these clients use different API schemas and endpoints, the serving infrastructure must expose the corresponding routes. The repository's [__client integration guide__](https://github.com/anyscale/llm-serving-for-coding-agents/blob/95af323db8951d1a7e5ea588a4e8c706ae15c115/part2-connect-clients-production/README.md) configures these three paths:\n\n| **Coding agent** | **API path** | \n|---|---|\n| Cursor | `/v1/chat/completions` | \n| Claude Code | `/v1/messages` | \n| Codex | `/v1/responses` | \n\nHosting an open-weight model on Anyscale can provide several benefits:\n\n- **Reduce cost:** Lower marginal token cost when utilization is high enough to offset fixed infrastructure expense.\n- **Control capacity:** Reduce dependence on a single provider's rate limits while managing your own capacity and queues.\n- **Security and data control:** Keep more of the inference path inside your network.\n- **Reduce model-provider lock-in:** Gain control over model and serving upgrades.\n- **Evolve the LLM with your own data:** Create a data flywheel by capturing approved agent traces, curating and evaluating them with[__Ray Data__](https://docs.ray.io/en/latest/data/data.html) , and post-training with[__Ray Train__](https://docs.ray.io/en/latest/train/train.html) or an RL framework.\n\n**Community and transparency:** Leverage the open ecosystem to understand and customize model behavior.\n\n### LinkRay Serve LLM scales it for production\n\nWhile vLLM handles core inference optimizations—such as [__prefix caching__](https://docs.vllm.ai/en/stable/features/automatic_prefix_caching/), [__chunked prefill__](https://docs.vllm.ai/en/stable/configuration/optimization/#chunked-prefill), [__quantization__](https://docs.vllm.ai/en/stable/features/quantization/quantized_kvcache/), [__speculative decoding__](https://docs.vllm.ai/en/stable/features/speculative_decoding/), and [__model parallelism__](https://docs.vllm.ai/en/stable/serving/parallelism_scaling/)—a robust deployment needs more. Ray Serve LLM adds a production layer to:\n\n- Scale replicas across multi-GPU environments.\n- Route requests intelligently based on load and cache state.\n- Expose OpenAI- and Anthropic-compatible APIs.\n- Stream tokens from replicas through HAProxy while bypassing the legacy Python ingress hop.\n- Centralize metrics for services, models, and GPUs.\n- Ergonomic builders for complex, multi-node deployments like Wide-EP and prefill disaggregation\n\nA published [__recent Ray Serve LLM benchmark__](https://www.anyscale.com/blog/high-performance-distributed-inference-ray-serve-llm-vllm-google-kubernetes-gke) measured cumulative throughput gains of up to 4.4x on prefill-heavy workloads and 24.8x on decode-heavy ones relative to an older, unbatched Ray Serve LLM baseline after introducing HAProxy, direct streaming, and RayExecutorV2. On Anyscale, Ray Serve LLM adds managed autoscaling, observability, and fault tolerance around the inference engine.\n\n## LinkDeploy an LLM for coding agents\n\nA practical deployment has four steps:\n\n1. Select a model based on task quality and serving footprint.\n2. Select hardware and a parallelism strategy.\n3. Configure the vLLM engine and Ray Serve deployment.\n4. Deploy the service and validate each coding client.\n\n### LinkStep 1: Select an LLM for task success, not leaderboard rank alone\n\nOpen-weight model quality is moving quickly. Agent leaderboards such as the [__Agent Arena ranking for open-source models__](https://arena.ai/leaderboard/agent?license=open-source) are fine places to start, and model families including Qwen, DeepSeek, Kimi, and GLM increasingly target coding and tool use. However, public rankings are only a starting point, and Anyscale LLM experts can help you select the best model for your specific workloads.\n\nCommon benchmarks include [__SWE-bench__](https://www.swebench.com/) for resolving real repository issues, [__Terminal-Bench__](https://www.tbench.ai/benchmarks) for terminal tasks, [__Toolathlon__](https://toolathlon.xyz/introduction) for long-horizon multi-tool workflows, [__τ-bench__](https://taubench.com/) for conversational tool use, [__CyberGym__](https://www.cybergym.io/cybergym/) for vulnerability analysis, and [__GDPval-AA v2__](https://artificialanalysis.ai/evaluations/gdpval-aa) for professional knowledge work. Scores often reflect the full agent system—not only the model—so compare results only when benchmark versions and evaluation setups match.\n\nEvaluate candidate models on your own work:\n\n- Multi-file edits and repository navigation.\n- Structured tool calls over several turns.\n- Long-context instruction following.\n- Debugging and test repair.\n- Latency and number of turns required to finish a task.\n\nA smaller model that completes common work in one reliable turn can be cheaper than a larger model that needs more GPUs. A stronger model may be cheaper for difficult tasks if it avoids retries. Measure **cost per successful task**, not only cost per million tokens.\n\n### LinkStep 2: Fit weights, runtime memory, and KV cache to the hardware\n\nGPU memory holds model weights, the KV cache, temporary activations, CUDA graphs, and runtime overhead. Weight memory is mostly fixed and scales with parameter count and precision, so **weight quantization** can be essential when unquantized weights constrain the deployment. The KV cache stores attention state for active requests and grows with context length and concurrency. For long-context coding-agent workloads, **KV cache quantization** can likewise reduce per-token memory. Quantizing both, when supported and quality-validated, leaves more room for concurrent long requests.\n\nWhen selecting a quantization strategy, consider the following formats:\n\n- [**__FP8__**](https://docs.anyscale.com/llm/serving/performance-optimization)**:** A strong option for both weights and the[__KV cache__](https://docs.vllm.ai/en/stable/features/quantization/quantized_kvcache/) when supported by the hardware and runtime. It roughly halves raw storage compared with BF16; quality remains model-, task-, and calibration-dependent.\n- [**__NVFP4__**](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/)**:** On NVIDIA Blackwell GPUs, this format halves raw storage for quantized tensors compared with FP8. Whole-checkpoint savings are smaller because some modules and metadata remain at higher precision; the tested Qwen checkpoint used approximately 22 GB versus 27 GB for FP8, a reduction of about 19%.\n- [**__MxFP4__**](https://huggingface.co/docs/transformers/quantization/mxfp4)**:** An open alternative for 4-bit precision, used by models such as`gpt-oss` .\n\nNote: Runtime setup and GPU architecture dictate the extent of hardware acceleration. While NVIDIA's [__Qwen3.6-27B-NVFP4__](https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4) weights are deployed here, the pinned vLLM release relies on a Marlin fallback instead of a native kernel for dense NVFP4 on RTX PRO 6000 hardware. Furthermore, because NVFP4 KV caching is not supported on this device, the system defaults to an FP8 KV cache.\n\n### LinkStep 3: Configure the engine for the model and workload\n\nA correct serving configuration is model-specific. In addition to memory and parallelism, it must use the right chat template, reasoning parser, tool-call parser, multimodal limits, and context length.\n\nThe repository's [__Qwen serving implementation__](https://github.com/anyscale/llm-serving-for-coding-agents/blob/95af323db8951d1a7e5ea588a4e8c706ae15c115/part3-optimize/serve_qwen3_6_27b_optimized.py#L138-L182) configures one GPU per replica, a 256K context limit, FP8 KV cache, local prefix caching, chunked prefill, and Qwen-specific reasoning and tool parsers. The following excerpt shows the relevant engine arguments; the implementation configures speculative decoding and compile-cache behavior separately:\n\n```\nengine_kwargs = {\n    \"tensor_parallel_size\": 1,\n    \"max_model_len\": 262144,\n    \"gpu_memory_utilization\": 0.9,\n    \"max_num_seqs\": 32,\n    \"max_num_batched_tokens\": 8192,\n    \"enable_prefix_caching\": True,\n    \"kv_cache_dtype\": \"fp8\",\n    \"quantization\": \"modelopt\",\n    \"reasoning_parser\": \"qwen3\",\n    \"tool_call_parser\": \"qwen3_coder\",\n    \"enable_auto_tool_choice\": True,\n    \"limit_mm_per_prompt\": {\"image\": 4, \"video\": 0},\n}\n```\n\n`max_model_len` caps a single request's context. `max_num_batched_tokens` serves as a scheduler budget for chunked prefill. Meanwhile, `max_num_seqs` restricts engine concurrency. \n\n### LinkStep 4: Enable direct streaming and connect the clients\n\nThe repository's [__always-on service configuration__](https://github.com/anyscale/llm-serving-for-coding-agents/blob/95af323db8951d1a7e5ea588a4e8c706ae15c115/part3-optimize/service-always-on.yaml#L12-L30) enables HAProxy and direct streaming at the service level:\n\n```\nenv_vars:\n  RAY_SERVE_ENABLE_HA_PROXY: \"1\"\n  RAY_SERVE_LLM_ENABLE_DIRECT_STREAMING: \"1\"\n```\n\nFollow the repository's [__coding-agent connection tutorial__](https://github.com/anyscale/llm-serving-for-coding-agents/blob/95af323db8951d1a7e5ea588a4e8c706ae15c115/part2-connect-clients-production/README.md). First, deploy the [__optimized always-on service__](https://github.com/anyscale/llm-serving-for-coding-agents/tree/95af323db8951d1a7e5ea588a4e8c706ae15c115/part3-optimize):\n\n```\nanyscale service deploy -f service-always-on.yaml --working-dir .\n```\n\nThen copy its public `base_url` and bearer token from **Anyscale Console → Services → Query**. The agent executes tools locally, but model requests—including instructions, repository context, conversation history, and tool results—and streamed model responses pass through the service.\n\n**Claude Code**\n\nExport the service URL and token, then run the repository's [__claude-service.sh__](https://github.com/anyscale/llm-serving-for-coding-agents/blob/95af323db8951d1a7e5ea588a4e8c706ae15c115/part2-connect-clients-production/claude-service.sh):\n\n```\nexport ANYSCALE_BASE_URL=<base_url>/v1\nexport ANYSCALE_API_KEY=<token>\n./claude-service.sh\n```\n\nThe launcher passes the service root to Claude Code as `ANTHROPIC_BASE_URL` and supplies the token through `ANTHROPIC_AUTH_TOKEN`.\n\n**Codex**\n\nFrom the same directory, run [__codex-service.sh__](https://github.com/anyscale/llm-serving-for-coding-agents/blob/95af323db8951d1a7e5ea588a4e8c706ae15c115/part2-connect-clients-production/codex-service.sh):\n\n```\n./codex-service.sh\n```\n\nIt registers the `/v1` URL as a custom provider, selects `wire_api=\"responses\"`, and sends the bearer token to `/v1/responses`.\n\n**Cursor**\n\nFollow the tutorial's [__Cursor setup__](https://github.com/anyscale/llm-serving-for-coding-agents/tree/95af323db8951d1a7e5ea588a4e8c706ae15c115/part2-connect-clients-production#cursor-gui) and enter these values under **Cursor Settings → Models → OpenAI API Key**:\n\n```\nOverride OpenAI Base URL:  <base_url>/v1\nOpenAI API Key:            <token>\nCustom model:              qwen3.6-27b\n```\n\n## LinkOptimize one replica for coding-agent traffic\n\n### LinkBenchmarking latency against defined SLOs\n\nFor comprehensive definitions of TTFT, ITL, TPOT, prefill, decode, and goodput, refer to [__Anyscale's LLM metrics guide__](https://docs.anyscale.com/llm/serving/benchmarking/metrics).\n\nRather than focusing solely on raw token throughput, prioritize goodput, the proportion of requests meeting target latency SLOs. While aggressive batching elevates overall throughput, it can negatively impact interactive streaming performance.\n\n### LinkReplay real agent sessions for benchmarks\n\nUniform synthetic prompts fail to capture a true agent loop. Real-world traffic combines long and short turns, repeated prefixes, tool-execution waits, and heavy-tailed session lifetimes. To accurately test your setup, convert your Claude Code JSONL sessions into timestamped request streams that preserve prompt growth and inter-turn delays. If you aren't collecting your own sessions yet, you can use the public [__WekaTrace Claude Code agent-session corpus__](https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126-256k).\n\nFor each candidate configuration, collect:\n\n- TTFT, TPOT, ITL, and end-to-end latency distributions.\n- Completed agent turns and successful tasks.\n- Input/output tokens and cached-prompt-token fraction.\n- Queue depth, preemption, KV eviction, and GPU utilization.\n- Tool-call errors and malformed responses.\n\n### LinkApply optimizations in dependency order\n\nThe [__single-GPU measurements__](https://github.com/anyscale/llm-serving-for-coding-agents/blob/95af323db8951d1a7e5ea588a4e8c706ae15c115/part3-optimize/notes/BENCHMARKS.md) below use one RTX PRO 6000. Most decode results were recorded using real Claude session prompts averaging about 73K tokens. Each row isolates a different knob; the gains are not cumulative.\n\n| **Optimization** | **Before** | **After** | **Measured change** | \n|---|---|---|---|\n| CUDA graphs vs. eager mode (FP8 weights) | 15.9 output tokens/s | 45.6 output tokens/s | 2.87x | \n| Multi-token prediction vs. base (NVFP4 weights) | 65 output tokens/s | 121 output tokens/s | 1.86x | \n| RunAI Streamer | About 85 s weight load | About 25 s weight load | 3.4x | \n| Restored compile cache, vLLM 0.25.1 | 48.5 s compile stage | 6.0 s compile stage | 8.0x | \n\n**1.** __Quantize weights__ **to fit the model efficiently.** Weight quantization can turn a multi-GPU deployment into a one-GPU deployment, removing inter-GPU communication and freeing memory for KV cache. The benefit depends on the GPU's native kernels and the checkpoint's calibration. Treat every precision change as both a performance and quality change.\n\n**2.** __Quantize the KV cache__ **when supported.** The optimized deployment uses FP8 KV cache. Capacity calculations estimated KV storage equivalent to approximately 3.27x 256K-token contexts with BF16 KV and 6.53x with FP8 KV on the tested 96 GB GPU. \n\n**3. Keep** __CUDA graphs__ **enabled for production.** Eager mode launches operations with more CPU involvement and is useful for debugging. CUDA graphs capture repeatable GPU work and reduce launch overhead. Disabling eager mode (keeping CUDA graphs enabled) increased the output rate from 15.9 to 45.6 tokens/s.\n\n**4. Test** __speculative decoding__ **against target workloads.** Multi-token prediction proposes and verifies several future tokens simultaneously to reduce TPOT; the `num_speculative_tokens` parameter controls the number of draft tokens predicted (for example, setting it to 3 to predict three tokens ahead). However, net speedup depends on acceptance rates and verification overhead. At concurrency 8, predicting three tokens yielded 99 output tokens/s and 0.50 turns/s—outperforming two tokens (80 tokens/s, 0.50 turns/s)—whereas predicting four tokens degraded performance to 74 tokens/s. Higher speculative depth is not inherently better. Prioritize output accuracy: upstream issues note corrupted tool calls when combining MTP with prefix caching. Verify client, model, parser, and vLLM compatibility with multi-turn regression tests before enabling MTP.\n\n**5. Streamline initialization to separate cold starts from steady-state serving.** Model download, weight loading, engine compilation, and replica readiness are distinct phases that impact deployment speed. Compile caches are sensitive to the exact software version, GPU architecture, model, parallelism, and flags, meaning a stale cache can fail or silently degrade performance. While RunAI Streamer reduced earlier weight-load times, the repository's [__incompatibility matrix__](https://github.com/anyscale/llm-serving-for-coding-agents/blob/95af323db8951d1a7e5ea588a4e8c706ae15c115/part3-optimize/notes/INCOMPATIBILITIES.md) documents a known conflict between its loader path and MTP in the tested vLLM version.\n\nIn practice, first establish correct tool use with speculative decoding disabled; use eager mode only as needed for debugging. Once correctness is verified, enable the intended quantization and context length, followed by CUDA graphs. From there, tune chunked prefill and test speculative decoding. Finally, optimize the cold start and rerun the full correctness and load suite.\n\n### LinkModel cost comparisons\n\nThe repository's [__cost model and assumptions__](https://github.com/anyscale/llm-serving-for-coding-agents/blob/95af323db8951d1a7e5ea588a4e8c706ae15c115/part3-optimize/notes/COST-ESTIMATE.md) assume roughly $4 per GPU-node hour for the RTX PRO 6000 shape, with 50 registered developers sharing a single GPU. Based on these assumptions:\n\n| **Planning mode** | **GPU-node hours/month** | **GPU-node cost/month** | **Cost per registered developer** | \n|---|---|---|---|\n| Always on | 730 | About $2,920 | About $58 | \n| Work hours only | 210 | About $840 | About $17 | \n\nClaude pricing via Azure Marketplace on [__Microsoft Foundry__](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/concepts/claude-models) follows [__Claude Opus 5 rates__](https://platform.claude.com/docs/en/about-claude/pricing#claude-in-microsoft-foundry-pricing): $5.00 per million tokens (MTok) for standard input, $0.50/MTok for cache reads, and $25.00/MTok for output.\n\nAn active developer routinely runs up $800 per month in equivalent Claude Code API costs. This aligns with industry benchmarks, including [__Pylon's observations__](https://www.linkedin.com/posts/martykausas_our-anthropic-bill-is-about-to-jump-from-activity-7470505066148352000-zqnt) of power users reaching $800/month and [__another similar estimate__](https://simonwillison.net/2026/May/27/product-market-fit/) of $1,199.79 over 30 days. Public trace telemetry confirms that coding agent workloads are heavily skewed toward input tokens:\n\n- The [__WekaTrace dataset__](https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126-256k) indicates a 117:1 input-to-output token ratio.\n- SyFI TraceLab's [__Claude trace__](https://tracelab.cs.washington.edu/) documents a 269:1 ratio with a 95.2% prompt cache hit rate.\n\nFactoring in SyFI's 95.2% cached input rate (billed as cache reads) and treating the remaining 4.8% as five-minute cache writes, Opus 5 pricing translates an $800 monthly spend into approximately 922M input tokens and 3.43M output tokens per developer. For an engineering team of 50 developers, overall API usage totals roughly $40,000 and 171M output tokens per month.\n\nThe $40,000 monthly figure serves purely as an API expenditure benchmark rather than a direct indicator of potential infrastructure savings. Because brief, configuration-specific decode tests ignore prefill overhead, request queuing, overall GPU utilization, latency-SLO targets, and variations in model quality, they cannot reliably forecast real-world worker capacity.\n\nSimilarly, per-developer GPU cost calculations represent preliminary planning projections instead of a total cost of ownership (TCO) or service-level commitment. To accurately determine required replica counts and net savings, evaluate sustained trace-replay goodput against target latency SLOs and task success rates, then factor in additional platform operational overhead.\n\n## LinkScale with autoscaling and cache-aware load balancing\n\n### LinkSeparate engine limits, admission limits, and autoscaling targets\n\nThe repository's [__autoscaling configuration__](https://github.com/anyscale/llm-serving-for-coding-agents/blob/95af323db8951d1a7e5ea588a4e8c706ae15c115/part3-optimize/serve_qwen3_6_27b_optimized.py#L204-L220) provides one concrete starting point, while [__Anyscale's parameter-tuning guide__](https://docs.anyscale.com/llm/serving/parameter-tuning#ray-serve-autoscaling-configuration) explains how to tune it. Three controls serve different purposes:\n\n- The vLLM `max_num_seqs` setting bounds sequences admitted by an engine.\n- Ray Serve's `max_ongoing_requests` bounds work sent to a replica.\n- The `target_ongoing_requests` setting tells the autoscaler when to add replicas.\n\nHere is the step-by-step process for setting up auto-scaling for LLM deployments serving coding agents:\n\n1. **Identify peak KV-cache capacity:** Check server startup logs to determine the maximum concurrent sequences supported at your maximum context length (e.g., 256K context windows typically sustain around 20 concurrent streams).\n2. **Estimate practical workload concurrency:** Adjust the sequence limit based on your average prompt size. For instance, with an average prompt length of 85K tokens, scale the target using (256K / 85K) x 20 to establish a target of roughly 60 concurrent requests.\n3. **Set hard admission bounds:** Match the engine's scheduler capacity (`max_num_seqs` in vLLM) and the deployment guard (`max_ongoing_requests` in Ray Serve) to this calculated upper limit (e.g., 60).\n4. **Establish autoscaling triggers:** Configure`target_ongoing_requests` roughly 20–30% lower than the admission ceiling (e.g., 40) to initiate replica scaling before requests queue up.\n5. **Track metrics and refine parameters:** Continuously evaluate system goodput and queuing. Reduce concurrency thresholds if prompt lengths approach the 256K ceiling to preserve tail latency as memory headroom narrows.\n\nFor coding agents, we recommend using Ray Serve LLM's [__KVAwareRouter__](https://docs.ray.io/en/master/serve/llm/user-guides/kv-aware-routing.html) or consistent hashing (session affinity). KV-aware routing combines prefix cache reuse with token-load-aware routing. However, note that KV-aware routing is currently only compatible with Codex. For other clients or incompatible setups, fall back to consistent hashing to maintain session stickiness.\n\nAs detailed in the [__blog__](https://www.anyscale.com/blog/llm-kv-token-aware-routing), the router uses KV cache overlap to estimate how much prefill can be skipped, then estimates the work the engine still has to do:\n\n```\nremaining prefill = incoming input tokens - reusable cached tokens\ntoken-load score = weighted remaining prefill + active prefill + ongoing decode\n```\n\nThe router selects the replica with the lowest weighted score and may trade some cache reuse for more balanced token load. On heterogeneous coding-agent traces, this can improve TTFT, TPOT, and throughput relative to maximizing cache reuse alone.\n\n### LinkExtend useful cache lifetime with KV cache offloading\n\nWhen a developer pauses to review code or write a prompt, other team traffic may evict their session's blocks from limited GPU memory. [__KV-cache offloading__](https://docs.ray.io/en/releases-2.58.0/serve/llm/user-guides/kv-cache-offloading.html) moves inactive blocks to larger CPU-memory or local-storage tiers so they can be restored instead of fully recomputed. For coding agents, offloading preserves session context when a user steps away and returns hours later—the history is retrieved from CPU/disk instead of recomputed, so they resume with a fast TTFT. \n\nIn an internal benchmark on a GLM-5.2 workload, adding a 512 GiB CPU KV-cache tier alongside GPU prefix caching significantly improved throughput and latency over GPU caching alone. Evaluated using a single node equipped with 8xRTX PRO 6000 GPUs (tensor parallelism of eight, NVFP4 weights, and FP8 KV cache), the benchmark replayed a seeded workload over a nominal 900-second window.\n\n| **Metric (900s Run)** | **GPU Cache Only** | **GPU Cache + CPU Tier Offload** | \n|---|---|---|\n| Completed Requests | 58 | 215 | \n| Request Throughput | 0.0625 req/s | 0.2367 req/s | \n\nKey operational takeaways include:\n\n- **Throughput Improvement:** Overall request throughput increased by 3.79x.\n- **Higher Cache Efficiency:** Across the 58 shared completed prompts, the cached-prompt-token fraction expanded from 6.1% to 74.9%.\n- **Reduced Latency:** Median time-to-first-token (TTFT) dropped from 48.4 to 9.2 seconds, with 49 out of 58 requests experiencing lower TTFT.\n\n## LinkMulti-Model Routing for Cost Optimization \n\nWhile replica routing optimizes hardware efficiency, multi-model routing optimizes overall spend. It dictates how much traffic pays frontier API premiums versus running on self-hosted infrastructure. Combining both gives you full control over performance and cost.\n\nThree common policies are:\n\n1. **Route by task complexity.** Send routine edits, searches, and known Agent Skills to a smaller open model; reserve a frontier model for difficult reasoning or critical changes.\n2. **Fall back when self-hosted LLM service capacity is constrained.** If GPUs cannot scale quickly enough for a burst, use an approved hosted model instead of allowing the queue to grow without bound.\n3. **Fall back to open models when hosted quotas are exhausted.** Keep work moving when a subscription or API reaches a limit.\n\nA CPU-only LiteLLM gateway can implement model selection in front of separate Ray Serve LLM services and external providers. The repository includes a [__deployable gateway__](https://github.com/anyscale/llm-serving-for-coding-agents/tree/95af323db8951d1a7e5ea588a4e8c706ae15c115/part4-litellm-router/gateway) with [__primary, fallback, and complexity-router configuration__](https://github.com/anyscale/llm-serving-for-coding-agents/blob/95af323db8951d1a7e5ea588a4e8c706ae15c115/part4-litellm-router/gateway/config.yaml#L27-L128), plus a [__routing-policy guide__](https://github.com/anyscale/llm-serving-for-coding-agents/blob/95af323db8951d1a7e5ea588a4e8c706ae15c115/part4-litellm-router/ROUTING.md). LiteLLM's [__auto-routing documentation__](https://docs.litellm.ai/docs/proxy/auto_routing) covers the underlying complexity of the router. Keyword and skill-aware rules are fast and explainable; a classifier can support richer policies. Explicit user overrides remain important because no router perfectly predicts task difficulty.\n\nIn our internal setup, the gateway routes simple requests to Qwen, intermediate to advanced text tasks to GLM, and complex reasoning or multimodal workloads to Claude. To prevent subsequent tool-result steps from unexpectedly changing models mid-stream, the router enforces session stickiness throughout the remaining agent session.\n\nMaintaining this session stickiness is critical for smooth operations. Because coding-agent iterations frequently pass tool outputs without a new user prompt, evaluating routing on a per-request basis risks shifting a single session across disparate models with conflicting chat formats, context bounds, and tool mechanics.\n\n### LinkContinuous improvement via observability and post-training\n\nTrack system performance across task, model, and platform dimensions using [__Anyscale service monitoring__](https://docs.anyscale.com/services/monitoring) alongside the [__Ray Serve LLM dashboard__](https://docs.ray.io/en/latest/serve/llm/user-guides/observability.html).\n\nHigh-quality execution traces drive an iterative post-training pipeline. Log multi-turn interactions through an observability platform such as Langfuse, curate and preprocess datasets with [__Ray Data__](https://docs.ray.io/en/latest/data/data.html), detect performance regressions using automated evaluation suites, and post-train open models with SkyRL, a reinforcement learning framework that runs on Ray and Anyscale. [__Coinbase took this approach for its Onramp fraud agent__](https://www.coinbase.com/blog/part-one-how-coinbase-built-evaluated-and-post-trained-a-fraud-agent). It post-trained Qwen3.5-9B with SkyRL on its proprietary fraud data, and the resulting model outperformed Claude Opus 4.5 on all four fraud-detection metrics while cutting end-to-end serving latency by 55%. Each round of post-training feeds the improved model back into production, which turns a static deployment into a self-improving agent flywheel.\n\n```\n/anyscale-workload-llm-serving Follow the guide in https://github.com/anyscale/llm-serving-for-coding-agents/tree/main/part3-optimize and deploy the coding-agent LLM service as written. Validate the generated artifacts in an Anyscale Workspace, deploy them as an Anyscale Service, and test the endpoint with /anyscale-platform-run skill.\n```\n\n#### LinkOptimize an existing LLM service\n\nPoint the serving skill at your current service and the repository's measured optimization example:\n\n```\n/anyscale-workload-llm-serving Deploy an optimized LLM service by following the guidance in https://github.com/anyscale/llm-serving-for-coding-agents/tree/main/part3-optimize. Use this code as a template, inspect my existing service, and adapt the model, GPU, context length, parsers, and performance settings as needed. Benchmark the baseline and optimized configurations with /anyscale-platform-run skill.\n```\n\nFeel free to substitute another LLM; the skill should revalidate model-specific parsers, runtime compatibility, memory fit, and performance settings rather than copying the Qwen configuration unchanged.", "url": "https://wpnews.pro/news/cut-coding-agent-costs-by-90-secure-llm-serving-with-ray-vllm-on-anyscale", "canonical_source": "https://anyscale.com/blog/run-coding-agents-on-your-own-gpus", "published_at": "2026-10-08 09:00:00+00:00", "updated_at": "2026-11-06 02:21:00.735866+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-agents", "mlops", "ai-tools"], "entities": ["Anyscale", "Ray Serve LLM", "vLLM", "Kunling Geng", "Aydin Abiar", "Xing Fang", "BMW", "Claude Code"], "also_reported_by": [], "takeaway": "Adding a 512 GiB CPU KV-cache tier to a single-replica GLM-5.2 deployment raised cached prompt tokens from 6.1% to 74.9% and cut median time-to-first-token from 48.4 to 9.2 seconds, while CUDA graphs lifted client-observed throughput from 15.9 to 45.6 tokens/s.", "recommendation": null, "alternates": {"html": "https://wpnews.pro/news/cut-coding-agent-costs-by-90-secure-llm-serving-with-ray-vllm-on-anyscale", "markdown": "https://wpnews.pro/news/cut-coding-agent-costs-by-90-secure-llm-serving-with-ray-vllm-on-anyscale.md", "text": "https://wpnews.pro/news/cut-coding-agent-costs-by-90-secure-llm-serving-with-ray-vllm-on-anyscale.txt", "jsonld": "https://wpnews.pro/news/cut-coding-agent-costs-by-90-secure-llm-serving-with-ray-vllm-on-anyscale.jsonld"}}