cd /news/ai-infrastructure/hill-climbing-vllm-and-sglang-config… · home › topics › ai-infrastructure › article
[ARTICLE · art-173976] src=twitter.com ↗ pub= topic=ai-infrastructure verified=true sentiment=↑ positive

Hill-climbing vLLM and SGLang configs on a single A100

A hill-climbing benchmark harness running 337 automated trials over 41 hours on a single Azure A100 80GB PCIe node ($4.15/hour on-demand) raised vLLM 0.34.0 single-user decode speed for Nanbeige 4.3-33B from 23.4 to 94.8 tokens/sec (4.1×) and SGLang 0.5.23 to 74.4 tokens/sec (3.1×), the author reported. Sustained chat throughput at ≤2s TTFT and ≤100ms TPOT rose from 1.47 to 3.21 requests/sec on vLLM (2.2×) and 3.24 requests/sec on SGLang, while p90 time to first token across 6-turn agentic sessions fell from 114 seconds to 1.05 seconds on vLLM and 0.94 seconds on SGLang. The tuning cut Azure Central US serving cost from $0.90 to $0.41 per 1M output tokens, and the author found no single static configuration won both single-user speed and batched capacity — leaving speculative decoding on cut INT4's 2-minute chat capacity 65%, from 2.38 to 0.82 requests/sec.

recommendation

Tune serving flags per workload rather than shipping one static config: disable speculative decoding for batched chat, since leaving it on cut INT4 capacity 65%, and run load tests past 2 minutes so the batch reaches steady state.

more in playbook: Inference and self-hosting
read4 min views3 publishedNov 21, 2026
Hill-climbing vLLM and SGLang configs on a single A100
Image: source

Over the past few months I read the entirety of @philipkiely's Inference Engineering and @cfregly's AI Systems Performance Engineering (awesome books btw). I wanted to understand firsthand what inference engineers go through when automating, tweaking, and productionizing model serving across hardware, traffic shapes, and latency targets. To experience that myself and test whether my vision for automated production inference infrastructure is possible, I built a hill-climbing benchmark harness and tuned a deployment from scratch. In a lot of cloud regions right now, Hopper and Blackwell GPUs are hard to get, while Ampere A100s are what most people may have a chance of reserving. Luckily, I was able to reserve a couple of single @Azure A100 80GB PCIe nodes in Central US ($4.15/hour on-demand). I paired the single nodes with @Nanbeige 4.3-33B (a 33B reasoning model from a family gaining traction on @ArtificialAnlys's small-model leaderboards). I then used @GitHub Copilot (with @Claude Opus 5.5) to run 337 automated hill-climb trials over 41 hours across @vllm_project 0.34.0 and @sgl_project 0.5.23. For this model and the four synthetic traffic shapes I tested, here is how much tuning improved performance over out-of-the-box vLLM (BF16 defaults) on a single A100:

  1. Single-user decode speed: Improved from 23.4 tokens/sec to 94.8 tokens/sec (4.1×) on vLLM, and 74.4 tokens/sec (3.1×) on SGLang.
  2. Sustained chat throughput (over 9-minute windows at ≤2s TTFT and ≤100ms TPOT): Increased from 1.47 requests/sec to 3.21 requests/sec (2.2×) on vLLM, and 3.24 requests/sec on SGLang.
  3. Multi-turn agentic latency (p90 time to first token across 6-turn sessions): Dropped from 114 seconds to 1.05 seconds on vLLM, and 0.94 seconds on SGLang.
  4. Azure Central US A100 serving cost ($4.15/hour on-demand): Cut sustained chat cost from $0.90 per 1M output tokens down to $0.41 per 1M output tokens.

To efficiently search the inference server config flag space (22 relevant settings in vLLM and 21 in SGLang), I built a controller on Azure Kubernetes Service (AKS) that runs each trial as an isolated pod on one A100 node (n0 or n1). The pod pairs the inference server with a

For a single user (concurrency=1), Nanbeige4.3-33B's ~62 GiB BF16 weights nearly fill an 80GB A100. Because every output token has to stream all ~62 GiB of weights from HBM (fitting ~4.6 ms + bytes / 1.69 TB/s, or 87% of the A100 PCIe's 1.935 TB/s peak bandwidth), BF16 tops out

On Hugging Face, @nanbeige released official BF16, FP8, INT8 (W8A16), and INT4 (W4A16) checkpoints. On an A100 (Ampere), all three quantized checkpoints run as weight-only (WxA16):

  1. The A100 has no FP8 tensor cores (those started on H100), so vLLM must unpack FP8 weights to

One of the biggest measurement lessons from this project was that 2-minute load tests end before the active batch reaches steady state. At vLLM's 2-minute synthetic capacity rate (2.61 requests/sec), the server averaged 99 running requests during minute 2, but grew to 210

In the 6-turn agentic benchmark (48 sessions arriving at 0.2 sessions/sec, a shared 3,000-token system prompt, and a new 1,200-token document added every turn), BF16 weights leave room for only 70,656 KV tokens in vLLM. Roughly 30 sessions are active at once holding ~2× that

No single static configuration won both single-user speed and batched capacity: leaving speculative decoding on at all batch sizes reduced INT4's 2-minute chat capacity by 65% (from 2.38 requests/sec down to 0.82 requests/sec), because verifying draft tokens adds compute per step

After screening weight formats, KV cache precision, and speculative decoding, I ran 8 coordinate-ascent hill climbs (199 challenger evaluations plus confirmation re-runs) across all 22 vLLM flags and 21 SGLang flags. To avoid accepting benchmark noise, score climbs required each

I also found a few settings that surprisingly regressed or failed during the search, including several that I expected to help:

  1. FP8 KV cache at batch size 1 (vLLM): I expected halving KV cache bytes to help or at least be neutral, but it reduced single-user decode speed by

I also found three benchmark bugs that I had to fix along the way:

  1. Passing --ignore-eos with a fixed 512-token output length inflated suffix decoding from 1.57× to 1.97×, and using synthetic random tokens made k=4 look 2.2× faster than k=6 (148.6 tokens/sec vs. 69.1

Whenever you quantize weights to INT4 or INT8 W8A8 (or quantize the KV cache to FP8), the obvious question is whether you degraded model quality to get faster benchmarks. Every winning config had to pass three automated quality checks against the unquantized BF16 baseline: 1.

With quality verified, here is what tuning did to serving cost at Central US Linux on-demand pricing for Standard_NC24ads_A100_v4 ($4.15/hour):

  1. Sustained chat: Cut cost from $0.90 per 1M output tokens on BF16 defaults down to $0.41 per 1M output tokens on both tuned vLLM and

As mentioned earlier, these winning @vllm_project and @sgl_project inference engine configs are specific to @nanbeige 4.3-33B, on a single @Azure A100 80GB PCIe node, and for the four synthetic traffic shapes I chose. Different models, hardware, prompt distributions, or latency

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @vllm 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/hill-climbing-vllm-a…] indexed:0 read:4min 2026-11-21 · —