Hill-climbing vLLM and SGLang configs on a single A100 A hill-climbing benchmark harness running 337 automated trials over 41 hours on a single Azure A100 80GB PCIe node ($4.15/hour on-demand) raised vLLM 0.34.0 single-user decode speed for Nanbeige 4.3-33B from 23.4 to 94.8 tokens/sec (4.1×) and SGLang 0.5.23 to 74.4 tokens/sec (3.1×), the author reported. Sustained chat throughput at ≤2s TTFT and ≤100ms TPOT rose from 1.47 to 3.21 requests/sec on vLLM (2.2×) and 3.24 requests/sec on SGLang, while p90 time to first token across 6-turn agentic sessions fell from 114 seconds to 1.05 seconds on vLLM and 0.94 seconds on SGLang. The tuning cut Azure Central US serving cost from $0.90 to $0.41 per 1M output tokens, and the author found no single static configuration won both single-user speed and batched capacity — leaving speculative decoding on cut INT4's 2-minute chat capacity 65%, from 2.38 to 0.82 requests/sec. Over the past few months I read the entirety of @philipkiely's Inference Engineering and @cfregly's AI Systems Performance Engineering awesome books btw . I wanted to understand firsthand what inference engineers go through when automating, tweaking, and productionizing model serving across hardware, traffic shapes, and latency targets. To experience that myself and test whether my vision for automated production inference infrastructure is possible, I built a hill-climbing benchmark harness and tuned a deployment from scratch. In a lot of cloud regions right now, Hopper and Blackwell GPUs are hard to get, while Ampere A100s are what most people may have a chance of reserving. Luckily, I was able to reserve a couple of single @Azure A100 80GB PCIe nodes in Central US $4.15/hour on-demand . I paired the single nodes with @Nanbeige 4.3-33B a 33B reasoning model from a family gaining traction on @ArtificialAnlys's small-model leaderboards . I then used @GitHub Copilot with @Claude Opus 5.5 to run 337 automated hill-climb trials over 41 hours across @vllm project 0.34.0 and @sgl project 0.5.23. For this model and the four synthetic traffic shapes I tested, here is how much tuning improved performance over out-of-the-box vLLM BF16 defaults on a single A100: 1. Single-user decode speed: Improved from 23.4 tokens/sec to 94.8 tokens/sec 4.1× on vLLM, and 74.4 tokens/sec 3.1× on SGLang. 2. Sustained chat throughput over 9-minute windows at ≤2s TTFT and ≤100ms TPOT : Increased from 1.47 requests/sec to 3.21 requests/sec 2.2× on vLLM, and 3.24 requests/sec on SGLang. 3. Multi-turn agentic latency p90 time to first token across 6-turn sessions : Dropped from 114 seconds to 1.05 seconds on vLLM, and 0.94 seconds on SGLang. 4. Azure Central US A100 serving cost $4.15/hour on-demand : Cut sustained chat cost from $0.90 per 1M output tokens down to $0.41 per 1M output tokens. To efficiently search the inference server config flag space 22 relevant settings in vLLM and 21 in SGLang , I built a controller on Azure Kubernetes Service AKS that runs each trial as an isolated pod on one A100 node n0 or n1 . The pod pairs the inference server with a For a single user concurrency=1 , Nanbeige4.3-33B's ~62 GiB BF16 weights nearly fill an 80GB A100. Because every output token has to stream all ~62 GiB of weights from HBM fitting ~4.6 ms + bytes / 1.69 TB/s, or 87% of the A100 PCIe's 1.935 TB/s peak bandwidth , BF16 tops out On Hugging Face, @nanbeige released official BF16, FP8, INT8 W8A16 , and INT4 W4A16 checkpoints. On an A100 Ampere , all three quantized checkpoints run as weight-only WxA16 : 1. The A100 has no FP8 tensor cores those started on H100 , so vLLM must unpack FP8 weights to One of the biggest measurement lessons from this project was that 2-minute load tests end before the active batch reaches steady state. At vLLM's 2-minute synthetic capacity rate 2.61 requests/sec , the server averaged 99 running requests during minute 2, but grew to 210 In the 6-turn agentic benchmark 48 sessions arriving at 0.2 sessions/sec, a shared 3,000-token system prompt, and a new 1,200-token document added every turn , BF16 weights leave room for only 70,656 KV tokens in vLLM. Roughly 30 sessions are active at once holding ~2× that No single static configuration won both single-user speed and batched capacity: leaving speculative decoding on at all batch sizes reduced INT4's 2-minute chat capacity by 65% from 2.38 requests/sec down to 0.82 requests/sec , because verifying draft tokens adds compute per step After screening weight formats, KV cache precision, and speculative decoding, I ran 8 coordinate-ascent hill climbs 199 challenger evaluations plus confirmation re-runs across all 22 vLLM flags and 21 SGLang flags. To avoid accepting benchmark noise, score climbs required each I also found a few settings that surprisingly regressed or failed during the search, including several that I expected to help: 1. FP8 KV cache at batch size 1 vLLM : I expected halving KV cache bytes to help or at least be neutral, but it reduced single-user decode speed by I also found three benchmark bugs that I had to fix along the way: 1. Passing --ignore-eos with a fixed 512-token output length inflated suffix decoding from 1.57× to 1.97×, and using synthetic random tokens made k=4 look 2.2× faster than k=6 148.6 tokens/sec vs. 69.1 Whenever you quantize weights to INT4 or INT8 W8A8 or quantize the KV cache to FP8 , the obvious question is whether you degraded model quality to get faster benchmarks. Every winning config had to pass three automated quality checks against the unquantized BF16 baseline: 1. With quality verified, here is what tuning did to serving cost at Central US Linux on-demand pricing for Standard NC24ads A100 v4 $4.15/hour : 1. Sustained chat: Cut cost from $0.90 per 1M output tokens on BF16 defaults down to $0.41 per 1M output tokens on both tuned vLLM and As mentioned earlier, these winning @vllm project and @sgl project inference engine configs are specific to @nanbeige 4.3-33B, on a single @Azure A100 80GB PCIe node, and for the four synthetic traffic shapes I chose. Different models, hardware, prompt distributions, or latency