Over the past few months I read the entirety of @philipkiely's Inference Engineering and @cfregly's AI Systems Performance Engineering (awesome books btw). I wanted to understand firsthand what inference engineers go through when automating, tweaking, and productionizing model serving across hardware, traffic shapes, and latency targets. To experience that myself and test whether my vision for automated production inference infrastructure is possible, I built a hill-climbing benchmark harness and tuned a deployment from scratch. In a lot of cloud regions right now, Hopper and Blackwell GPUs are hard to get, while Ampere A100s are what most people may have a chance of reserving. Luckily, I was able to reserve a couple of single @Azure A100 80GB PCIe nodes in Central US ($4.15/hour on-demand). I paired the single nodes with @Nanbeige 4.3-33B (a 33B reasoning model from a family gaining traction on @ArtificialAnlys's small-model leaderboards). I then used @GitHub Copilot (with @Claude Opus 5.5) to run 337 automated hill-climb trials over 41 hours across @vllm_project 0.34.0 and @sgl_project 0.5.23. For this model and the four synthetic traffic shapes I tested, here is how much tuning improved performance over out-of-the-box vLLM (BF16 defaults) on a single A100:
- Single-user decode speed: Improved from 23.4 tokens/sec to 94.8 tokens/sec (4.1×) on vLLM, and 74.4 tokens/sec (3.1×) on SGLang.
- Sustained chat throughput (over 9-minute windows at ≤2s TTFT and ≤100ms TPOT): Increased from 1.47 requests/sec to 3.21 requests/sec (2.2×) on vLLM, and 3.24 requests/sec on SGLang.
- Multi-turn agentic latency (p90 time to first token across 6-turn sessions): Dropped from 114 seconds to 1.05 seconds on vLLM, and 0.94 seconds on SGLang.
- Azure Central US A100 serving cost ($4.15/hour on-demand): Cut sustained chat cost from $0.90 per 1M output tokens down to $0.41 per 1M output tokens.
To efficiently search the inference server config flag space (22 relevant settings in vLLM and 21 in SGLang), I built a controller on Azure Kubernetes Service (AKS) that runs each trial as an isolated pod on one A100 node (n0 or n1). The pod pairs the inference server with a
For a single user (concurrency=1), Nanbeige4.3-33B's ~62 GiB BF16 weights nearly fill an 80GB A100. Because every output token has to stream all ~62 GiB of weights from HBM (fitting ~4.6 ms + bytes / 1.69 TB/s, or 87% of the A100 PCIe's 1.935 TB/s peak bandwidth), BF16 tops out
On Hugging Face, @nanbeige released official BF16, FP8, INT8 (W8A16), and INT4 (W4A16) checkpoints. On an A100 (Ampere), all three quantized checkpoints run as weight-only (WxA16):
- The A100 has no FP8 tensor cores (those started on H100), so vLLM must unpack FP8 weights to
One of the biggest measurement lessons from this project was that 2-minute load tests end before the active batch reaches steady state. At vLLM's 2-minute synthetic capacity rate (2.61 requests/sec), the server averaged 99 running requests during minute 2, but grew to 210
In the 6-turn agentic benchmark (48 sessions arriving at 0.2 sessions/sec, a shared 3,000-token system prompt, and a new 1,200-token document added every turn), BF16 weights leave room for only 70,656 KV tokens in vLLM. Roughly 30 sessions are active at once holding ~2× that
No single static configuration won both single-user speed and batched capacity: leaving speculative decoding on at all batch sizes reduced INT4's 2-minute chat capacity by 65% (from 2.38 requests/sec down to 0.82 requests/sec), because verifying draft tokens adds compute per step
After screening weight formats, KV cache precision, and speculative decoding, I ran 8 coordinate-ascent hill climbs (199 challenger evaluations plus confirmation re-runs) across all 22 vLLM flags and 21 SGLang flags. To avoid accepting benchmark noise, score climbs required each
I also found a few settings that surprisingly regressed or failed during the search, including several that I expected to help:
- FP8 KV cache at batch size 1 (vLLM): I expected halving KV cache bytes to help or at least be neutral, but it reduced single-user decode speed by
I also found three benchmark bugs that I had to fix along the way:
- Passing --ignore-eos with a fixed 512-token output length inflated suffix decoding from 1.57× to 1.97×, and using synthetic random tokens made k=4 look 2.2× faster than k=6 (148.6 tokens/sec vs. 69.1
Whenever you quantize weights to INT4 or INT8 W8A8 (or quantize the KV cache to FP8), the obvious question is whether you degraded model quality to get faster benchmarks. Every winning config had to pass three automated quality checks against the unquantized BF16 baseline: 1.
With quality verified, here is what tuning did to serving cost at Central US Linux on-demand pricing for Standard_NC24ads_A100_v4 ($4.15/hour):
- Sustained chat: Cut cost from $0.90 per 1M output tokens on BF16 defaults down to $0.41 per 1M output tokens on both tuned vLLM and
As mentioned earlier, these winning @vllm_project and @sgl_project inference engine configs are specific to @nanbeige 4.3-33B, on a single @Azure A100 80GB PCIe node, and for the four synthetic traffic shapes I chose. Different models, hardware, prompt distributions, or latency