Hot reload vLLM and sglang configs A new open-source tool called trimtab lets operators change SGLang and vLLM scheduler settings live, cutting configuration changes from a 1-7 minute redeploy to about 15 ms, with zero dropped requests on H100, RTX PRO 6000, and B200 GPUs. The patches, roughly 100 lines per engine, are submitted as upstream PRs (sglang #37661, vllm #55018) and support hot, warm, and cold knob classes, with only weight changes still requiring a restart. Change SGLang and vLLM scheduler settings while the server runs. No restart, weights never leave the GPU. - Concurrency cap, queue limit, prefill size, schedule policy, log level change live in about 15 ms. Was a full redeploy. - KV pool resize with weights resident in 2 s on SGLang, 8 to 10 s on vLLM. Was a 1 to 7 minute redeploy plus a weight reload. - Zero dropped requests across every run, measured on H100, RTX PRO 6000, and B200 with both engines. - Every scheduler knob is live or seconds-warm. Only changing the weights themselves parallelism, quantization, the model still needs a restart. - A patch of about 100 lines per engine, upstream PRs open on both sglang 37661 https://github.com/sgl-project/sglang/pull/37661 , vllm 55018 https://github.com/vllm-project/vllm/pull/55018 . A trim tab is the small surface on a ship's rudder that steers the rudder that steers the ship. trimtab is a small patch and a control plane that steer inference engines without restarting them. Inference engines read their flags once at startup. To change a concurrency cap, a queue limit, or a prefill chunk size you kill the process and reload the weights. On the machines in this repo that costs one to seven minutes per change. Tuning becomes an offline chore and every experiment pays a reboot. Most of those settings are plain scheduler state. Nothing on the GPU is allocated when they change. They are startup flags because nobody wired a setter. When SGLang or vLLM starts, it copies a few flags into variables on the scheduler object. From then on the flag is dead. The scheduler reads the variable many times a second to decide what to run next. trimtab changes that variable on the running server. Each engine already has a way to send a command into the scheduler. SGLang has a POST /set internal state route that fans out to every scheduler rank, guarded by an allowlist of five niche keys. vLLM dispatches utility RPCs to EngineCore by method name. The patch adds a handler on each side that takes the new value, checks it against the ceiling the engine sized memory for at boot, and writes it. The scheduler picks it up on its next step. The SGLang patch is about 70 lines. The vLLM patch is about 90 and needs the server to run with VLLM SERVER DEV MODE=1 , the same gate vLLM puts on its own dev routes. The patches are scheduler-level Python with no kernels and no architecture-specific code, so trimtab runs on any GPU the engine runs on. trimtab supports the knobs in the manifests and nothing else. Every other flag is set at launch as before. Knobs come in three classes. Hot changes live. Warm rebuilds the KV pool in place with weights resident. Cold relaunches. A knob is hot only after the line where the scheduler reads it on every step has been checked in the engine source. That line is the last column. Every knob below was set, read back, and restored on a live engine on real hardware the knob sweep in the results . | engine | knob | allowed values | per-step read | |---|---|---|---| | sglang 0.5.18 | max running requests | 1 to boot value | scheduler.py 2274, 2302, 3626 | | sglang 0.5.18 | max queued requests | 0 or more | scheduler.py 3077 | | sglang 0.5.18 | chunked prefill size | 1 or more | scheduler.py 3600 | | sglang 0.5.18 | max prefill tokens | 1 to KV pool size | scheduler.py 3621 | | sglang 0.5.18 | schedule policy | fcfs, lpm, dfs-weight, lof, random, priority | schedule policy.py 257, 294, 297 | | sglang 0.5.18 | schedule conservativeness | above 0, rebuilds the new-token-ratio watermarks | scheduler.py 3620, 3890 to 3948 | | sglang 0.5.18 | log level | DEBUG, INFO, WARNING, ERROR | every log call | | vllm 0.28.0 | max num seqs | 1 to boot value | v1/core/sched/scheduler.py 792 | | vllm 0.28.0 | max num batched tokens | 1 to boot value | v1/core/sched/scheduler.py 440, 529 | | vllm 0.28.0 | long prefill token threshold | 0 or more | v1/core/sched/scheduler.py 441, 598, 992 | | vllm 0.28.0 | log level | DEBUG, INFO, WARNING, ERROR | every log call | SGLang's five upstream keys pp max micro batch size, two speculative accept thresholds, two dspark controls still work through the same route. Knobs that look hot and are not, with the reason recorded in the manifest so nobody re-investigates. Speculative depth on both engines, the draft CUDA graphs are captured for a fixed depth at boot. SGLang's watchdog timeout, it is passed by value into a thread at boot. vLLM's scheduling policy, the waiting queue object is built for one policy at init and switching needs a queue rebuild. Neither engine has a scheduler-level request deadline knob, deadlines are per-request parameters. Both engines, measured. The engine drains, frees the KV pool and CUDA graphs, rebuilds pools, backends and graphs at the new size, and rewires the scheduler. The weights never move. SGLang needs --enable-memory-saver , vLLM needs --enable-sleep-mode , both of which the supervisor and bench add. On SGLang, mem fraction static and max total num tokens the KV pool size , and raising the max running requests ceiling. The engine drains, unmaps the old KV pool and CUDA graphs through its own memory saver, rebuilds pools, attention backends and graphs at the new size, and rewires every scheduler component to the new objects. The 29 GB of weights never move. Needs the server launched with --enable-memory-saver , which the supervisor and bench add for SGLang. The supervisor tries this path before a relaunch. On vLLM, gpu memory utilization the KV pool size , co-changing max num seqs in the same rebuild. Measured, three consecutive resizes each followed by a served generation. SGLang on RTX PRO 6000: 31 to 32 s call to first token with prefill CUDA graphs on 29 s of that is capturing them , 2.0 to 2.4 s with them off --disable-prefill-cuda-graph , freeing under 0.6 s. vLLM on H100: 8.3 to 10.1 s call to first token, freeing 35.7 GiB in ~0.2 s. A relaunch on the same GPU is 56 to 86 s warm, 141 to 391 s cold. These are existing engine flags baked into GPU memory or graph capture at boot. trimtab does not make them hot. The supervisor knows which they are and, when a desired version changes one, drains, stops the engine, relaunches with the new flag, waits for health, and re-applies the hot knobs. A cold knob missing from its table is refused, not guessed. | engine | cold knobs the supervisor relaunches with | |---|---| | sglang | tp size, ep size, quantization, kv cache dtype, cuda graph max bs, attention backend, page size, speculative algorithm, speculative draft model path | | vllm | tensor parallel size, quantization, kv cache dtype, max model len, cuda graph sizes, speculative config | A redeploy is image pull, weight download, pod scheduling, and engine boot. The supervisor's relaunch skips the first three and pays the boot against weights already on local disk. Measured below, that boot took 55 to 281 seconds. These knobs change the weights' own layout parallelism, quantization or the model itself, so the pool-only warm path cannot cover them. We started a real SGLang or vLLM server on one GPU, kept it busy with 32 concurrent generation requests, and changed its concurrency cap twenty times while it ran. The server acknowledged each change in about 15 ms. On SGLang the new cap was in force in under a tenth of a second. Not one request failed. A restart on the same machines takes one to seven minutes. The bench. One GPU, Qwen/Qwen3.8-27B-FP8. 32 threads keep about 32 generation requests in flight. The bench sets the cap to 8, then back to its boot value, twenty times, two seconds apart. For each change it records the time until the server acknowledges the command API and the time until the scheduler reports the new cap as in force effect . Generation requests that fail during the run count as dropped. p50 and p95 are over the twenty changes. | GPU | engine | boot cap | changes ok | API p50 ms | API p95 ms | effect p50 ms | effect p95 ms | requests completed | dropped | |---|---|---|---|---|---|---|---|---|---| | H100 80GB | sglang 0.5.18 | 24 | 20/20 | 16 | 50 | 87 | 126 | 489 | 0 | | RTX PRO 6000 Blackwell | sglang 0.5.18 | 33 | 20/20 | 19 | 32 | 79 | 100 | 429 | 0 | | B200 | sglang 0.5.18 | 80 | 20/20 | 17 | 48 | 78 | 124 | 806 | 0 | | H100 80GB | vllm 0.28.0 | 256 | 20/20 | 17 | 80 | 347 | 1296 | 531 | 0 | | RTX PRO 6000 Blackwell | vllm 0.28.0 | 256 | 20/20 | 20 | 40 | 1304 | 2664 | 444 | 0 | | B200 | vllm 0.28.0 | 256 | 20/20 | 12 | 18 | 649 | 1529 | 947 | 0 | vLLM's effect number is larger because vLLM asserts on every step that no more requests are running than the cap allows. A lower cap cannot fully apply until enough in-flight requests finish. trimtab stops new admissions at once and tightens the enforced cap as requests complete. The bench measures the enforced value, so the drain is in the number. Raising the cap is immediate on both engines. SGLang applies a lower cap immediately and lets in-flight requests finish above it. What a restart costs on the same machines. Cold disk is the first boot after downloading weights. Warm cache is the second boot of the same weights, which is what the supervisor pays for a cold knob. | GPU | engine | boot, cold disk | boot, warm cache | |---|---|---|---| | H100 80GB | sglang | 147 s | 146 s | | RTX PRO 6000 Blackwell | sglang | 141 s | 56 s | | B200 | sglang | 428 s | 281 s | | H100 80GB | vllm | 321 s | 75 s | | RTX PRO 6000 Blackwell | vllm | 60 s | 55 s | | B200 | vllm | 166 s | 86 s | Per-change rows and the HTTP status of every control call are in bench/results . docs/results.md is generated from those files. docs/environment-notes.md records the environment problems the runs hit image entrypoints, driver floors, SM120 attention, hybrid model caps and how the scripts handle each. The patches are not upstream yet, so you apply them to the engine installed in your container. The patcher edits the installed tree in place, verifies every anchor, writes a .trimtab-orig backup, and refuses if the version differs. SGLang, end to end. pip install -e /path/to/trimtab python3 engine/sglang/apply patch.py --check first to dry-run python3 -m sglang.launch server --model-path