cd /news/ai-infrastructure/automated-k8s-hill-climbing-harness-… Β· home β€Ί topics β€Ί ai-infrastructure β€Ί article
[ARTICLE Β· art-174003] src=github.com β†— pub= topic=ai-infrastructure verified=true sentiment=↑ positive

Automated K8s hill-climbing harness and tuning vLLM and SGLang on a single A100

A new open-source harness automates Kubernetes-based coordinate-ascent hill-climbing to tune vLLM and SGLang inference engines, running 337 trials on Nanbeige4.3-33B on a single NVIDIA A100 80GB PCIe GPU. The harness isolates each trial in its own pod on a dedicated GPU node, calibrates run-to-run noise at 3Οƒ, requires paired confirmation re-runs before accepting a challenger, and gates configs on GSM8K, long-context passkey retrieval at 8k/16k/28k, and teacher-forced NLL against 3,827 BF16 reference tokens. Capacity searches target the highest Poisson arrival rate where at least 90% of requests meet TTFT ≀ 2 s and TPOT ≀ 100 ms, with steady-state steps of 420–540 s and results logged to a 432-trial SQLite ledger.

recommendation

Calibrate run-to-run noise at 3Οƒ and require a paired confirmation re-run before accepting any vLLM or SGLang config change; score capacity as the highest Poisson rate where β‰₯90% of requests meet TTFT ≀ 2 s and TPOT ≀ 100 ms.

more in playbook: Inference and self-hosting
read4 min views3 publishedNov 22, 2026
Automated K8s hill-climbing harness and tuning vLLM and SGLang on a single A100
Image: Michielbdejong (auto-discovered)

An automated Kubernetes coordinate-ascent hill-climbing harness for LLM inference engines (vLLM and SGLang), with pod-per-trial GPU isolation, 3Οƒ run-to-run noise calibration, paired confirmation re-runs, 7–9 minute steady-state capacity searches, and automated quality gates (GSM8K, long-context passkey retrieval, and teacher-forced logprob NLL against BF16).

Benchmark Results & Winning Configs (Nanbeige4.3-33B on 1Γ— NVIDIA A100 80GB PCIe): If you are looking for the 337-trial benchmark study, all 10 charts, what worked vs. what regressed, and copy-paste vLLM / SGLang serving commands, see RESULTS.md (or query the 432-trial SQLite database directly in results/ledger.sqlite).

  1. Pod-per-trial GPU isolation (harness/runner.py): - Each trial runs as an isolated Kubernetes pod pinned to a dedicated GPU node ("lane", e.g. n0 orn1 ), pairing the inference server container (vLLM orSGLang ) with a benchmark sidecar overlocalhost .
  • The pod automatically sizes /dev/shm (32 GiB base plus any configured CPU KV offload region), captures startup and tail server logs, samples GPU power/clock/utilization (nvidia-smi ) and per-process CPU usage, and writes structured trial metrics intoresults/ledger.sqlite .
  1. Each trial runs as an isolated Kubernetes pod pinned to a dedicated GPU node ("lane", e.g.
  2. Versioned workload plans (harness/plans.py) and sidecar drivers: - Single-user decode ( t1n, t1v): vllm bench serve at concurrency 1 on disjoint SPEC-Bench prompt slices with natural stopping (--Temperature 0.6 --top-p 0.95 , thinking on) plus an 8k-token prefill TTFT guard.
  • Poisson capacity searches (t2s, t2c, t2ss@r, t2css@r): Geometric rate ladder followed by bisection to find the highest Poisson arrival rate where β‰₯90% of requests meet bothTTFT ≀ 2 s andTPOT ≀ 100 ms . Steady-state plans (t2ss@r ,t2css@r ) run 420–540 s steps and score only requests arriving after the initial 2–3 minute ramp-up and at least 1–1.5 minutes before arrivals end.
  • Multi-turn agentic sessions (t3, harness/t3_driver.py): 48 concurrent 6-turn sessions arriving at 0.2 sessions/s with a shared 3,000-token system prompt, a new 1,200-token document per turn, and 15 s think time between turns, scored by p90 TTFT across all 288 turns.
  • Quality gate (quality2, harness/quality_driver.py): 200 GSM8K problems (thinking on), 9 passkeys buried at 8k / 16k / 28k context, and teacher-forced negative log-likelihood (NLL) + top-1 token agreement over 3,827 greedy BF16 reference tokens.
  1. Noise-calibrated coordinate ascent (harness/climber.py): - Score climbs ( t1n, t3): A challenger must beat the incumbent by max(3σ, floor) on the same node (whereσ is estimated from all repeated runs of the incumbent) and then repeat that win on a fresh confirmation run on the same node.
  • Capacity climbs (t2, t2c): Each challenger is probed for one step at 1.05Γ— the incumbent's capacity; if β‰₯90% of requests meet the SLO and a fresh confirmation run agrees, it runs a full rate search to become the new incumbent.

  • Supports coupled moves (e.g. speculative method Γ— draft length), guided kicks, and leave-one-out ablations (harness/campaign.py ).

  • RESULTS.md β€” Full benchmark report, 10 charts (assets/*.png ), ablation table, 3 benchmark bugs fixed, and winningvLLM /SGLang commands forNanbeige4.3-33B on a single A100 80GB PCIe.

  • harness/ β€” Core Python package:

    • space.py : Search space (22 vLLM and 21 SGLang settings across layers L0–L6), validity rules, server flag rendering, canonical config hashing, and quality-gate numerics classes.
    • plans.py : Workload definitions and parameterized probe / steady-state rate-search builders.
    • runner.py : Kubernetes pod lifecycle, log capture, and GPU/CPU telemetry collection.
    • bench_driver.py ,quality_driver.py ,t3_driver.py : In-pod benchmark sidecar drivers.
    • climber.py ,campaign.py ,score.py ,ledger.py ,cli.py ,settings.py : Coordinate-ascent climber, multi-stage campaign runner, scorer, SQLite ledger, and CLI.
  • campaigns/ β€” Declarative campaign JSONs ( main-v3.json ,followup-v3.json ,main.json ).

  • results/ledger.sqlite β€” Complete 432-trial SQLite database from the A100 campaigns.

  • assets/ β€” Charts generated directly from results/ledger.sqlite viatools/post_charts.py .

  • deploy/ β€” Kubernetes manifest templates ( 10-nvme-prep.yaml ,20-nvidia-device-plugin.tmpl.yaml ,30-download-models.tmpl.yaml ,40-controller.tmpl.yaml ,50-build-w8a8.tmpl.yaml ,models.lock ).

  • tools/ β€” Operational scripts ( deploy.sh ,pull.sh ,stage_sharegpt.sh ,post_charts.py ,publish_hf_w8a8.sh ,render.py ,check_leaks.py ,install_hooks.sh ).

  • tests/ β€” Offline simulator and unit tests ( sim_climb.py ,sim_campaign.py ,test_bench_driver.py ,test_env.py ,test_resolved.py ).

  • docs/ β€” Background research, book review synthesis, and engineering decision logs:

python3 -m tests.sim_climb
python3 -m tests.sim_campaign
python3 -m tests.test_bench_driver
python3 -m tests.test_env
python3 -m tests.test_resolved
python3 -m harness.cli status --campaign main-v3
python3 -m harness.cli report --campaign main-v3:climb-t1n-vllm
python3 tools/post_charts.py results/ledger.sqlite assets

Environment-specific identifiers (kube context, node names, container registry, subscription) live only in a gitignored .env and are blocked from commits by tools/check_leaks.py:

cp .env.example .env          # fill in your cluster context, lane nodes, and registry
tools/install_hooks.sh        # install pre-commit and pre-push leak-check hooks
tools/deploy.sh main-v3       # build controller image and launch campaign

You can also run individual trials or climbs from the CLI:

python3 -m harness.cli run --engine vllm --set variant=int4 --plan t1n --lane n0
python3 -m harness.cli batch --file configs.json --plan t2s --campaign screen-1
python3 -m harness.cli climb --track t2 --start start.json --campaign climb-t2
── more in #ai-infrastructure 4 stories Β· sorted by recency
── more on @vllm 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain β€” perfect for shipping the agent you just read about.

$git push zahid main
β†’ Live at https://your-agent.zahid.host βœ“
Get free account β†’ Pricing
from €0/mo Β· no card required
LIVE [news/automated-k8s-hill-c…] indexed:0 read:4min 2026-11-22 Β· β€”