{"slug": "inference-optimisation-skill-hill-climb-local-llm-serving-with-an-agent", "title": "Inference Optimisation Skill: Hill-Climb Local LLM Serving With an Agent", "summary": "Sybil Solutions published the inference-optimisation-skill repository in November 2026, packaging a measured LLM serving hill-climbing workflow into a portable agent skill that has drawn about 1,700 GitHub stars. The skill targets decode, prefill, speculative decoding, KV cache size, startup and TTFT on any hardware and quantization under a memory cap, and its README reports gains including 43.6 to 53 tok/s decode on an Intel Arc B390 laptop and a 4× H200 NVL DeepSeek-V4.1-Flash setup that went from crashing to 229.1 tok/s decode. The workflow requires candidates to beat both neighboring control runs with at least 6 samples and no metric regressing more than 3% before passing long-context needle and full-KV soak gates.", "body_md": "**The [inference-optimisation-skill](https://github.com/sybil-solutions/inference-optimisation-skill?ref=explainx) repository turns months of measured LLM serving hill-climbs into a portable agent workflow.** Sybil Solutions published it in **November 2026** with about **1,700 GitHub stars**. The skill targets **decode, prefill, speculative decoding, KV cache size, startup, and TTFT** on **any hardware and quantization**, under a **memory cap**, with **long-context stability gates** before you call a build production-ready.\n\n## TL;DR\n\n| Piece | Role | \n|---|---|\n| `SKILL.md` | Procedure: inventory, ceilings, one-lever loop, stop rules | \n| `references/` | ceilings.md, levers.md, gotchas.md, publishing.md | \n| `scripts/bench.py` | OpenAI-compatible harness; writes `ledger.jsonl` | \n| `scripts/ceiling.py` | Theoretical tok/s caps from hardware JSON | \n| `scripts/queue.sh` | Interleaved control/candidate/control runs | \n| Install path | `~/.claude/skills/inference-optimisation` for Claude Code | \n\n## Why another \"make LLM fast\" repo matters\n\nMost speed guides list **flags**. This repo argues **noise is ±5–10%**, so single-run wins are often drift. The authors encoded what worked on **Intel Arc B390**, **Arc Pro B70**, **RTX 3090**, and **4× H200 NVL**, including **failures** in PLAN logs.\n\nThat pairs with industry moves toward **agent-driven tuning**, such as [Together Compiler](https://explainx.ai/blog/together-compiler-agents-optimize-inference-1111-tps-minimax-1205-tps-deepseek-claimed-vs-verified-2026) and [Microsoft MAI hill-climbing in Copilot Excel](https://explainx.ai/blog/microsoft-mai-hill-climbing-copilot-excel-nadella-july-2026). Here the agent is **yours**, running against **your** rack or laptop.\n\n## Results the README publishes\n\nKV cache slots with one highlighted token stored across decode steps\n\nThe README table is the credibility anchor. Selected rows:\n\n| Hardware | Stack | Before → after (README) | \n|---|---|---|\n| Arc B390 laptop | Ornith-1.5-35B EXL3, SGLang XPU + MTP | 43.6 → 53 tok/s decode, 1 user | \n| Arc B390 | Same | Prefill 256K: 297 → 628 tok/s | \n| Arc Pro B70 | Qwen3.8-27B, vLLM + custom XPU kernels | 91.2 tok/s decode at 1 user | \n| RTX 3090 | Qwen3.6-35B-A3B EXL3 + MTP | 251.7 / 357.9 tok/s prose/code | \n| 4× H200 NVL | DeepSeek-V4.1-Flash, vLLM + speculative decoding | Crashed → 229.1 tok/s decode, 17,546 prefill | \n\nEach line points to **local-ai-registry** pull requests or **trellis-serve** docs. explainx.ai does not re-run those benches; treat numbers as **author-measured** until you replicate on your silicon.\n\nThe **B390** example includes a **ceiling calculation**: ~100 tok/s decode for one user needs **two useful speculative drafts**, but the model accepts ~14% at depth 2, so **practical ceiling ~88–97 tok/s**. Hitting **~55% of physics** is labeled honest progress, not failure.\n\n## The workflow loop\n\nSmall draft token running ahead of verification on parallel rails\n\nThe ASCII diagram in the README flows:\n\n1. **Measure hardware** →`THEORY.md` bandwidth and FLOPs per token.\n2. **Baseline profile** → pick the lever with largest ceiling gap × likelihood.\n3. **queue.sh** runs**serialized** candidate vs**neighbouring controls** with memory watchdog and**≥6 samples** .\n4. Accept only if beat **both** controls and**no metric regresses >3%** .\n5. **Long-context needle gate** and**concurrent full-KV soak** .\n6. **Pinned clean build** and**six acceptance gates** before publish.\n\nThat is stricter than tweeting a **GGUF quant** swap. It matches how [Together AI and Ollama prefix-cache guidance](https://explainx.ai/blog/together-ai-ollama-prefix-cache-long-horizon-agents-2026) thinks about **stable prompts** for caches.\n\n## Scripts builders actually run\n\n### bench.py\n\nHits any **OpenAI-compatible** server for:\n\n- Startup and short-prompt **TTFT**\n- **c1** decode median and**cN** aggregate decode\n- Cold prefill at multiple lengths\n- **Needle retrieval** in long context\n- Peak memory with cap watchdog\n\nToken counts come from **server usage stats**, not streamed chunks, avoiding a common benchmarking trap the README calls out.\n\n### ceiling.py\n\nFeed a JSON like `examples/ceiling-ornith-b390.json` to print **decode, prefill, TTFT, startup, and KV maxima**. Agents are instructed to run this **before** chasing arbitrary tok/s targets.\n\n### queue.sh\n\nRuns **control / candidate / control** interleaves so thermal drift and background jobs do not fake wins.\n\n## Install for Claude Code and other agents\n\n```\ngit clone https://github.com/sybil-solutions/inference-optimisation-skill\nmkdir -p ~/.claude/skills\ncp -r inference-optimisation-skill/skills/inference-optimisation ~/.claude/skills/\n```\n\nThen prompt examples from the README:\n\n- \"hill-climb decode and prefill for the model on :8000 under a 40 GiB cap\"\n- \"what's the theoretical max tok/s for this model on this machine?\"\n\nFor non-Claude agents, copy `skills/inference-optimisation/SKILL.md` or paste `PROMPT.md`.\n\nQuick start without an agent:\n\n```\nS=skills/inference-optimisation\npython3 $S/scripts/ceiling.py $S/examples/ceiling-ornith-b390.json\npython3 $S/scripts/bench.py run configs/base.json --long 131072\n```\n\n## Principles that survived real climbs\n\nThe README highlights ideas that generalize beyond Sybil's stacks:\n\n| Principle | Why it matters | \n|---|---|\n| Ceilings first | Know if 53 tok/s is great or hopeless | \n| Interleaved controls | Kill ±10% noise | \n| Real tokenizer lengths | Avoid fake prefill wins | \n| Unified memory accounting | cgroups miss GPU residency on APUs | \n| Spec decode cost per step | Acceptance rate alone misleads | \n| Log failures | Most value is what did not work | \n\nGotchas for **[vLLM](https://github.com/vllm-project/vllm?ref=explainx), [SGLang](https://github.com/sgl-project/sglang?ref=explainx), llama.cpp, ExLlama** and **Intel, NVIDIA, Apple, AMD** sit in `references/gotchas.md`.\n\n## How this compares to cloud-only tuning\n\nIf you rent **Together** or **OpenRouter**, you optimize **routing and batching**, not **kernel levers**. This skill is for teams running **[Qwen on Together pricing](https://explainx.ai/blog/qwen4-max-2-4-trillion-moe-closed-model-together-ai-pricing-1m-context-november-2026)** or **self-hosted vLLM** on owned GPUs.\n\nFor **KV cache concepts**, start with [what is KV cache](https://explainx.ai/blog/what-is-kv-cache-llm-inference-explained-guide-2026). For **Rubin vs GB300 marketing numbers**, see [vLLM Rubin claimed vs verified](https://explainx.ai/blog/vllm-rubin-93x-throughput-claimed-vs-verified-november-2026).\n\n## Agent skill ecosystem context\n\nOne highlighted skill card pulled from a shelf of agent skill files\n\nNvidia's **[SkillEvaluator guide](https://explainx.ai/blog/nvidia-skillevaluator-agent-skills-three-tier-cli-guide-2026)** pushes **CI gates on agent skills**. This skill is the opposite flavor: a **domain skill** with **Python harnesses** instead of only markdown instructions.\n\n[Poki's iPhone GUI agent skill](https://explainx.ai/blog/poki-open-source-iphone-gui-agent-claude-code-install-2026) shows the same pattern: **ship SKILL.md + install path + real benchmarks**.\n\n## When not to use this skill\n\nSkip it if you only call **ChatGPT APIs** from a laptop. Skip it if you lack **root or container access** to inference servers. Skip it if you cannot afford **multi-hour** soak tests.\n\nUse it when **on-prem Qwen, DeepSeek, or MoE models** already serve agents and **p95 latency** blocks rollout.\n\n## Replication checklist\n\n1. Copy **base.json** config with your`base_url` and model id.\n2. Run **ceiling.py** with honest bandwidth/FLOPs JSON.\n3. Capture **baseline ledger** before any flag change.\n4. Change **one lever** documented in`levers.md` .\n5. Pass **needle + soak** before tweeting results.\n\n## Inside references/levers.md\n\nThe skill splits levers into **general** (apply on any stack) and **stack-specific** (vLLM, SGLang, llama.cpp quirks). Agents are told to load **levers.md** only when a ceiling gap points at a category:\n\n| Lever family | Typical symptom | README mindset | \n|---|---|---|\n| Speculative decoding | Good acceptance, slow step time | Optimize **per-step cost** , prune draft vocab | \n| KV cache layout | OOM at long context | Shrink bytes/token before chasing quant swaps | \n| Prefill batching | Great decode, slow first token | Separate prefill tuning from decode sweeps | \n| MTP / multi-token prediction | Unstable at 256K | Run needle gate before calling win | \n| Engine startup | Minutes to first request | Track startup separately in ledger | \n\nThe **Arc B70** row where **SGLang beat vLLM** on the same card (**82.6 vs 75.7 tok/s** decode) is a reminder that **engine choice** is itself a lever, not a one-time decision.\n\n## Unified memory on laptops matters\n\nThe README warns that **cgroup memory** on Intel **unified memory** laptops does not see full GPU residency. `mem.sh` exists because **40 GiB cap** violations only show up when driver footprint counts.\n\nIf you run **local Qwen** on a **64 GB LPDDR5X** machine, pair this skill with [Nvidia long-context compaction paper context](https://explainx.ai/blog/nvidia-you-dont-need-long-context-compaction-sssm-arxiv-november-2026) so you do not optimize decode while ignoring **context eviction** policy.\n\n## Publishing.md and the six acceptance gates\n\nBefore Sybil publishes a **local-ai-registry** recipe, the skill demands:\n\n1. **Pinned engine versions** and model hashes.\n2. **Clean rebuild** proving the binary matches the best ledger row.\n3. **Needle test** at declared max context.\n4. **Concurrent users** at declared concurrency without collapse.\n5. **Regression guard** on neighbour controls.\n6. **Public results card** linking raw`ledger.jsonl` .\n\nThat discipline is why the README can cite **Registry Lab gates** moving from **28.6 → 41.0 decode** on B390 without hand-waving.\n\n## Failure logging as a product feature\n\n`PLAN.md + ledger.jsonl` is where rejected configs live. Agents trained only on success posts miss that **most hill-climbs are negative results**. The skill explicitly tells the agent to **write down failures** because the next lever choice depends on knowing **which knobs already failed**.\n\nFor teams building **internal agent platforms**, copy that pattern into your **MLOps tickets**: attach **failed bench JSON** the same way you attach **failed unit tests**.\n\n## Pairing with Claude Code workshops\n\nIf your org is booking [Claude Code workshops](https://explainx.ai/blog/what-is-claude-ai-workshop-formats-pricing-how-to-choose-2026), add a **local inference lab** module: install this skill, point at a staging vLLM, and require **ceiling.py output** in the PR description before merging server flags.\n\n## Ollama and Apple Silicon paths\n\nWhile README highlights **Arc and CUDA**, `gotchas.md` also covers **Apple and AMD**. [Perplexity Lily Apple Silicon engine](https://explainx.ai/blog/perplexity-lily-apple-silicon-inference-engine-2026) is a different approach (custom engine), but **llama.cpp levers** in the skill still apply when you serve through **OpenAI-compatible adapters**.\n\n## SGLang vs vLLM choice in 2026\n\nYour cloud may ship **Dynamo-wrapped SGLang** while your laptop runs **stock vLLM**. The skill refuses to treat them as interchangeable: **gotchas.md** is per engine.\n\nWhen reading [SemiAnalysis-style throughput debates](https://explainx.ai/blog/vllm-rubin-93x-throughput-claimed-vs-verified-november-2026), remember those are **rack-scale** stories. This skill is **single-node honest benches** you can rerun tonight.\n\n## When agents mis-measure\n\nCommon traps the skill blocks:\n\n- Counting **stream chunks** instead of tokenizer tokens.\n- Using **short prompts** that hide prefill quadratic cost.\n- Ignoring **battery power** on laptops (bench flags runs on AC).\n- Calling a win when **looping output** detector fires.\n\nRunning `python3 bench.py show` after every queue should be non-negotiable.\n\n## Enterprise angle: why Sybil open-sourced it\n\nSybil ties the skill to **public registry recipes** with PR links. Open-sourcing the methodology lets other hosts replicate **before/after tables** without sharing proprietary kernels.\n\nThat mirrors [AWS's agent skill for distributed KV cache](https://explainx.ai/blog/aws-distributed-kv-caching-agent-skill-elasticache-november-2026): vendors publish **playbooks**, not just **sliders**.\n\n## Quick FAQ for searchers asking \"is local LLM worth it\"\n\nLocal LLM serving is worth the ops burden when **any** of these hold: **PII cannot leave VPC**, **inference spend exceeds one FTE**, **p95 latency blocks an agent product**, or **you already own GPUs depreciating on the balance sheet**.\n\nThis skill does not pick **which model**; it makes a chosen model **faster without lying about numbers**.\n\n## Related reading\n\nSources: [inference-optimisation-skill README](https://github.com/sybil-solutions/inference-optimisation-skill?ref=explainx) (accessed November 22, 2026).", "url": "https://wpnews.pro/news/inference-optimisation-skill-hill-climb-local-llm-serving-with-an-agent", "canonical_source": "https://explainx.ai/blog/inference-optimisation-skill-sybil-hill-climb-local-llm-november-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog", "published_at": "2026-11-22 00:00:00+00:00", "updated_at": "2026-11-22 02:42:03.570445+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-agents", "large-language-models", "ai-tools", "mlops"], "entities": ["Sybil Solutions", "inference-optimisation-skill", "Intel Arc B390", "Intel Arc Pro B70", "RTX 3090", "Nvidia H200 NVL", "DeepSeek-V4.1-Flash", "Claude Code"], "also_reported_by": [], "takeaway": "Accept a serving change only if it beats both neighboring control runs across at least 6 samples with no metric regressing more than 3%, since run-to-run noise is ±5–10% and single-run wins are often drift.", "recommendation": {"advice": "Interleave candidate runs with control runs via queue.sh, require at least 6 samples and no metric regression above 3%, then pass long-context needle and full-KV soak gates before pinning a clean build.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/inference-optimisation-skill-hill-climb-local-llm-serving-with-an-agent", "markdown": "https://wpnews.pro/news/inference-optimisation-skill-hill-climb-local-llm-serving-with-an-agent.md", "text": "https://wpnews.pro/news/inference-optimisation-skill-hill-climb-local-llm-serving-with-an-agent.txt", "jsonld": "https://wpnews.pro/news/inference-optimisation-skill-hill-climb-local-llm-serving-with-an-agent.jsonld"}}