Inference Optimisation Skill: Hill-Climb Local LLM Serving With an Agent Sybil Solutions published the inference-optimisation-skill repository in November 2026, packaging a measured LLM serving hill-climbing workflow into a portable agent skill that has drawn about 1,700 GitHub stars. The skill targets decode, prefill, speculative decoding, KV cache size, startup and TTFT on any hardware and quantization under a memory cap, and its README reports gains including 43.6 to 53 tok/s decode on an Intel Arc B390 laptop and a 4× H200 NVL DeepSeek-V4.1-Flash setup that went from crashing to 229.1 tok/s decode. The workflow requires candidates to beat both neighboring control runs with at least 6 samples and no metric regressing more than 3% before passing long-context needle and full-KV soak gates. The inference-optimisation-skill https://github.com/sybil-solutions/inference-optimisation-skill?ref=explainx repository turns months of measured LLM serving hill-climbs into a portable agent workflow. Sybil Solutions published it in November 2026 with about 1,700 GitHub stars . The skill targets decode, prefill, speculative decoding, KV cache size, startup, and TTFT on any hardware and quantization , under a memory cap , with long-context stability gates before you call a build production-ready. TL;DR | Piece | Role | |---|---| | SKILL.md | Procedure: inventory, ceilings, one-lever loop, stop rules | | references/ | ceilings.md, levers.md, gotchas.md, publishing.md | | scripts/bench.py | OpenAI-compatible harness; writes ledger.jsonl | | scripts/ceiling.py | Theoretical tok/s caps from hardware JSON | | scripts/queue.sh | Interleaved control/candidate/control runs | | Install path | ~/.claude/skills/inference-optimisation for Claude Code | Why another "make LLM fast" repo matters Most speed guides list flags . This repo argues noise is ±5–10% , so single-run wins are often drift. The authors encoded what worked on Intel Arc B390 , Arc Pro B70 , RTX 3090 , and 4× H200 NVL , including failures in PLAN logs. That pairs with industry moves toward agent-driven tuning , such as Together Compiler https://explainx.ai/blog/together-compiler-agents-optimize-inference-1111-tps-minimax-1205-tps-deepseek-claimed-vs-verified-2026 and Microsoft MAI hill-climbing in Copilot Excel https://explainx.ai/blog/microsoft-mai-hill-climbing-copilot-excel-nadella-july-2026 . Here the agent is yours , running against your rack or laptop. Results the README publishes KV cache slots with one highlighted token stored across decode steps The README table is the credibility anchor. Selected rows: | Hardware | Stack | Before → after README | |---|---|---| | Arc B390 laptop | Ornith-1.5-35B EXL3, SGLang XPU + MTP | 43.6 → 53 tok/s decode, 1 user | | Arc B390 | Same | Prefill 256K: 297 → 628 tok/s | | Arc Pro B70 | Qwen3.8-27B, vLLM + custom XPU kernels | 91.2 tok/s decode at 1 user | | RTX 3090 | Qwen3.6-35B-A3B EXL3 + MTP | 251.7 / 357.9 tok/s prose/code | | 4× H200 NVL | DeepSeek-V4.1-Flash, vLLM + speculative decoding | Crashed → 229.1 tok/s decode, 17,546 prefill | Each line points to local-ai-registry pull requests or trellis-serve docs. explainx.ai does not re-run those benches; treat numbers as author-measured until you replicate on your silicon. The B390 example includes a ceiling calculation : ~100 tok/s decode for one user needs two useful speculative drafts , but the model accepts ~14% at depth 2, so practical ceiling ~88–97 tok/s . Hitting ~55% of physics is labeled honest progress, not failure. The workflow loop Small draft token running ahead of verification on parallel rails The ASCII diagram in the README flows: 1. Measure hardware → THEORY.md bandwidth and FLOPs per token. 2. Baseline profile → pick the lever with largest ceiling gap × likelihood. 3. queue.sh runs serialized candidate vs neighbouring controls with memory watchdog and ≥6 samples . 4. Accept only if beat both controls and no metric regresses 3% . 5. Long-context needle gate and concurrent full-KV soak . 6. Pinned clean build and six acceptance gates before publish. That is stricter than tweeting a GGUF quant swap. It matches how Together AI and Ollama prefix-cache guidance https://explainx.ai/blog/together-ai-ollama-prefix-cache-long-horizon-agents-2026 thinks about stable prompts for caches. Scripts builders actually run bench.py Hits any OpenAI-compatible server for: - Startup and short-prompt TTFT - c1 decode median and cN aggregate decode - Cold prefill at multiple lengths - Needle retrieval in long context - Peak memory with cap watchdog Token counts come from server usage stats , not streamed chunks, avoiding a common benchmarking trap the README calls out. ceiling.py Feed a JSON like examples/ceiling-ornith-b390.json to print decode, prefill, TTFT, startup, and KV maxima . Agents are instructed to run this before chasing arbitrary tok/s targets. queue.sh Runs control / candidate / control interleaves so thermal drift and background jobs do not fake wins. Install for Claude Code and other agents git clone https://github.com/sybil-solutions/inference-optimisation-skill mkdir -p ~/.claude/skills cp -r inference-optimisation-skill/skills/inference-optimisation ~/.claude/skills/ Then prompt examples from the README: - "hill-climb decode and prefill for the model on :8000 under a 40 GiB cap" - "what's the theoretical max tok/s for this model on this machine?" For non-Claude agents, copy skills/inference-optimisation/SKILL.md or paste PROMPT.md . Quick start without an agent: S=skills/inference-optimisation python3 $S/scripts/ceiling.py $S/examples/ceiling-ornith-b390.json python3 $S/scripts/bench.py run configs/base.json --long 131072 Principles that survived real climbs The README highlights ideas that generalize beyond Sybil's stacks: | Principle | Why it matters | |---|---| | Ceilings first | Know if 53 tok/s is great or hopeless | | Interleaved controls | Kill ±10% noise | | Real tokenizer lengths | Avoid fake prefill wins | | Unified memory accounting | cgroups miss GPU residency on APUs | | Spec decode cost per step | Acceptance rate alone misleads | | Log failures | Most value is what did not work | Gotchas for vLLM https://github.com/vllm-project/vllm?ref=explainx , SGLang https://github.com/sgl-project/sglang?ref=explainx , llama.cpp, ExLlama and Intel, NVIDIA, Apple, AMD sit in references/gotchas.md . How this compares to cloud-only tuning If you rent Together or OpenRouter , you optimize routing and batching , not kernel levers . This skill is for teams running Qwen on Together pricing https://explainx.ai/blog/qwen4-max-2-4-trillion-moe-closed-model-together-ai-pricing-1m-context-november-2026 or self-hosted vLLM on owned GPUs. For KV cache concepts , start with what is KV cache https://explainx.ai/blog/what-is-kv-cache-llm-inference-explained-guide-2026 . For Rubin vs GB300 marketing numbers , see vLLM Rubin claimed vs verified https://explainx.ai/blog/vllm-rubin-93x-throughput-claimed-vs-verified-november-2026 . Agent skill ecosystem context One highlighted skill card pulled from a shelf of agent skill files Nvidia's SkillEvaluator guide https://explainx.ai/blog/nvidia-skillevaluator-agent-skills-three-tier-cli-guide-2026 pushes CI gates on agent skills . This skill is the opposite flavor: a domain skill with Python harnesses instead of only markdown instructions. Poki's iPhone GUI agent skill https://explainx.ai/blog/poki-open-source-iphone-gui-agent-claude-code-install-2026 shows the same pattern: ship SKILL.md + install path + real benchmarks . When not to use this skill Skip it if you only call ChatGPT APIs from a laptop. Skip it if you lack root or container access to inference servers. Skip it if you cannot afford multi-hour soak tests. Use it when on-prem Qwen, DeepSeek, or MoE models already serve agents and p95 latency blocks rollout. Replication checklist 1. Copy base.json config with your base url and model id. 2. Run ceiling.py with honest bandwidth/FLOPs JSON. 3. Capture baseline ledger before any flag change. 4. Change one lever documented in levers.md . 5. Pass needle + soak before tweeting results. Inside references/levers.md The skill splits levers into general apply on any stack and stack-specific vLLM, SGLang, llama.cpp quirks . Agents are told to load levers.md only when a ceiling gap points at a category: | Lever family | Typical symptom | README mindset | |---|---|---| | Speculative decoding | Good acceptance, slow step time | Optimize per-step cost , prune draft vocab | | KV cache layout | OOM at long context | Shrink bytes/token before chasing quant swaps | | Prefill batching | Great decode, slow first token | Separate prefill tuning from decode sweeps | | MTP / multi-token prediction | Unstable at 256K | Run needle gate before calling win | | Engine startup | Minutes to first request | Track startup separately in ledger | The Arc B70 row where SGLang beat vLLM on the same card 82.6 vs 75.7 tok/s decode is a reminder that engine choice is itself a lever, not a one-time decision. Unified memory on laptops matters The README warns that cgroup memory on Intel unified memory laptops does not see full GPU residency. mem.sh exists because 40 GiB cap violations only show up when driver footprint counts. If you run local Qwen on a 64 GB LPDDR5X machine, pair this skill with Nvidia long-context compaction paper context https://explainx.ai/blog/nvidia-you-dont-need-long-context-compaction-sssm-arxiv-november-2026 so you do not optimize decode while ignoring context eviction policy. Publishing.md and the six acceptance gates Before Sybil publishes a local-ai-registry recipe, the skill demands: 1. Pinned engine versions and model hashes. 2. Clean rebuild proving the binary matches the best ledger row. 3. Needle test at declared max context. 4. Concurrent users at declared concurrency without collapse. 5. Regression guard on neighbour controls. 6. Public results card linking raw ledger.jsonl . That discipline is why the README can cite Registry Lab gates moving from 28.6 → 41.0 decode on B390 without hand-waving. Failure logging as a product feature PLAN.md + ledger.jsonl is where rejected configs live. Agents trained only on success posts miss that most hill-climbs are negative results . The skill explicitly tells the agent to write down failures because the next lever choice depends on knowing which knobs already failed . For teams building internal agent platforms , copy that pattern into your MLOps tickets : attach failed bench JSON the same way you attach failed unit tests . Pairing with Claude Code workshops If your org is booking Claude Code workshops https://explainx.ai/blog/what-is-claude-ai-workshop-formats-pricing-how-to-choose-2026 , add a local inference lab module: install this skill, point at a staging vLLM, and require ceiling.py output in the PR description before merging server flags. Ollama and Apple Silicon paths While README highlights Arc and CUDA , gotchas.md also covers Apple and AMD . Perplexity Lily Apple Silicon engine https://explainx.ai/blog/perplexity-lily-apple-silicon-inference-engine-2026 is a different approach custom engine , but llama.cpp levers in the skill still apply when you serve through OpenAI-compatible adapters . SGLang vs vLLM choice in 2026 Your cloud may ship Dynamo-wrapped SGLang while your laptop runs stock vLLM . The skill refuses to treat them as interchangeable: gotchas.md is per engine. When reading SemiAnalysis-style throughput debates https://explainx.ai/blog/vllm-rubin-93x-throughput-claimed-vs-verified-november-2026 , remember those are rack-scale stories. This skill is single-node honest benches you can rerun tonight. When agents mis-measure Common traps the skill blocks: - Counting stream chunks instead of tokenizer tokens. - Using short prompts that hide prefill quadratic cost. - Ignoring battery power on laptops bench flags runs on AC . - Calling a win when looping output detector fires. Running python3 bench.py show after every queue should be non-negotiable. Enterprise angle: why Sybil open-sourced it Sybil ties the skill to public registry recipes with PR links. Open-sourcing the methodology lets other hosts replicate before/after tables without sharing proprietary kernels. That mirrors AWS's agent skill for distributed KV cache https://explainx.ai/blog/aws-distributed-kv-caching-agent-skill-elasticache-november-2026 : vendors publish playbooks , not just sliders . Quick FAQ for searchers asking "is local LLM worth it" Local LLM serving is worth the ops burden when any of these hold: PII cannot leave VPC , inference spend exceeds one FTE , p95 latency blocks an agent product , or you already own GPUs depreciating on the balance sheet . This skill does not pick which model ; it makes a chosen model faster without lying about numbers . Related reading Sources: inference-optimisation-skill README https://github.com/sybil-solutions/inference-optimisation-skill?ref=explainx accessed November 22, 2026 .