cd /news/ai-infrastructure/inference-optimisation-skill-hill-cl… Β· home β€Ί topics β€Ί ai-infrastructure β€Ί article
[ARTICLE Β· art-174025] src=explainx.ai β†— pub= topic=ai-infrastructure verified=true sentiment=↑ positive

Inference Optimisation Skill: Hill-Climb Local LLM Serving With an Agent

Sybil Solutions published the inference-optimisation-skill repository in November 2026, packaging a measured LLM serving hill-climbing workflow into a portable agent skill that has drawn about 1,700 GitHub stars. The skill targets decode, prefill, speculative decoding, KV cache size, startup and TTFT on any hardware and quantization under a memory cap, and its README reports gains including 43.6 to 53 tok/s decode on an Intel Arc B390 laptop and a 4Γ— H200 NVL DeepSeek-V4.1-Flash setup that went from crashing to 229.1 tok/s decode. The workflow requires candidates to beat both neighboring control runs with at least 6 samples and no metric regressing more than 3% before passing long-context needle and full-KV soak gates.

recommendation

Interleave candidate runs with control runs via queue.sh, require at least 6 samples and no metric regression above 3%, then pass long-context needle and full-KV soak gates before pinning a clean build.

more in playbook: Inference and self-hosting
by read8 min views1 publishedNov 22, 2026
Inference Optimisation Skill: Hill-Climb Local LLM Serving With an Agent
Image: Explainx (auto-discovered)

The inference-optimisation-skill repository turns months of measured LLM serving hill-climbs into a portable agent workflow. Sybil Solutions published it in November 2026 with about 1,700 GitHub stars. The skill targets decode, prefill, speculative decoding, KV cache size, startup, and TTFT on any hardware and quantization, under a memory cap, with long-context stability gates before you call a build production-ready.

TL;DR #

Piece Role
SKILL.md Procedure: inventory, ceilings, one-lever loop, stop rules
references/ ceilings.md, levers.md, gotchas.md, publishing.md
scripts/bench.py OpenAI-compatible harness; writes ledger.jsonl
scripts/ceiling.py Theoretical tok/s caps from hardware JSON
scripts/queue.sh Interleaved control/candidate/control runs
Install path ~/.claude/skills/inference-optimisation for Claude Code

Why another "make LLM fast" repo matters #

Most speed guides list flags. This repo argues noise is Β±5–10%, so single-run wins are often drift. The authors encoded what worked on Intel Arc B390, Arc Pro B70, RTX 3090, and 4Γ— H200 NVL, including failures in PLAN logs.

That pairs with industry moves toward agent-driven tuning, such as Together Compiler and Microsoft MAI hill-climbing in Copilot Excel. Here the agent is yours, running against your rack or laptop.

Results the README publishes #

KV cache slots with one highlighted token stored across decode steps

The README table is the credibility anchor. Selected rows:

Hardware Stack Before β†’ after (README)
Arc B390 laptop Ornith-1.5-35B EXL3, SGLang XPU + MTP 43.6 β†’ 53 tok/s decode, 1 user
Arc B390 Same Prefill 256K: 297 β†’ 628 tok/s
Arc Pro B70 Qwen3.8-27B, vLLM + custom XPU kernels 91.2 tok/s decode at 1 user
RTX 3090 Qwen3.6-35B-A3B EXL3 + MTP 251.7 / 357.9 tok/s prose/code
4Γ— H200 NVL DeepSeek-V4.1-Flash, vLLM + speculative decoding Crashed β†’ 229.1 tok/s decode, 17,546 prefill

Each line points to local-ai-registry pull requests or trellis-serve docs. explainx.ai does not re-run those benches; treat numbers as author-measured until you replicate on your silicon.

The B390 example includes a ceiling calculation: ~100 tok/s decode for one user needs two useful speculative drafts, but the model accepts ~14% at depth 2, so practical ceiling ~88–97 tok/s. Hitting ~55% of physics is labeled honest progress, not failure.

The workflow loop #

Small draft token running ahead of verification on parallel rails

The ASCII diagram in the README flows:

  1. Measure hardware β†’THEORY.md bandwidth and FLOPs per token.
  2. Baseline profile β†’ pick the lever with largest ceiling gap Γ— likelihood.
  3. queue.sh runsserialized candidate vsneighbouring controls with memory watchdog and**β‰₯6 samples** .
  4. Accept only if beat both controls andno metric regresses >3% .
  5. Long-context needle gate andconcurrent full-KV soak .
  6. Pinned clean build andsix acceptance gates before publish.

That is stricter than tweeting a GGUF quant swap. It matches how Together AI and Ollama prefix-cache guidance thinks about stable prompts for caches.

Scripts builders actually run #

bench.py

Hits any OpenAI-compatible server for:

  • Startup and short-prompt TTFT
  • c1 decode median andcN aggregate decode
  • Cold prefill at multiple lengths
  • Needle retrieval in long context
  • Peak memory with cap watchdog

Token counts come from server usage stats, not streamed chunks, avoiding a common benchmarking trap the README calls out.

ceiling.py

Feed a JSON like examples/ceiling-ornith-b390.json to print decode, prefill, TTFT, startup, and KV maxima. Agents are instructed to run this before chasing arbitrary tok/s targets.

queue.sh

Runs control / candidate / control interleaves so thermal drift and background jobs do not fake wins.

Install for Claude Code and other agents #

git clone https://github.com/sybil-solutions/inference-optimisation-skill
mkdir -p ~/.claude/skills
cp -r inference-optimisation-skill/skills/inference-optimisation ~/.claude/skills/

Then prompt examples from the README:

  • "hill-climb decode and prefill for the model on :8000 under a 40 GiB cap"
  • "what's the theoretical max tok/s for this model on this machine?"

For non-Claude agents, copy skills/inference-optimisation/SKILL.md or paste PROMPT.md.

Quick start without an agent:

S=skills/inference-optimisation
python3 $S/scripts/ceiling.py $S/examples/ceiling-ornith-b390.json
python3 $S/scripts/bench.py run configs/base.json --long 131072

Principles that survived real climbs #

The README highlights ideas that generalize beyond Sybil's stacks:

Principle Why it matters
Ceilings first Know if 53 tok/s is great or hopeless
Interleaved controls Kill Β±10% noise
Real tokenizer lengths Avoid fake prefill wins
Unified memory accounting cgroups miss GPU residency on APUs
Spec decode cost per step Acceptance rate alone misleads
Log failures Most value is what did not work

Gotchas for vLLM, SGLang, llama.cpp, ExLlama and Intel, NVIDIA, Apple, AMD sit in references/gotchas.md.

How this compares to cloud-only tuning #

If you rent Together or OpenRouter, you optimize routing and batching, not kernel levers. This skill is for teams running Qwen on Together pricing or self-hosted vLLM on owned GPUs.

For KV cache concepts, start with what is KV cache. For Rubin vs GB300 marketing numbers, see vLLM Rubin claimed vs verified.

Agent skill ecosystem context #

One highlighted skill card pulled from a shelf of agent skill files

Nvidia's SkillEvaluator guide pushes CI gates on agent skills. This skill is the opposite flavor: a domain skill with Python harnesses instead of only markdown instructions.

Poki's iPhone GUI agent skill shows the same pattern: ship SKILL.md + install path + real benchmarks.

When not to use this skill #

Skip it if you only call ChatGPT APIs from a laptop. Skip it if you lack root or container access to inference servers. Skip it if you cannot afford multi-hour soak tests.

Use it when on-prem Qwen, DeepSeek, or MoE models already serve agents and p95 latency blocks rollout.

Replication checklist #

  1. Copy base.json config with yourbase_url and model id.
  2. Run ceiling.py with honest bandwidth/FLOPs JSON.
  3. Capture baseline ledger before any flag change.
  4. Change one lever documented inlevers.md .
  5. Pass needle + soak before tweeting results.

Inside references/levers.md #

The skill splits levers into general (apply on any stack) and stack-specific (vLLM, SGLang, llama.cpp quirks). Agents are told to load levers.md only when a ceiling gap points at a category:

Lever family Typical symptom README mindset
Speculative decoding Good acceptance, slow step time Optimize per-step cost , prune draft vocab
KV cache layout OOM at long context Shrink bytes/token before chasing quant swaps
Prefill batching Great decode, slow first token Separate prefill tuning from decode sweeps
MTP / multi-token prediction Unstable at 256K Run needle gate before calling win
Engine startup Minutes to first request Track startup separately in ledger

The Arc B70 row where SGLang beat vLLM on the same card (82.6 vs 75.7 tok/s decode) is a reminder that engine choice is itself a lever, not a one-time decision.

Unified memory on laptops matters #

The README warns that cgroup memory on Intel unified memory laptops does not see full GPU residency. mem.sh exists because 40 GiB cap violations only show up when driver footprint counts.

If you run local Qwen on a 64 GB LPDDR5X machine, pair this skill with Nvidia long-context compaction paper context so you do not optimize decode while ignoring context eviction policy.

Publishing.md and the six acceptance gates #

Before Sybil publishes a local-ai-registry recipe, the skill demands:

  1. Pinned engine versions and model hashes.
  2. Clean rebuild proving the binary matches the best ledger row.
  3. Needle test at declared max context.
  4. Concurrent users at declared concurrency without collapse.
  5. Regression guard on neighbour controls.
  6. Public results card linking rawledger.jsonl .

That discipline is why the README can cite Registry Lab gates moving from 28.6 β†’ 41.0 decode on B390 without hand-waving.

Failure logging as a product feature #

PLAN.md + ledger.jsonl is where rejected configs live. Agents trained only on success posts miss that most hill-climbs are negative results. The skill explicitly tells the agent to write down failures because the next lever choice depends on knowing which knobs already failed.

For teams building internal agent platforms, copy that pattern into your MLOps tickets: attach failed bench JSON the same way you attach failed unit tests.

Pairing with Claude Code workshops #

If your org is booking Claude Code workshops, add a local inference lab module: install this skill, point at a staging vLLM, and require ceiling.py output in the PR description before merging server flags.

Ollama and Apple Silicon paths #

While README highlights Arc and CUDA, gotchas.md also covers Apple and AMD. Perplexity Lily Apple Silicon engine is a different approach (custom engine), but llama.cpp levers in the skill still apply when you serve through OpenAI-compatible adapters.

SGLang vs vLLM choice in 2026 #

Your cloud may ship Dynamo-wrapped SGLang while your laptop runs stock vLLM. The skill refuses to treat them as interchangeable: gotchas.md is per engine.

When reading SemiAnalysis-style throughput debates, remember those are rack-scale stories. This skill is single-node honest benches you can rerun tonight.

When agents mis-measure #

Common traps the skill blocks:

  • Counting stream chunks instead of tokenizer tokens.
  • Using short prompts that hide prefill quadratic cost.
  • Ignoring battery power on laptops (bench flags runs on AC).
  • Calling a win when looping output detector fires.

Running python3 bench.py show after every queue should be non-negotiable.

Enterprise angle: why Sybil open-sourced it #

Sybil ties the skill to public registry recipes with PR links. Open-sourcing the methodology lets other hosts replicate before/after tables without sharing proprietary kernels.

That mirrors AWS's agent skill for distributed KV cache: vendors publish playbooks, not just sliders.

Quick FAQ for searchers asking "is local LLM worth it" #

Local LLM serving is worth the ops burden when any of these hold: PII cannot leave VPC, inference spend exceeds one FTE, p95 latency blocks an agent product, or you already own GPUs depreciating on the balance sheet.

This skill does not pick which model; it makes a chosen model faster without lying about numbers.

Sources: inference-optimisation-skill README (accessed November 22, 2026).

── more in #ai-infrastructure 4 stories Β· sorted by recency
── more on @sybil solutions 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain β€” perfect for shipping the agent you just read about.

$git push zahid main
β†’ Live at https://your-agent.zahid.host βœ“
Get free account β†’ Pricing
from €0/mo Β· no card required
LIVE [news/inference-optimisati…] indexed:0 read:8min 2026-11-22 Β· β€”