# Inference Optimisation Skill: Hill-Climb Local LLM Serving With an Agent

> Source: <https://explainx.ai/blog/inference-optimisation-skill-sybil-hill-climb-local-llm-november-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog>
> Published: 2026-11-22 00:00:00+00:00

**Recommendation:** Interleave candidate runs with control runs via queue.sh, require at least 6 samples and no metric regression above 3%, then pass long-context needle and full-KV soak gates before pinning a clean build.
More in the playbook "Inference and self-hosting": <https://wpnews.pro/playbooks/inference-serving>

**The [inference-optimisation-skill](https://github.com/sybil-solutions/inference-optimisation-skill?ref=explainx) repository turns months of measured LLM serving hill-climbs into a portable agent workflow.** Sybil Solutions published it in **November 2026** with about **1,700 GitHub stars**. The skill targets **decode, prefill, speculative decoding, KV cache size, startup, and TTFT** on **any hardware and quantization**, under a **memory cap**, with **long-context stability gates** before you call a build production-ready.

## TL;DR

| Piece | Role | 
|---|---|
| `SKILL.md` | Procedure: inventory, ceilings, one-lever loop, stop rules | 
| `references/` | ceilings.md, levers.md, gotchas.md, publishing.md | 
| `scripts/bench.py` | OpenAI-compatible harness; writes `ledger.jsonl` | 
| `scripts/ceiling.py` | Theoretical tok/s caps from hardware JSON | 
| `scripts/queue.sh` | Interleaved control/candidate/control runs | 
| Install path | `~/.claude/skills/inference-optimisation` for Claude Code | 

## Why another "make LLM fast" repo matters

Most speed guides list **flags**. This repo argues **noise is ±5–10%**, so single-run wins are often drift. The authors encoded what worked on **Intel Arc B390**, **Arc Pro B70**, **RTX 3090**, and **4× H200 NVL**, including **failures** in PLAN logs.

That pairs with industry moves toward **agent-driven tuning**, such as [Together Compiler](https://explainx.ai/blog/together-compiler-agents-optimize-inference-1111-tps-minimax-1205-tps-deepseek-claimed-vs-verified-2026) and [Microsoft MAI hill-climbing in Copilot Excel](https://explainx.ai/blog/microsoft-mai-hill-climbing-copilot-excel-nadella-july-2026). Here the agent is **yours**, running against **your** rack or laptop.

## Results the README publishes

KV cache slots with one highlighted token stored across decode steps

The README table is the credibility anchor. Selected rows:

| Hardware | Stack | Before → after (README) | 
|---|---|---|
| Arc B390 laptop | Ornith-1.5-35B EXL3, SGLang XPU + MTP | 43.6 → 53 tok/s decode, 1 user | 
| Arc B390 | Same | Prefill 256K: 297 → 628 tok/s | 
| Arc Pro B70 | Qwen3.8-27B, vLLM + custom XPU kernels | 91.2 tok/s decode at 1 user | 
| RTX 3090 | Qwen3.6-35B-A3B EXL3 + MTP | 251.7 / 357.9 tok/s prose/code | 
| 4× H200 NVL | DeepSeek-V4.1-Flash, vLLM + speculative decoding | Crashed → 229.1 tok/s decode, 17,546 prefill | 

Each line points to **local-ai-registry** pull requests or **trellis-serve** docs. explainx.ai does not re-run those benches; treat numbers as **author-measured** until you replicate on your silicon.

The **B390** example includes a **ceiling calculation**: ~100 tok/s decode for one user needs **two useful speculative drafts**, but the model accepts ~14% at depth 2, so **practical ceiling ~88–97 tok/s**. Hitting **~55% of physics** is labeled honest progress, not failure.

## The workflow loop

Small draft token running ahead of verification on parallel rails

The ASCII diagram in the README flows:

1. **Measure hardware** →`THEORY.md` bandwidth and FLOPs per token.
2. **Baseline profile** → pick the lever with largest ceiling gap × likelihood.
3. **queue.sh** runs**serialized** candidate vs**neighbouring controls** with memory watchdog and**≥6 samples** .
4. Accept only if beat **both** controls and**no metric regresses >3%** .
5. **Long-context needle gate** and**concurrent full-KV soak** .
6. **Pinned clean build** and**six acceptance gates** before publish.

That is stricter than tweeting a **GGUF quant** swap. It matches how [Together AI and Ollama prefix-cache guidance](https://explainx.ai/blog/together-ai-ollama-prefix-cache-long-horizon-agents-2026) thinks about **stable prompts** for caches.

## Scripts builders actually run

### bench.py

Hits any **OpenAI-compatible** server for:

- Startup and short-prompt **TTFT**
- **c1** decode median and**cN** aggregate decode
- Cold prefill at multiple lengths
- **Needle retrieval** in long context
- Peak memory with cap watchdog

Token counts come from **server usage stats**, not streamed chunks, avoiding a common benchmarking trap the README calls out.

### ceiling.py

Feed a JSON like `examples/ceiling-ornith-b390.json` to print **decode, prefill, TTFT, startup, and KV maxima**. Agents are instructed to run this **before** chasing arbitrary tok/s targets.

### queue.sh

Runs **control / candidate / control** interleaves so thermal drift and background jobs do not fake wins.

## Install for Claude Code and other agents

```
git clone https://github.com/sybil-solutions/inference-optimisation-skill
mkdir -p ~/.claude/skills
cp -r inference-optimisation-skill/skills/inference-optimisation ~/.claude/skills/
```

Then prompt examples from the README:

- "hill-climb decode and prefill for the model on :8000 under a 40 GiB cap"
- "what's the theoretical max tok/s for this model on this machine?"

For non-Claude agents, copy `skills/inference-optimisation/SKILL.md` or paste `PROMPT.md`.

Quick start without an agent:

```
S=skills/inference-optimisation
python3 $S/scripts/ceiling.py $S/examples/ceiling-ornith-b390.json
python3 $S/scripts/bench.py run configs/base.json --long 131072
```

## Principles that survived real climbs

The README highlights ideas that generalize beyond Sybil's stacks:

| Principle | Why it matters | 
|---|---|
| Ceilings first | Know if 53 tok/s is great or hopeless | 
| Interleaved controls | Kill ±10% noise | 
| Real tokenizer lengths | Avoid fake prefill wins | 
| Unified memory accounting | cgroups miss GPU residency on APUs | 
| Spec decode cost per step | Acceptance rate alone misleads | 
| Log failures | Most value is what did not work | 

Gotchas for **[vLLM](https://github.com/vllm-project/vllm?ref=explainx), [SGLang](https://github.com/sgl-project/sglang?ref=explainx), llama.cpp, ExLlama** and **Intel, NVIDIA, Apple, AMD** sit in `references/gotchas.md`.

## How this compares to cloud-only tuning

If you rent **Together** or **OpenRouter**, you optimize **routing and batching**, not **kernel levers**. This skill is for teams running **[Qwen on Together pricing](https://explainx.ai/blog/qwen4-max-2-4-trillion-moe-closed-model-together-ai-pricing-1m-context-november-2026)** or **self-hosted vLLM** on owned GPUs.

For **KV cache concepts**, start with [what is KV cache](https://explainx.ai/blog/what-is-kv-cache-llm-inference-explained-guide-2026). For **Rubin vs GB300 marketing numbers**, see [vLLM Rubin claimed vs verified](https://explainx.ai/blog/vllm-rubin-93x-throughput-claimed-vs-verified-november-2026).

## Agent skill ecosystem context

One highlighted skill card pulled from a shelf of agent skill files

Nvidia's **[SkillEvaluator guide](https://explainx.ai/blog/nvidia-skillevaluator-agent-skills-three-tier-cli-guide-2026)** pushes **CI gates on agent skills**. This skill is the opposite flavor: a **domain skill** with **Python harnesses** instead of only markdown instructions.

[Poki's iPhone GUI agent skill](https://explainx.ai/blog/poki-open-source-iphone-gui-agent-claude-code-install-2026) shows the same pattern: **ship SKILL.md + install path + real benchmarks**.

## When not to use this skill

Skip it if you only call **ChatGPT APIs** from a laptop. Skip it if you lack **root or container access** to inference servers. Skip it if you cannot afford **multi-hour** soak tests.

Use it when **on-prem Qwen, DeepSeek, or MoE models** already serve agents and **p95 latency** blocks rollout.

## Replication checklist

1. Copy **base.json** config with your`base_url` and model id.
2. Run **ceiling.py** with honest bandwidth/FLOPs JSON.
3. Capture **baseline ledger** before any flag change.
4. Change **one lever** documented in`levers.md` .
5. Pass **needle + soak** before tweeting results.

## Inside references/levers.md

The skill splits levers into **general** (apply on any stack) and **stack-specific** (vLLM, SGLang, llama.cpp quirks). Agents are told to load **levers.md** only when a ceiling gap points at a category:

| Lever family | Typical symptom | README mindset | 
|---|---|---|
| Speculative decoding | Good acceptance, slow step time | Optimize **per-step cost** , prune draft vocab | 
| KV cache layout | OOM at long context | Shrink bytes/token before chasing quant swaps | 
| Prefill batching | Great decode, slow first token | Separate prefill tuning from decode sweeps | 
| MTP / multi-token prediction | Unstable at 256K | Run needle gate before calling win | 
| Engine startup | Minutes to first request | Track startup separately in ledger | 

The **Arc B70** row where **SGLang beat vLLM** on the same card (**82.6 vs 75.7 tok/s** decode) is a reminder that **engine choice** is itself a lever, not a one-time decision.

## Unified memory on laptops matters

The README warns that **cgroup memory** on Intel **unified memory** laptops does not see full GPU residency. `mem.sh` exists because **40 GiB cap** violations only show up when driver footprint counts.

If you run **local Qwen** on a **64 GB LPDDR5X** machine, pair this skill with [Nvidia long-context compaction paper context](https://explainx.ai/blog/nvidia-you-dont-need-long-context-compaction-sssm-arxiv-november-2026) so you do not optimize decode while ignoring **context eviction** policy.

## Publishing.md and the six acceptance gates

Before Sybil publishes a **local-ai-registry** recipe, the skill demands:

1. **Pinned engine versions** and model hashes.
2. **Clean rebuild** proving the binary matches the best ledger row.
3. **Needle test** at declared max context.
4. **Concurrent users** at declared concurrency without collapse.
5. **Regression guard** on neighbour controls.
6. **Public results card** linking raw`ledger.jsonl` .

That discipline is why the README can cite **Registry Lab gates** moving from **28.6 → 41.0 decode** on B390 without hand-waving.

## Failure logging as a product feature

`PLAN.md + ledger.jsonl` is where rejected configs live. Agents trained only on success posts miss that **most hill-climbs are negative results**. The skill explicitly tells the agent to **write down failures** because the next lever choice depends on knowing **which knobs already failed**.

For teams building **internal agent platforms**, copy that pattern into your **MLOps tickets**: attach **failed bench JSON** the same way you attach **failed unit tests**.

## Pairing with Claude Code workshops

If your org is booking [Claude Code workshops](https://explainx.ai/blog/what-is-claude-ai-workshop-formats-pricing-how-to-choose-2026), add a **local inference lab** module: install this skill, point at a staging vLLM, and require **ceiling.py output** in the PR description before merging server flags.

## Ollama and Apple Silicon paths

While README highlights **Arc and CUDA**, `gotchas.md` also covers **Apple and AMD**. [Perplexity Lily Apple Silicon engine](https://explainx.ai/blog/perplexity-lily-apple-silicon-inference-engine-2026) is a different approach (custom engine), but **llama.cpp levers** in the skill still apply when you serve through **OpenAI-compatible adapters**.

## SGLang vs vLLM choice in 2026

Your cloud may ship **Dynamo-wrapped SGLang** while your laptop runs **stock vLLM**. The skill refuses to treat them as interchangeable: **gotchas.md** is per engine.

When reading [SemiAnalysis-style throughput debates](https://explainx.ai/blog/vllm-rubin-93x-throughput-claimed-vs-verified-november-2026), remember those are **rack-scale** stories. This skill is **single-node honest benches** you can rerun tonight.

## When agents mis-measure

Common traps the skill blocks:

- Counting **stream chunks** instead of tokenizer tokens.
- Using **short prompts** that hide prefill quadratic cost.
- Ignoring **battery power** on laptops (bench flags runs on AC).
- Calling a win when **looping output** detector fires.

Running `python3 bench.py show` after every queue should be non-negotiable.

## Enterprise angle: why Sybil open-sourced it

Sybil ties the skill to **public registry recipes** with PR links. Open-sourcing the methodology lets other hosts replicate **before/after tables** without sharing proprietary kernels.

That mirrors [AWS's agent skill for distributed KV cache](https://explainx.ai/blog/aws-distributed-kv-caching-agent-skill-elasticache-november-2026): vendors publish **playbooks**, not just **sliders**.

## Quick FAQ for searchers asking "is local LLM worth it"

Local LLM serving is worth the ops burden when **any** of these hold: **PII cannot leave VPC**, **inference spend exceeds one FTE**, **p95 latency blocks an agent product**, or **you already own GPUs depreciating on the balance sheet**.

This skill does not pick **which model**; it makes a chosen model **faster without lying about numbers**.

## Related reading

Sources: [inference-optimisation-skill README](https://github.com/sybil-solutions/inference-optimisation-skill?ref=explainx) (accessed November 22, 2026).
