{"slug": "show-hn-reame-a-cpu-inference-server-that-gets-faster-as-it-runs", "title": "Show HN: Reame – a CPU inference server that gets faster as it runs", "summary": "Reame, a CPU inference server built on llama.cpp, launches as a lean LLM server optimized for cheap hardware like shared vCPUs and ARM boxes. It uses persistent KV caching, n-gram drafting, and speculative decoding to accelerate repetitive AI workloads, making later requests cost a fraction of the first. The server targets narrow tasks such as document extraction and batch pipelines, offering an OpenAI-compatible API and zero-config CLI.", "body_md": "**A lean, fully-tested LLM inference server built on llama.cpp — designed for the hardware you already have: shared vCPUs, free tiers, 2-core ARM boxes.**\n\nReame is not the first inference server. It's the first one that treats cheap CPU hardware as a first-class citizen instead of a fallback. Its thesis is simple:\n\nOn a CPU, never compute the same thing twice.\n\nReame is built for **narrow, repetitive AI workloads over your own data, on\nhardware you already pay for** — the case where the answer lives in the\ncontext you provide, not in the model's general knowledge. That is exactly\nwhere a small model matches a frontier one (we measured 100% accuracy on\nlong-context extraction with a 7B on a free 2-core ARM box) and where\nReame's memory makes request #100 cost a fraction of request #1.\n\n| Use case | Why it fits | Suggested model |\n|---|---|---|\n| Document extraction & classification (RAG, invoices, tickets, scraping) | answers live in the context; prompts share prefixes → the disk cache pays | Qwen2.5 1.5B–7B |\n| Batch pipelines (tag 10k products overnight, meta descriptions, email triage) | repetitive by nature → Palimpsest drafts them; €0 per token, no rate limits | Qwen2.5 1.5B–3B |\n| AI features inside a thin-margin SaaS | a €5 VPS instead of a metered API keeps unit economics alive | Qwen2.5 1.5B–7B |\n| Privacy-bound work (legal, medical, public sector) | data never leaves your server — full sovereignty | Qwen2.5 7B |\n| Private code autocomplete (Continue.dev + OpenAI-compatible API) | line-level completion is a narrow task; code never leaves the laptop | Qwen2.5-Coder 1.5B |\n\n**What Reame is NOT for** — said plainly, because trust is built here: a\ngeneral-purpose ChatGPT replacement (frontier reasoning and broad knowledge\nneed frontier parameter counts), agentic coding assistants, or creative\nlong-form writing at scale. If your task needs a 100B-class brain, buy one;\nif it needs *your documents processed privately, forever, at zero marginal\ncost* — that's a realm you can own.\n\n- 🗂️\n**Persistent shared-prefix KV cache**— prompt prefixes are snapshotted to disk (zstd, checksummed, LRU-budgeted) and reused** across different prompts, restarts and processes**. A system prompt is paid for once, by the first user. - 📜\n**Palimpsest: the server remembers what it generated**— every completed generation feeds an on-disk n-gram archive; future requests draft from it at zero cost. Domain workloads repeat themselves — let them pay off. - 🎭\n**Il Suggeritore: grammar as a draft source**— constrained decoding uses structure to*forbid*tokens; Reame inverts it and uses structure to*propose*them. List numbering, bullets and format tokens are speculated for free on content nobody has ever generated before. - 🔮\n**Self-regulating speculative decoding**— a small draft model*or*zero-cost n-gram lookup proposes tokens; the target verifies them in one batched pass. Reame*measures*whether speculation pays on your hardware and switches it off by itself when it doesn't. - 🏛️\n**The Conclave: consensus as a quality knob**—`--best-of N`\n\ngenerates N candidate answers to the same prompt in one interleaved batch (one prefill, cloned into the others via KV copy; every weight read shared) and elects the winner by majority on the final result. The moment an absolute majority agrees, the stragglers are stopped. Honestly measured: it squeezes roughly one extra correct answer per quiz out of*the model you already run*— it does not make a 1.5B out-reason a 3B (consensus fixes variance, not bias). - 👥\n**Interleaved multi-user serving**— N concurrent generations advance together inside single multi-sequence batches, sharing every read of the model weights (the cost that dominates memory-bound CPU decoding). - 🌐\n**OpenAI-compatible REST API**—`/v1/completions`\n\n,`/v1/chat/completions`\n\n, SSE streaming, sessions, bearer auth, metrics. Point any OpenAI client at it. - ⚡\n**Zero-config CLI**—`reame run qwen2.5-1.5b`\n\ndownloads the model once, autoconfigures threads/KV/cache for the host and drops into a chat (or`--serve`\n\n). No config file until you want one. - 🧪\n**210 isolated test cases**— every layer is mockable and tested without a model; correctness of the multi-sequence, speculative and KV-clone paths is pinned against real models in integration tests.\n\nEvery number below was produced by the shipped binary on the hardware named — including the negative results that shaped the design.\n\n| Hardware | Model | Configuration | Result |\n|---|---|---|---|\nOracle Cloud free tier (2× ARM, 12 GB, €0/mo) |\nQwen2.5-7B Q4_K_M | plain, KV q8_0 | 3.3 tok/s |\n| Oracle Cloud free tier | TriLM 3.9B ternary TQ2_0 | 1.1 GB total RAM | ~10 tok/s |\n| Apple M3 Pro (6 threads) | Qwen2.5-1.5B Q4_K_M | plain | 52 tok/s |\n| Shared Contabo VPS (18 oversubscribed vCPUs) | 1.5B + 0.5B draft | speculative, 87% acceptance | 3.2× speedup |\n| Shared Contabo VPS | TinyLlama 1.1B | warm disk cache vs cold | 4.8× end-to-end |\n| Apple M3 Pro | Qwen2.5-1.5B | prompt-lookup on a rewrite task | 1.44× |\n| Apple M3 Pro | TinyLlama, 3 concurrent users | interleaved vs serialized | 1.6× |\n| Apple M3 Pro | Qwen2.5-1.5B, repeated request | archive speculation (palimpsest) | 2.3× (22→51 tok/s) |\n| Apple M3 Pro | Qwen2.5-1.5B, fresh list generation | form drafting (suggeritore) | 2.1× (4.4s→2.1s) |\n| Apple M3 Pro | Qwen2.5-1.5B ×5 candidates | Conclave: shared prefill + early consensus + fast nucleus | 8-question quiz wall 97s → ~50s |\n| Apple M3 Pro | Qwen2.5-1.5B `--best-of 5` vs single |\n3 arithmetic quizzes, strict grading | +0.5 to +2 correct, ~2.5× wall (not 5×) |\n| Oracle Cloud free tier | OLMoE 7B-A1B (MoE) vs dense 7B |\nsame 8-needle long-context test | 100% accuracy both · 17.8 vs 3.3 tok/s (5.4×) |\n\nTwo negative results that matter. On heavily oversubscribed shared vCPUs a draft\nmodel runs as slowly as its target, so speculation is counter-productive there —\nReame detects this and disables it at runtime. And the Conclave does **not**\nclose the gap to a model twice the size on hard reasoning: majority voting\ncorrects random slips, not systematic misunderstanding — we measured a 1.5B ×5\nland between the 1.5B and a 3B, never above the 3B. Benchmarks that only show\nwins are advertising; these are engineering.\n\n**Shared-prefix disk cache.** Prompts are split into fixed token blocks; a chain\nhash keys a KV snapshot at every block boundary. A *different* prompt that shares\na prefix restores the longest cached boundary and decodes only its own tail.\nUnlike GPU-resident prefix caches, snapshots live on NVMe: they survive restarts.\n\n**Self-regulating speculation.** Classic Leviathan/Chen acceptance (the rejected\ntoken is resampled from the residual distribution, so the output distribution is\nexactly the target's), with two CPU-first twists: the draft source can be free\nn-gram lookup mined from the prompt itself — ideal for extraction and rewrite\nworkloads — and a feedback controller adapts the draft length and turns\nspeculation off when measured acceptance or draft economics go negative.\n\n**The Conclave.** `--best-of N`\n\nsubmits N attempts at the same prompt to the\ninterleaved scheduler: attempt 0 is the untouched anchor (greedy stays greedy),\nthe explorers shift seed and heat up. The scheduler notices the identical\nprompts and **clones the prompt KV** instead of prefilling N times (copy the\ndonor's cache, decode only the last prompt token — argmax-verified equal to a\nfull prefill). Election is an exact-majority vote on each candidate's final\nnumber, with a Jaccard text-medoid fallback for prose; the moment a majority\nexists the remaining candidates are stopped mid-generation, and the CLI reports\n`CONCLAVE consensus=k/N`\n\nso a caller can escalate only when the conclave split.\nUse it as a quality knob: more accuracy from the model your hardware can\nafford, paid in idle interleaved compute rather than a bigger model's RAM.\n\n```\nreame list                                  # model catalog + what's on disk\nreame run qwen2.5-1.5b                      # download once, auto-config, chat\nreame run qwen2.5-1.5b \"Explain mmap\"       # one-shot answer\nreame run qwen2.5-1.5b --serve              # OpenAI-compatible API on :8080\nreame run qwen2.5-1.5b \"12*13-50?\" --best-of 5   # the Conclave\n```\n\n`run`\n\nresolves a catalog name (or any local GGUF path), downloads to\n`~/.reame/models`\n\non first use and picks threads, KV quantization and cache\ndirectory for the host. A config file is only needed when you want control.\n\n**Homebrew** (macOS / Linux):\n\n```\nbrew tap swellweb/reame\nbrew install reame\n```\n\n**Prebuilt binaries** — Linux x64/arm64 and macOS arm64 on the\n[releases page](https://github.com/swellweb/reame/releases)\n(runtime dependency: libzstd).\n\n**npm** (`npx reame`\n\n): planned — binaries are already built per platform.\n\n```\ngit clone https://github.com/swellweb/reame\ncd reame\ngit submodule update --init --depth 1 third_party/llama.cpp\n./build.sh                       # Release build + 210 test cases\n\n./scripts/download_models.sh     # TinyLlama (test model, ~670 MB)\n\n./build/src/reame --config config/reame.conf --prompt \"Hello\" --max-tokens 32\n./build/src/reame --config config/reame.conf --serve   # OpenAI-compatible API\n```\n\nDependencies: CMake ≥ 3.16, a C++17 compiler, and for the server Boost (headers), nlohmann-json and zstd:\n\n```\n# Debian/Ubuntu\nsudo apt install build-essential cmake libboost-dev nlohmann-json3-dev libzstd-dev pkg-config\n# macOS\nbrew install cmake boost nlohmann-json zstd pkg-config\n[model]\npath = models/qwen2.5-7b-instruct-q4_k_m.gguf\ncontext_length = 4096      # total KV budget (shared across users when parallel > 1)\nthreads = 4                # fewer is often faster on shared vCPUs — measure!\n\n[memory]\nkv_cache_type = q8_0       # f16 | q8_0 | q4_0 — halve/quarter context RAM\n\n[speculative]\nenabled = true\nmode = lookup              # model (needs draft_model_path) | lookup (no 2nd model)\n\n[cache]\ndirectory = /opt/reame/cache\nmax_size_mb = 4096         # LRU byte budget on disk\n\n[server]\nport = 8080\napi_key =                  # bearer auth when set\nparallel = 1               # >1 = interleaved multi-user serving\n```\n\n| Endpoint | Description |\n|---|---|\n`POST /v1/completions` |\ntext completion (SSE with `\"stream\": true` ) |\n`POST /v1/chat/completions` |\nchat completion |\n`POST /v1/sessions` · `.../save` · `.../load` · `DELETE .../{id}` |\nKV session snapshots |\n`GET /metrics` |\nrequest counters + speculative/cache metrics |\n`GET /health` |\nliveness (auth-exempt) |\n\nReame's footprint is watt-scale, not kilowatt-scale: it targets machines that already exist and are already powered on — no new silicon is racked to serve your model. We don't claim better joules-per-token than a saturated datacenter GPU — we claim you don't need one.\n\nReame is young and deliberately **opinionated and focused**: CPU-only serving,\none model per process, correctness pinned by tests at every layer. Not goals:\nGPU offload, training, model management UX. The llama.cpp submodule is pinned to\na known-good commit and bumped deliberately.\n\nDocumentation in Italian: [docs/README.it.md](/swellweb/reame/blob/main/docs/README.it.md).\n\nThe laptop story is the same one command: `reame run qwen2.5-1.5b`\n\ndownloads,\nautoconfigures and chats — nothing to learn. From there the two projects\ndiverge: Ollama optimizes for running *many* models casually; Reame optimizes\nfor serving *one* workload seriously on hardware that costs nothing. The\ndifference is one sentence:\n\nOllama runs models. Reame remembers having run them.\n\nGeneral-purpose servers treat every request as brand new: compute, discard, repeat. On a GPU that's fine — compute is cheap. On a cheap CPU, compute is the most expensive thing you have, and throwing it away is the cardinal sin. Everything in Reame attacks that: the disk prefix cache, the generation archive, the grammar prompter, self-regulating speculation, interleaved multi-user batches, the Conclave. None of it exists in Ollama.\n\nThe practical consequence: **a Reame server gets faster the longer it runs.**\nThe hundredth request costs a fraction of the first — the system prompt was\npaid once, similar answers draft themselves from the archive, structure is\nspeculated for free. No other server has that property.\n\nReame is free, MIT-licensed and built on nights and free-tier hardware. If it\nsaves you API bills or GPU rent, consider [sponsoring](https://github.com/sponsors/swellweb)\nthe work — sponsorships fund the roadmap: ARCA (the shared memory daemon),\nwarm-ahead prefill, and first-class MoE serving.\n\nReame stands on the shoulders of [llama.cpp](https://github.com/ggml-org/llama.cpp)\n(all tensor kernels; MIT). The disk-first cache thesis was inspired by\nantirez's DwarfStar4 line of thinking; the speculative pipeline by DeepSeek's\nDSpark work and the Leviathan/Chen speculative sampling theorem; archive\ndrafting is a shipped, persistent take on retrieval-based speculation (REST);\nform drafting inverts grammar-constrained decoding. Ideas are cited, numbers\nare ours.", "url": "https://wpnews.pro/news/show-hn-reame-a-cpu-inference-server-that-gets-faster-as-it-runs", "canonical_source": "https://github.com/swellweb/reame", "published_at": "2026-07-11 16:27:44+00:00", "updated_at": "2026-07-11 16:34:58.651809+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools", "ai-products"], "entities": ["Reame", "llama.cpp", "Qwen2.5", "Continue.dev", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/show-hn-reame-a-cpu-inference-server-that-gets-faster-as-it-runs", "markdown": "https://wpnews.pro/news/show-hn-reame-a-cpu-inference-server-that-gets-faster-as-it-runs.md", "text": "https://wpnews.pro/news/show-hn-reame-a-cpu-inference-server-that-gets-faster-as-it-runs.txt", "jsonld": "https://wpnews.pro/news/show-hn-reame-a-cpu-inference-server-that-gets-faster-as-it-runs.jsonld"}}