{"slug": "show-hn-spinf-serve-inference-server-for-bulk-scoring-and-decisions", "title": "Show HN: Spinf-serve – inference server for bulk scoring and decisions", "summary": "Spinf released spinf-serve, an Apache 2.0 inference server that scores many classification and extraction questions against a piece of content in a single forward pass on one 24 GB NVIDIA GPU, serving an OpenAI-compatible API on port 8000. The server returns a probability over each supplied option rather than generated text, and reports a floor_p value for the same question on empty content so users can see what the content added — in the company's example, the 12B model moves \"billing\" from 0.47 to 0.91. Spinf said it built the engine for its own daily market index read from millions of news articles, and published it because users asked to run it on their own hardware; it requires Linux, Python 3.12, CUDA 13 drivers and fp8 tensor cores on Ada, Hopper or Blackwell cards, with Ampere cards running int8 weights instead.", "body_md": "**Bulk classification and decision inference on one GPU.**\n\nspinf-serve reads a piece of content once and answers many questions about it in the same pass. Each answer is a probability over the options you give, not generated text. It runs on one 24 GB NVIDIA GPU and serves an OpenAI-compatible API, the same API as the hosted spinf service. Apache 2.0.\n\nWe built this engine for our own work: a daily market index read from millions of news articles, each article asked dozens of narrow, calibrated questions. A serving engine built for chat re-reads the content for every question and generates one token at a time, which made that job slow and expensive. spinf-serve lays every question out as a branch of one read of the content and packs many contents into one forward pass. Bulk classification is where it started and where it is strongest.\n\nThe same mechanism is what is now called a **decision model**: a model that returns a probability over a fixed set\nof answers (yes or no, a label, a level) instead of a paragraph. Any open instruction model works this way in\nspinf-serve, and dedicated decision models run as they are. The hosted spinf API grew out of this engine; we are\npublishing it because people asked to run it on their own hardware.\n\nIt is not a chat server. Nothing is generated unless you ask for it, and it is built for throughput rather than for the latency of a single request.\n\nLinux, an NVIDIA GPU with 24 GB and fp8 tensor cores (Ada, Hopper or Blackwell: RTX 4090 / 5090, L4, L40S, RTX 6000\nAda, H100), a driver supporting CUDA 13, Python 3.12. Ampere cards run with int8 weights instead\n([configuration](https://github.com/spinfinc/spinf-serve/blob/main/docs/configuration.md#models)).\n\n```\npip install \"spinf-serve[server]\"\nspinf-serve serve spinf/gemma-4-12b-fp8          # downloads the checkpoint on first start; serves on port 8000\n```\n\nOr the image, with the NVIDIA Container Toolkit installed (the volume keeps the model and the compile caches;\narguments after the image name are those of `spinf-serve serve`):\n\n```\ndocker run --gpus all -p 8000:8000 -v spinf-cache:/root/.cache ghcr.io/spinfinc/spinf-serve\n```\n\nThen ask. Two classifications and one extraction from a single read of the content:\n\n```\ncurl http://localhost:8000/v1/score \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"spinf/gemma-4-12b-fp8\",\n    \"messages\": [\n      { \"role\": \"user\",\n        \"content\": \"Customer support message:\\nI was charged twice for September, please refund one of the charges.\" }\n    ],\n    \"scoring\": {\n      \"queries\": [\n        { \"id\": \"topic\",\n          \"template\": \"\\n\\nTopic (billing, technical, account, shipping, or cancellation):{?}\",\n          \"options\": [\" billing\", \" technical\", \" account\", \" shipping\", \" cancellation\"] },\n        { \"id\": \"refund\",\n          \"template\": \"\\n\\nIs the customer asking for a refund?\\nAnswer:{?}\",\n          \"options\": [\" yes\", \" no\"] },\n        { \"id\": \"month\",\n          \"template\": \"\\n\\nQ: Which month was charged twice?\\nA: \\\"{*}\\\"\" }\n      ]\n    }\n  }'\n```\n\nFor each scored question the response gives every option's probability, and the same probability on empty content (real numbers from the 12B model):\n\n```\n{ \"id\": \"refund\", \"combinations\": [ { \"values\": {}, \"options\": [\n    { \"text\": \" yes\", \"logprob\": -0.90, \"p\": 0.72, \"floor_p\": 0.60 },\n    { \"text\": \" no\",  \"logprob\": -1.86, \"p\": 0.28, \"floor_p\": 0.40 } ],\n    \"residual\": 0.44 } ] }\n```\n\n`p` is the probability among your options. `floor_p` is the same question with no content, so `p - floor_p` is what\nthe content added (here `topic` moves \" billing\" from 0.47 to 0.91). `residual` is the probability that went to words\noutside your options. For `month` the response lists the most likely answers with their probabilities (\"September\" first).\nThe same body, with an API key, runs against `api.spinf.com`; the OpenAI SDKs work through `/v1/chat/completions` with\nthe `scoring` block in `extra_body` ([quickstart](https://github.com/spinfinc/spinf-serve/blob/main/docs/quickstart.md)).\n\n- **One read, many questions.** The content is read once; every question is a branch of the same pass and costs its\nown tokens only. Placeholders turn one template into a grid of questions.\n- **Scores you can threshold.** A probability per option with its floor and log-probability; a decision is a\nthreshold you choose on your own data ([calibration](https://github.com/spinfinc/spinf-serve/blob/main/docs/calibration.md) ).\n- **Extraction** from the same read:`{*}` text,`{n}` numbers and`{$}` amounts decoded with their probabilities,\nor free text generated into a slot.\n- **The same score whatever else is in the batch.** With the fp8 checkpoints, a content's scores are bit-identical\nalone or packed with thousands of others, across calls, days and servers. The`system_fingerprint` says when two\nresponses are not comparable.\n- **Images, audio and video frames** in the content, for the models that read them.\n- **Prompt packs.** Ready-made, calibrated question sets for a job (content moderation, triage, product listings,\nreviews, brand, entertainment), preloaded and named in a request ([below](#prompt-packs) ).\n- **Hosted or self-hosted.** A client switches between this server and`api.spinf.com` by changing the base URL and\nthe key.[spinf-client](https://github.com/spinfinc/spinf-client) talks to both; the[agent skills](https://github.com/spinfinc/skills) teach a coding agent the API, this server and the pack format.\n\nMeasured on one RTX 4090 with Gemma 4 12B in fp8: 24 Wikinews articles of about 470 tokens, 56 two-option questions\neach, 1,344 questions in all (the public job set in `fixtures/`), sent over HTTP to spinf-serve and to vLLM 0.26 with\nprefix caching. The script and the full results are in [benchmarks/](https://github.com/spinfinc/spinf-serve/blob/main/benchmarks/README.md).\n\n| one RTX 4090, warm | spinf-serve | vLLM 0.26, prefix caching, 128 requests in flight | \n|---|---|---|\n| questions answered per second | **491** | 200 | \n| tokens of work per second (content once, then each question's own tokens) | 11,200 | 4,600 | \n| the 1,344 questions | **2.7 s** | 6.7 s | \n\nvLLM already reuses each article across its 56 questions through the prefix cache. What it cannot do is answer them in one forward pass: it runs 1,344 single-token requests, while spinf-serve lays the 56 questions out as branches of one read and packs the 24 articles into two 16k-token passes. The gap grows with the number of questions per content.\n\nIn practice, per 4090 with the 12B model: the `spinf/moderation` pack (106 questions per message, 22 categories)\nscores about 20,000 messages per hour; a news article asked 74 questions is read at nearly 700 questions per second.\nExtraction slots add decoding work (a searched slot costs about as much as 200 to 600 tokens of scoring, a generated\none up to about 2,000) but share the read.\n\nA job is rarely one question: content moderation is 106 of them, each calibrated against normal traffic and combined\ninto category scores and a decision. A **prompt pack** is one file holding the questions, how the content is presented\nto the model, how the answers turn into scores and how the scores turn into a decision. The server preloads packs and\na request names one (`\"packs\": [{\"id\": \"spinf/moderation\"}]`), getting the pack's categories and decision back next\nto its own questions, from the same read.\n\nThe official packs are in [spinf-prompts](https://github.com/spinfinc/spinf-prompts), loaded by default. The server\nloads packs from any GitHub repository or local folder given with `--packs`; **[PACKS.md](https://github.com/spinfinc/spinf-serve/blob/main/PACKS.md) lists every\npack we know of**, ours and others', and an [issue](https://github.com/spinfinc/spinf-serve/issues/new?template=pack.yml)\nadds yours. Format, request fields and sources: [docs/prompt-packs.md](https://github.com/spinfinc/spinf-serve/blob/main/docs/prompt-packs.md).\n\n| name | based on | GPU | reads | \n|---|---|---|---|\n| `spinf/gemma-4-12b-fp8` (default) | Google Gemma 4 12B, FP8 with static activation scales | one 24 GB | text, images, audio, video | \n| `spinf/gemma-4-31b-fp8` | Google Gemma 4 31B, FP8 static | one 48 GB | text, images, video | \n| `spinf/qwen3.8-27b-int4` | Qwen3.8-27B, GPTQ int4 | one 24 GB | text, images, video | \n| `spinf/pplx-decider-v1-27b-int4` | Perplexity pplx-decider-v1-27b, GPTQ int4: a decision model with a 255-code readout | one 24 GB | text; scores only, with its own prompts and packs | \n\nEach is a checkpoint we quantized and calibrated, published on the [Hugging Face Hub](https://huggingface.co/spinf)\nwith its recipe (`quantizations/`); the engine pins the revision it was parity-tested on. Any other Hub checkpoint of\na supported architecture family runs by its path, quantized at load time.\n\n[docs/](https://github.com/spinfinc/spinf-serve/blob/main/docs/README.md): [quickstart](https://github.com/spinfinc/spinf-serve/blob/main/docs/quickstart.md), [examples](https://github.com/spinfinc/spinf-serve/blob/main/docs/examples.md), [scoring API](https://github.com/spinfinc/spinf-serve/blob/main/docs/scoring-api.md),\n[configuration](https://github.com/spinfinc/spinf-serve/blob/main/docs/configuration.md), [writing questions](https://github.com/spinfinc/spinf-serve/blob/main/docs/writing-questions.md), [calibration](https://github.com/spinfinc/spinf-serve/blob/main/docs/calibration.md),\n[prompt packs](https://github.com/spinfinc/spinf-serve/blob/main/docs/prompt-packs.md), [FAQ](https://github.com/spinfinc/spinf-serve/blob/main/docs/faq.md); the [guides](https://github.com/spinfinc/spinf-serve/blob/main/docs/guides/index.md) (decision models,\nprobabilities you can act on, few-shot classification, sentiment, survey coding, moderation, intent routing); and the\nengineering documents on how the engine works ([architecture](https://github.com/spinfinc/spinf-serve/blob/main/docs/ARCHITECTURE.md),\n[batch invariance](https://github.com/spinfinc/spinf-serve/blob/main/docs/BATCH_INVARIANCE.md), [precision](https://github.com/spinfinc/spinf-serve/blob/main/docs/PRECISION.md), [the Qwen stack](https://github.com/spinfinc/spinf-serve/blob/main/docs/QWEN.md),\n[extraction](https://github.com/spinfinc/spinf-serve/blob/main/docs/EXTRACTION.md)).\n\nVersion 0.1. The engine (scoring, extraction, generation, media, batch invariance, the two model stacks) is complete\nand parity-tested against recorded references; the server speaks the public API of the hosted service; the four\ncheckpoints are public. What is still missing is listed in [docs/README.md](https://github.com/spinfinc/spinf-serve/blob/main/docs/README.md#status).\n\nQuestions, ideas and bugs go to the [issues](https://github.com/spinfinc/spinf-serve/issues) of this repository.\nThere is no forum or chat, and nothing is too small.\n\n`spinf_serve/` holds the engine (`protocol`, `compiler`, `packer`, `scorer`, `extract`, `service`), the served models'\nmanifests, the `packs` module (pure Python, also used by the client and the hosted API), one model stack per architecture\nfamily in `models/`, the quantization schemes in `quant/`, the HTTP server in `server/` and the `spinf-serve` command.\n`quantizations/` has one folder per published checkpoint (recipe, calibration output, model card), `fixtures/` the public\nparity fixture and references, `benchmarks/` the throughput comparison. The map of the code is\n[docs/ARCHITECTURE.md](https://github.com/spinfinc/spinf-serve/blob/main/docs/ARCHITECTURE.md).\n\n```\nuv sync --extra server\nuv run pytest                                   # CPU tests (the official packs' golden fixtures need ../spinf-prompts or SPINF_PROMPTS_DIR)\nuv run ruff check . && uv run ruff format --check .\nCUDA_VISIBLE_DEVICES=0 uv run spinf-serve parity --fixture fixtures/wikinews_20261024.json.gz \\\n    --reference fixtures/wikinews_20261024.spinf-gemma-4-12b-fp8.json.gz --passes 2\n```\n\nEvery change that touches a kernel, a compile boundary or a quantization scheme ships with the GPU parity run:\nbit-identical log-probabilities on the recorded reference of each published checkpoint ([fixtures/](https://github.com/spinfinc/spinf-serve/blob/main/fixtures/README.md)).\nDependencies are pinned exactly; they change only together with a parity run. See [CONTRIBUTING.md](https://github.com/spinfinc/spinf-serve/blob/main/CONTRIBUTING.md).\n\nApache 2.0 (`LICENSE`, `NOTICE`). The models we run and re-quantize, the projects we build on, the techniques we\nimplement and the data we checked on are credited in [ACKNOWLEDGEMENTS.md](https://github.com/spinfinc/spinf-serve/blob/main/ACKNOWLEDGEMENTS.md).", "url": "https://wpnews.pro/news/show-hn-spinf-serve-inference-server-for-bulk-scoring-and-decisions", "canonical_source": "https://github.com/spinfinc/spinf-serve", "published_at": "2026-10-30 17:03:12+00:00", "updated_at": "2026-10-30 17:16:51.592964+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-infrastructure", "ai-tools", "large-language-models", "developer-tools"], "entities": ["Spinf", "spinf-serve", "NVIDIA", "CUDA 13", "spinf/gemma-4-12b-fp8", "api.spinf.com", "RTX 4090", "H100"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-spinf-serve-inference-server-for-bulk-scoring-and-decisions", "markdown": "https://wpnews.pro/news/show-hn-spinf-serve-inference-server-for-bulk-scoring-and-decisions.md", "text": "https://wpnews.pro/news/show-hn-spinf-serve-inference-server-for-bulk-scoring-and-decisions.txt", "jsonld": "https://wpnews.pro/news/show-hn-spinf-serve-inference-server-for-bulk-scoring-and-decisions.jsonld"}}