Show HN: Spinf-serve – inference server for bulk scoring and decisions

Spinf released spinf-serve, an Apache 2.0 inference server that scores many classification and extraction questions against a piece of content in a single forward pass on one 24 GB NVIDIA GPU, serving an OpenAI-compatible API on port 8000. The server returns a probability over each supplied option rather than generated text, and reports a floor_p value for the same question on empty content so users can see what the content added — in the company's example, the 12B model moves "billing" from 0.47 to 0.91. Spinf said it built the engine for its own daily market index read from millions of news articles, and published it because users asked to run it on their own hardware; it requires Linux, Python 3.12, CUDA 13 drivers and fp8 tensor cores on Ada, Hopper or Blackwell cards, with Ampere cards running int8 weights instead.

Bulk classification and decision inference on one GPU. spinf-serve reads a piece of content once and answers many questions about it in the same pass. Each answer is a probability over the options you give, not generated text. It runs on one 24 GB NVIDIA GPU and serves an OpenAI-compatible API, the same API as the hosted spinf service. Apache 2.0. We built this engine for our own work: a daily market index read from millions of news articles, each article asked dozens of narrow, calibrated questions. A serving engine built for chat re-reads the content for every question and generates one token at a time, which made that job slow and expensive. spinf-serve lays every question out as a branch of one read of the content and packs many contents into one forward pass. Bulk classification is where it started and where it is strongest. The same mechanism is what is now called a decision model : a model that returns a probability over a fixed set of answers yes or no, a label, a level instead of a paragraph. Any open instruction model works this way in spinf-serve, and dedicated decision models run as they are. The hosted spinf API grew out of this engine; we are publishing it because people asked to run it on their own hardware. It is not a chat server. Nothing is generated unless you ask for it, and it is built for throughput rather than for the latency of a single request. Linux, an NVIDIA GPU with 24 GB and fp8 tensor cores Ada, Hopper or Blackwell: RTX 4090 / 5090, L4, L40S, RTX 6000 Ada, H100 , a driver supporting CUDA 13, Python 3.12. Ampere cards run with int8 weights instead configuration https://github.com/spinfinc/spinf-serve/blob/main/docs/configuration.md models . pip install "spinf-serve server " spinf-serve serve spinf/gemma-4-12b-fp8 downloads the checkpoint on first start; serves on port 8000 Or the image, with the NVIDIA Container Toolkit installed the volume keeps the model and the compile caches; arguments after the image name are those of spinf-serve serve : docker run --gpus all -p 8000:8000 -v spinf-cache:/root/.cache ghcr.io/spinfinc/spinf-serve Then ask. Two classifications and one extraction from a single read of the content: curl http://localhost:8000/v1/score \ -H "Content-Type: application/json" \ -d '{ "model": "spinf/gemma-4-12b-fp8", "messages": { "role": "user", "content": "Customer support message:\nI was charged twice for September, please refund one of the charges." } , "scoring": { "queries": { "id": "topic", "template": "\n\nTopic billing, technical, account, shipping, or cancellation :{?}", "options": " billing", " technical", " account", " shipping", " cancellation" }, { "id": "refund", "template": "\n\nIs the customer asking for a refund?\nAnswer:{?}", "options": " yes", " no" }, { "id": "month", "template": "\n\nQ: Which month was charged twice?\nA: \"{ }\"" } } }' For each scored question the response gives every option's probability, and the same probability on empty content real numbers from the 12B model : { "id": "refund", "combinations": { "values": {}, "options": { "text": " yes", "logprob": -0.90, "p": 0.72, "floor p": 0.60 }, { "text": " no", "logprob": -1.86, "p": 0.28, "floor p": 0.40 } , "residual": 0.44 } } p is the probability among your options. floor p is the same question with no content, so p - floor p is what the content added here topic moves " billing" from 0.47 to 0.91 . residual is the probability that went to words outside your options. For month the response lists the most likely answers with their probabilities "September" first . The same body, with an API key, runs against api.spinf.com ; the OpenAI SDKs work through /v1/chat/completions with the scoring block in extra body quickstart https://github.com/spinfinc/spinf-serve/blob/main/docs/quickstart.md . - One read, many questions. The content is read once; every question is a branch of the same pass and costs its own tokens only. Placeholders turn one template into a grid of questions. - Scores you can threshold. A probability per option with its floor and log-probability; a decision is a threshold you choose on your own data calibration https://github.com/spinfinc/spinf-serve/blob/main/docs/calibration.md . - Extraction from the same read: { } text, {n} numbers and {$} amounts decoded with their probabilities, or free text generated into a slot. - The same score whatever else is in the batch. With the fp8 checkpoints, a content's scores are bit-identical alone or packed with thousands of others, across calls, days and servers. The system fingerprint says when two responses are not comparable. - Images, audio and video frames in the content, for the models that read them. - Prompt packs. Ready-made, calibrated question sets for a job content moderation, triage, product listings, reviews, brand, entertainment , preloaded and named in a request below prompt-packs . - Hosted or self-hosted. A client switches between this server and api.spinf.com by changing the base URL and the key. spinf-client https://github.com/spinfinc/spinf-client talks to both; the agent skills https://github.com/spinfinc/skills teach a coding agent the API, this server and the pack format. Measured on one RTX 4090 with Gemma 4 12B in fp8: 24 Wikinews articles of about 470 tokens, 56 two-option questions each, 1,344 questions in all the public job set in fixtures/ , sent over HTTP to spinf-serve and to vLLM 0.26 with prefix caching. The script and the full results are in benchmarks/ https://github.com/spinfinc/spinf-serve/blob/main/benchmarks/README.md . | one RTX 4090, warm | spinf-serve | vLLM 0.26, prefix caching, 128 requests in flight | |---|---|---| | questions answered per second | 491 | 200 | | tokens of work per second content once, then each question's own tokens | 11,200 | 4,600 | | the 1,344 questions | 2.7 s | 6.7 s | vLLM already reuses each article across its 56 questions through the prefix cache. What it cannot do is answer them in one forward pass: it runs 1,344 single-token requests, while spinf-serve lays the 56 questions out as branches of one read and packs the 24 articles into two 16k-token passes. The gap grows with the number of questions per content. In practice, per 4090 with the 12B model: the spinf/moderation pack 106 questions per message, 22 categories scores about 20,000 messages per hour; a news article asked 74 questions is read at nearly 700 questions per second. Extraction slots add decoding work a searched slot costs about as much as 200 to 600 tokens of scoring, a generated one up to about 2,000 but share the read. A job is rarely one question: content moderation is 106 of them, each calibrated against normal traffic and combined into category scores and a decision. A prompt pack is one file holding the questions, how the content is presented to the model, how the answers turn into scores and how the scores turn into a decision. The server preloads packs and a request names one "packs": {"id": "spinf/moderation"} , getting the pack's categories and decision back next to its own questions, from the same read. The official packs are in spinf-prompts https://github.com/spinfinc/spinf-prompts , loaded by default. The server loads packs from any GitHub repository or local folder given with --packs ; PACKS.md https://github.com/spinfinc/spinf-serve/blob/main/PACKS.md lists every pack we know of , ours and others', and an issue https://github.com/spinfinc/spinf-serve/issues/new?template=pack.yml adds yours. Format, request fields and sources: docs/prompt-packs.md https://github.com/spinfinc/spinf-serve/blob/main/docs/prompt-packs.md . | name | based on | GPU | reads | |---|---|---|---| | spinf/gemma-4-12b-fp8 default | Google Gemma 4 12B, FP8 with static activation scales | one 24 GB | text, images, audio, video | | spinf/gemma-4-31b-fp8 | Google Gemma 4 31B, FP8 static | one 48 GB | text, images, video | | spinf/qwen3.8-27b-int4 | Qwen3.8-27B, GPTQ int4 | one 24 GB | text, images, video | | spinf/pplx-decider-v1-27b-int4 | Perplexity pplx-decider-v1-27b, GPTQ int4: a decision model with a 255-code readout | one 24 GB | text; scores only, with its own prompts and packs | Each is a checkpoint we quantized and calibrated, published on the Hugging Face Hub https://huggingface.co/spinf with its recipe quantizations/ ; the engine pins the revision it was parity-tested on. Any other Hub checkpoint of a supported architecture family runs by its path, quantized at load time. docs/ https://github.com/spinfinc/spinf-serve/blob/main/docs/README.md : quickstart https://github.com/spinfinc/spinf-serve/blob/main/docs/quickstart.md , examples https://github.com/spinfinc/spinf-serve/blob/main/docs/examples.md , scoring API https://github.com/spinfinc/spinf-serve/blob/main/docs/scoring-api.md , configuration https://github.com/spinfinc/spinf-serve/blob/main/docs/configuration.md , writing questions https://github.com/spinfinc/spinf-serve/blob/main/docs/writing-questions.md , calibration https://github.com/spinfinc/spinf-serve/blob/main/docs/calibration.md , prompt packs https://github.com/spinfinc/spinf-serve/blob/main/docs/prompt-packs.md , FAQ https://github.com/spinfinc/spinf-serve/blob/main/docs/faq.md ; the guides https://github.com/spinfinc/spinf-serve/blob/main/docs/guides/index.md decision models, probabilities you can act on, few-shot classification, sentiment, survey coding, moderation, intent routing ; and the engineering documents on how the engine works architecture https://github.com/spinfinc/spinf-serve/blob/main/docs/ARCHITECTURE.md , batch invariance https://github.com/spinfinc/spinf-serve/blob/main/docs/BATCH INVARIANCE.md , precision https://github.com/spinfinc/spinf-serve/blob/main/docs/PRECISION.md , the Qwen stack https://github.com/spinfinc/spinf-serve/blob/main/docs/QWEN.md , extraction https://github.com/spinfinc/spinf-serve/blob/main/docs/EXTRACTION.md . Version 0.1. The engine scoring, extraction, generation, media, batch invariance, the two model stacks is complete and parity-tested against recorded references; the server speaks the public API of the hosted service; the four checkpoints are public. What is still missing is listed in docs/README.md https://github.com/spinfinc/spinf-serve/blob/main/docs/README.md status . Questions, ideas and bugs go to the issues https://github.com/spinfinc/spinf-serve/issues of this repository. There is no forum or chat, and nothing is too small. spinf serve/ holds the engine protocol , compiler , packer , scorer , extract , service , the served models' manifests, the packs module pure Python, also used by the client and the hosted API , one model stack per architecture family in models/ , the quantization schemes in quant/ , the HTTP server in server/ and the spinf-serve command. quantizations/ has one folder per published checkpoint recipe, calibration output, model card , fixtures/ the public parity fixture and references, benchmarks/ the throughput comparison. The map of the code is docs/ARCHITECTURE.md https://github.com/spinfinc/spinf-serve/blob/main/docs/ARCHITECTURE.md . uv sync --extra server uv run pytest CPU tests the official packs' golden fixtures need ../spinf-prompts or SPINF PROMPTS DIR uv run ruff check . && uv run ruff format --check . CUDA VISIBLE DEVICES=0 uv run spinf-serve parity --fixture fixtures/wikinews 20261024.json.gz \ --reference fixtures/wikinews 20261024.spinf-gemma-4-12b-fp8.json.gz --passes 2 Every change that touches a kernel, a compile boundary or a quantization scheme ships with the GPU parity run: bit-identical log-probabilities on the recorded reference of each published checkpoint fixtures/ https://github.com/spinfinc/spinf-serve/blob/main/fixtures/README.md . Dependencies are pinned exactly; they change only together with a parity run. See CONTRIBUTING.md https://github.com/spinfinc/spinf-serve/blob/main/CONTRIBUTING.md . Apache 2.0 LICENSE , NOTICE . The models we run and re-quantize, the projects we build on, the techniques we implement and the data we checked on are credited in ACKNOWLEDGEMENTS.md https://github.com/spinfinc/spinf-serve/blob/main/ACKNOWLEDGEMENTS.md .