{"slug": "show-hn-resin-fast-native-scheduler-for-llm-inference-engines", "title": "Show HN: Resin, fast native scheduler for LLM inference engines", "summary": "Eric Zhang released Resin, a Rust-based headless scheduler, KV cache and metrics server for LLM inference that lets developers write model-specific forward-pass kernels instead of using a full engine. Using Resin, Zhang built standalone engines for DeepSeek-V4.1-Flash and GLM-5.3-Flash on Blackwell GPUs in 12 hours that came within 5-10% of DeepInfer on prefill and decode and, he says, rival vLLM and SGLang. Resin ships as pip-installable `resin` plus separate `resin-models-cuda` and `resin-models-torch` runner packages, requires Python 3.11+ and Rust, and Zhang explicitly warns it is \"not production-quality software,\" directing production users to SGLang or vLLM.", "body_md": "Resin is not an inference engine. It's a library of *building blocks* for LLM\ninference.\n\nAt its core is the headless `Engine`: a fast Rust scheduler, KV cache, and\nmetrics server. You then write the forward pass as **model-specific kernels,\noptimized for your hardware.**\n\n```\ntoken IDs ──► Rust Engine ──► frozen BatchPlan ──► Python Runner ───► token IDs\n                  ▲                                    │\n                  └──────── ownership & commit ────────┘\n```\n\nResin also has optional HTTP serving and Prometheus metrics.\n\nWarning\n\nThis is not production-quality software. If you're looking for an actual inference engine, please use SGLang or vLLM. Read on if you want to hack on LLM inference systems, new models or hardware.\n\nWith the right kernels, this makes it possible to autonomously \"vibe code\" even\nthe most complex models. In 12 hours, we used Resin to build standalone engines\nfor [DeepSeek-V4.1-Flash](https://github.com/ekzhang/resin/blob/main/models/cuda/resin_models_cuda/deepseek_v41) and\n[GLM-5.3-Flash](https://github.com/ekzhang/resin/blob/main/models/cuda/resin_models_cuda/glm5_3_flash) on Blackwell GPUs\nthat rival vLLM and SGLang.\n\n| Prefill | Decode | \n|---|---|\n\n*Prefill and decode for DeepSeek V4.1 Flash, within 5-10% of DeepInfer.*\n\nStart with the two-page [architecture overview](https://github.com/ekzhang/resin/blob/main/docs/architecture.md).\n\nEither fork the repository or import `resin` and `resin_models_cuda` as standalone\npackages. Then use the following skills:\n\n- [**add-resin-model**](https://github.com/ekzhang/resin/blob/main/skills/add-resin-model/SKILL.md) -\nCreate an inference engine for a new model from scratch, using Resin.\n- [**optimize-resin-performance**](https://github.com/ekzhang/resin/blob/main/skills/optimize-resin-performance/SKILL.md) -\nIterate on your inference engine to improve performance with an \"autoresearch\"\nworkflow, diagnose bottlenecks and validate improvements, drawing on past\nlearnings.\n\nImporting `resin` does not import PyTorch or initialize CUDA.\n\nInstall the core library directly from GitHub with Python 3.11+ and Rust installed:\n\n```\npip install \"git+https://github.com/ekzhang/resin.git\"\n```\n\nPip builds the Rust extension locally. The import remains `resin`.\nUse `@<commit>` after `.git` to pin a revision.\n\nModel runners are separate packages: [`resin-models-cuda`](https://github.com/ekzhang/resin/blob/main/models/cuda/pyproject.toml)\nfor optimized CUDA implementations and\n[`resin-models-torch`](https://github.com/ekzhang/resin/blob/main/models/torch/pyproject.toml) for reference implementations.\nThe repository's development workspace installs both; the examples below assume\nthat workspace or a separately installed runner package.\n\nResin requires Python 3.11+ and Rust.\n\n```\nuv run scripts/rust_build_and_types.py\n```\n\n`uv run` syncs the project dependencies automatically. The build command then\nsyncs type-library sources and builds the Rust extension with generated stubs.\n\nThe Modal harnesses are launched from the repository root. Install the development dependencies first, then authenticate with Modal once if this is a new machine:\n\n```\nuv run modal setup\n```\n\n`modal setup` is interactive and only needs to be repeated when the local\nModal credentials change.\n\nAll models share one benchmark CLI and one parameterized serving app. The model\nregistry in [`deploy/models.py`](https://github.com/ekzhang/resin/blob/main/deploy/models.py) supplies checkpoint revisions,\nhardware, parallelism, and model-specific runner construction.\n\n```\n# Populate the pinned checkpoint without allocating a GPU.\nuv run scripts/bench.py --backend modal --model gpt-oss-20b --download-only\n\n# Sweep prefill and decode, save traces, and plot the measured results.\nuv run scripts/bench.py --backend modal --model gpt-oss-20b \\\n  --context-lengths 4096,8192 --batch-sizes 1,2,4,8 \\\n  --output bench/gpt-oss-20b.jsonl --trace-dir bench/traces --plot\n```\n\nFor local GPUs, use `--backend local --checkpoint /path/to/checkpoint`; models\nwith a separate draft checkpoint also accept `--assistant-checkpoint`. The\nlocal backend is the default. Modal submits the repository's source into the\nshared CUDA image, so the client machine does not need CUDA installed.\n\nPrefill charts compare complete-context latency with input tokens/s/GPU; decode\ncharts compare per-user output tokens/s with output tokens/s/GPU. Natural draft\nacceptance and model routing are the defaults. `--accepted-drafts K` and\n`--balanced-experts` explicitly enable synthetic workloads. See\n[benchmarking](https://github.com/ekzhang/resin/blob/main/docs/benchmarks.md) for the timing protocol and trace scope.\n\nFor an already running endpoint, `scripts/check_server.py URL --output REPORT`\nchecks serving correctness, prefix-cache reuse, context-size TTFT, and optional\nimage/reasoning/stress cases. It can also evaluate a small local MMLU CSV or\nJSONL subset with `--eval-file` (defaults: `--eval-limit 100`,\n`--eval-concurrency 32`); see\n[server checks](https://github.com/ekzhang/resin/blob/main/docs/benchmarks.md#http-correctness-and-latency-checks).\n\n``` python\nfrom resin import Engine, SamplingParams\nfrom resin_models_torch.lfm2 import Lfm2TorchRunner\n\nrunner = Lfm2TorchRunner(device=\"cpu\")\n\nwith Engine(runner, context_length=32768) as engine:\n    request = engine.submit(\n        [1, 6, 24131],\n        sampling_params=SamplingParams(\n            max_output_tokens=8,\n            stop_token_ids=[runner.eos_token_id],\n            temperature=1.0,\n            top_p=0.95,\n        ),\n    )\n    while engine.has_work():\n        for event in engine.step():\n            print(event)\n    generated_ids = request.take_result()\n```\n\n`SamplingParams` is the frozen request policy for generation limits, stop token\nIDs, sampling, and logprob options. Use the recipe's `stop_token_ids`\nfor chat requests, or supply the model's stop tokens explicitly for raw token\nrequests. `Engine` has no default EOS token.\n\nA custom synchronous runner only needs one method:\n\n``` python\nimport numpy as np\n\nfrom resin import BatchPlan, ForwardOutput\n\nclass Runner:\n    def forward(self, plan: BatchPlan) -> ForwardOutput:\n        return ForwardOutput(\n            token_ids=np.zeros(plan.num_sequences, dtype=np.uint32),\n            token_indptr=np.arange(plan.num_sequences + 1, dtype=np.uint32),\n        )\n```\n\n`ForwardOutput` carries flat token IDs and row offsets, with optional logprob\ncolumns. The same result type covers ordinary and speculative decoding.\n\nManaged history and recurrent state are cache groups declared through a frozen\n`RunnerConfig`. The Engine binds their one physical arena during construction.\nPass `prefix_cache=True` to reuse aligned prefixes across requests.\n\nThe built-in native server exposes one Engine through OpenAI Chat Completions and\nResponses, Anthropic Messages, and a small streaming chat page. Model-specific\nprompt rendering and incremental response parsing live in the independent\n`resin-recipe` Rust crate. Native model recipes implement the same `Recipe`\nprotocol as application-provided recipes. Stock image/video preprocessing also\nruns in Rust; the model runner performs the GPU vision encoder.\n\nThe bridge bounds queued jobs with `HttpServer(request_queue_capacity=1024)`.\nThis shared intake queue absorbs bursts while requests await engine admission;\nit does not reserve GPU cache or change the engine's active batch size. Active\nHTTP sessions (128 per replica), admissions before an inference step (4 per\nreplica), and buffered events per client (64) remain bounded. A full intake\nqueue returns HTTP 429; slow or disconnected clients are cancelled without\nblocking other requests. Actor failure stops the listener, and\n`HttpServer.shutdown()` requests shutdown after the current model step.\n\n```\n# Run one LFM2 prompt in the terminal.\nuv run python scripts/lfm2_chat.py\n\n# Serve LFM2 locally over HTTP.\nuv run python scripts/lfm2_serve.py\n```\n\nThen open [http://127.0.0.1:8000](http://127.0.0.1:8000). The page reports prompt and generated token\ncounts, elapsed generation time, and output tokens per second. The same process\nserves `/v1/chat/completions`, `/v1/responses`, `/v1/messages`, and `/v1/models`.\nBoth examples explicitly use the CPU Torch runner.\n\n[`deploy/modal_serve.py`](https://github.com/ekzhang/resin/blob/main/deploy/modal_serve.py) serves every registered model.\nSelect `RESIN_MODEL` at launch; each selection has a separate app and endpoint,\nwith shared download, warmup, process management, and HTTP startup code.\nPopulate its pinned checkpoint before deploying:\n\n```\nRESIN_MODEL=gpt-oss-20b uv run modal run deploy/modal_serve.py\nRESIN_MODEL=gpt-oss-20b uv run modal deploy deploy/modal_serve.py\n\nRESIN_MODEL=deepseek-v41 uv run modal run deploy/modal_serve.py\nRESIN_MODEL=deepseek-v41 uv run modal deploy deploy/modal_serve.py\n```\n\nUse `uv run scripts/bench.py --help` to see the current model keys accepted by\n`RESIN_MODEL` and the benchmark CLI. The shared\n[`MODEL_SPECS` registry](https://github.com/ekzhang/resin/blob/main/deploy/models.py) defines each selection's checkpoint,\nhardware, and serving limits. Add `--publish-only` to the download command to\nadopt complete files already in the checkpoint volume without downloading their\ncontents. Serving and benchmarking use the same checkpoint inventory and\ncompletion marker.\n\nMulti-rank deployments use the shared distributed worker, Engine, and HTTP\ncoordinator. For example, DeepSeek uses attention DP and expert parallelism,\nwhile GPT-OSS 120B uses attention and expert tensor parallelism. Requests are\nrouted round-robin across attention-DP replicas unless `routed_dp_rank` in the\nJSON body or `resin-routed-dp-rank` in the header selects a replica. Single-rank\ndeployments use a local Engine through the same HTTP frontend.\n\nEach deployed URL serves `/`, `/health`, `/v1/models`,\n`/v1/chat/completions`, and `/v1/responses`; model configuration controls\n`/v1/messages` support. The functions\nscale to zero after 15 idle minutes and are capped at one GPU container apiece.\nThe chat demo renders Markdown and TeX math and requests up to 16,384 output\ntokens by default. Model loading, CUDA graph capture, and one real prompt\nwarmup all complete before the endpoint starts accepting traffic.\nBecause `requires_proxy_auth=False`, anyone with a URL can submit inference\nrequests and incur GPU usage; remove the deployment when it is no longer\nneeded.\n\nSelect a native recipe by name and supply its tokenizer or multimodal processor. Messages and content blocks are Rust-owned typed objects:\n\n``` python\nimport resin\nfrom resin.recipe import get_native_recipe\nfrom resin.server import HttpServer\n\nrecipe = get_native_recipe(\"gpt_oss\", tokenizer)\nprompt = recipe.render(\n    [\n        resin.recipe.Message(resin.recipe.Role.USER, [resin.recipe.Text(\"Hello!\")]),\n    ]\n)\nparser = recipe.create_parser(prompt)\n\nHttpServer(engine, recipe=recipe, model=\"my-model\", stream_interval=4).serve()\n```\n\nThe native layer currently provides:\n\n- sticky continual batching with prefill priority and memory retraction;\n- phase-homogeneous prefill or decode batches, token budgets, and prefill chunks;\n- one flat physical cache pool shared by History, Window, and State groups;\n- sparse, chronological `CacheGroupPlan` rows and exact append destinations;\n- opt-in exact-token LRU prefix caching with immutable page sharing and sliding-window and recurrent-state copy-on-write;\n- retryable pre-accept failure and fail-stop post-accept quarantine.\n\n`crates/resin-core` is Python-independent Rust containing request\nlifecycle, scheduling, token-range planning, grouped cache allocation, and\nexecution transactions. It uses `rkyv` for the distributed\nplan wire format. `crates/resin-py` converts Python inputs and adapts runner\ncallbacks, plans, errors, handles, and events to that core. `crates/resin-recipe`\nowns conversation formats and CPU media preparation; `crates/resin-http` owns\nthe serving protocols and streaming responses.\n\nHistory cache grows on demand to each frozen plan. Window cache grows toward bounded cyclic storage whose plan row is projected into chronological order. With prefix caching enabled, a block trie indexes exact token prefixes, cached pages become immutable readers, and Window pages are copied on write when a request advances its ring. State groups retain a two-checkpoint ring. At an aligned prefix hit, the input checkpoint stays immutable while the next page is writable; the runner copies only when input and output pages differ. LRU entries use spare capacity in the same pool and are evicted when live requests need it.\n\nThe top-level `resin` package exposes the Engine, immutable plans and request\npolicies, runner protocols, cache layouts, and metrics types. See\n[`resin/__init__.py`](https://github.com/ekzhang/resin/blob/main/resin/__init__.py) for the current exports and\n[`resin/protocols.py`](https://github.com/ekzhang/resin/blob/main/resin/protocols.py) for the runner contracts.\n\nDistributed coordination lives in `resin.backend.distributed`, with explicit\nprocess groups in `resin.backend.topology`. Concrete runners are imports from\nseparately installed architecture submodules such as\n`resin_models_cuda.deepseek_v41`, `resin_models_cuda.gpt_oss`, and\n`resin_models_torch.lfm2`.\n\n```\ncargo test --all-targets\nuv run pytest -q\nuv run ruff format --check .\nuv run ruff check .\nuv run ty check\nuv run python scripts/cloc.py\n```\n\nGPU validation is model- and machine-specific; local tests use fakes or skip unavailable accelerator libraries.\n\nThe code-size report uses Tokei to count production Rust/Python by component, excluding comments, blank lines, tests, deployment tools, and generated stubs. Use the current report instead of a checked-in line-count snapshot.\n\nFor resident prefill/decode sweeps, optional fixed speculative acceptance and\nbalanced routing, Perfetto traces, and throughput/latency charts, see\n[benchmarking](https://github.com/ekzhang/resin/blob/main/docs/benchmarks.md).\n`Engine` collects native request/cache metrics and optional runner observations\nby default (`enable_metrics=True`). Use `enable_metrics=False` to opt out.\nThe HTTP server publishes `/metrics`; see [metrics](https://github.com/ekzhang/resin/blob/main/docs/metrics.md)\nfor Prometheus queries and forwarding through an OTLP Collector.\n\n- `crates/resin-core/src/engine.rs` — request lifecycle and execution transactions\n- `crates/resin-py/src/lib.rs` — Python objects and the native Engine adapter\n- `crates/resin-py/src/planning.rs` — conversion from core batch/resource views to`BatchPlan`\n- `crates/resin-py/src/runner.rs` — execution discovery and one-time resource binding\n- `crates/resin-py/src/cache.rs` — Python adapters for core KV descriptors\n- `crates/resin-core/src/scheduler.rs` — cohort selection and queue-age policy\n- `crates/resin-core/src/planning.rs` — typed token-range planning\n- `crates/resin-core/src/resources/` — grouped cache allocation, prefix reuse, and leases\n- `crates/resin-recipe/` — native conversation formats, parsing, and CPU media preparation\n- `crates/resin-http/` — HTTP protocols, streaming, and serving lifecycle\n- `resin/backend/` — reusable execution and resource gadgets\n- `models/` — separately installable model definitions, loaders, and runners\n- `scripts/` — examples, benchmarks, and local maintenance tools\n- `deploy/` — shared model registry, checkpoint/cache helpers, and parameterized Modal serving/benchmark adapters\n- `tests/` — native and runner-contract coverage\n\nResin was based on and inspired by systems from SGLang, TokenSpeed, and DeepInfer.", "url": "https://wpnews.pro/news/show-hn-resin-fast-native-scheduler-for-llm-inference-engines", "canonical_source": "https://github.com/ekzhang/resin", "published_at": "2026-10-26 17:06:14+00:00", "updated_at": "2026-10-26 17:11:22.127356+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "developer-tools", "ai-research"], "entities": ["Resin", "Eric Zhang", "DeepSeek-V4.1-Flash", "GLM-5.3-Flash", "vLLM", "SGLang", "DeepInfer", "Blackwell"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-resin-fast-native-scheduler-for-llm-inference-engines", "markdown": "https://wpnews.pro/news/show-hn-resin-fast-native-scheduler-for-llm-inference-engines.md", "text": "https://wpnews.pro/news/show-hn-resin-fast-native-scheduler-for-llm-inference-engines.txt", "jsonld": "https://wpnews.pro/news/show-hn-resin-fast-native-scheduler-for-llm-inference-engines.jsonld"}}