{"slug": "show-hn-neurobrix-pytorch-free-local-inference-with-a-rust-grammar-engine", "title": "Show HN: NeuroBrix – PyTorch-free local inference with a Rust grammar engine", "summary": "NeuroBrix released version 0.6.5, adding constrained decoding enforced by its own Rust-based grammar engine (packaged separately as nbxgrammar 0.1.0) for the OpenAI API's response_format json_schema, tool_choice: required or named function, and parallel_tool_calls: false with a forced call. The release also moves the oldest KV-cache pages to host memory so long conversations continue past the card's cache ceiling, refusing requests beyond the stated limit with the OpenAI error 400 context_length_exceeded, and on two V100-32GB cards Qwen3-Coder-30B-A3B dropped from 6.7 s to 0.33 s per token through run and a served 64-token answer from about 3 minutes to about 30 s. The library ships as nbxtensor 0.1.0, installs via pip install neurobrix without PyTorch, and adds /v1/embeddings; a 79,931-token single prompt took 39 minutes on two V100-32GB.", "body_md": "## Repository navigation\n\n# NeuroBrix 0.6.5\n\nNeuroBrix 0.6.5 brings constrained decoding and a faster, longer-reaching serving daemon, with nbxtensor 0.1.0 and nbxgrammar 0.1.0.\n\n- **Constrained decoding on the OpenAI API.**`response_format: json_schema` (a subset of JSON Schema; a constraining\n keyword outside it is refused by name),`tool_choice: required` or a named function, and`parallel_tool_calls: false`\n with a forced call are enforced by NeuroBrix's own grammar engine (written in Rust): the model can only write what the\nschema or the template's tool-call format allows. A reasoning model answers after its reasoning.\n- **Long conversations on a serving daemon.**`serve` states its context ceiling when it starts; past the card's KV\n cache the oldest pages move to host memory and decoding goes on, exactly; a conversation that leaves the cache comes\n back from host memory and its next turn prefills only what is new; a request longer than the ceiling is refused by\n `/v1/chat/completions` with the OpenAI error`400 context_length_exceeded` .\nA very long single prompt is answered slowly: 79 931 tokens took 39 minutes on two V100-32GB.\n- **Mixture-of-experts models on several cards run at the cards' speed.** On two V100-32GB, Qwen3-Coder-30B-A3B went\n from 6.7 s to 0.33 s a token through`run` , and a served 64-token answer from about 3 minutes to about 30 s; the\ndaemon keeps such a model loaded between requests.\n- **`/v1/embeddings` and an `embedding` model family** , ready for the first embedding model of the catalogue.\n- **A served request re-plans after a device out-of-memory** , as`run` does; laptops with a hybrid CPU or an\n integrated GPU are described correctly;`neurobrix import` recognises a model already installed.\n- **The hub is at [neurobrix.org](https://neurobrix.org)** and the documentation at\n [docs.neurobrix.org](https://docs.neurobrix.org) . Contributions are made under the\n [NeuroBrix Contributor Assignment Agreement](https://github.com/NeuroBrix/neurobrix/blob/main/docs/legal/contributor-assignment-agreement.md) .\n\nInstall: `pip install neurobrix` (the Triton engine, no PyTorch) or `pip install \"neurobrix[pytorch]\"`; on a Mac,\n\n`python -m nbxtensor install-metal`. The library is nbxtensor 0.1.0.\n\nThe grammar engine is its own package, `nbxgrammar` 0.1.0, which `pip install neurobrix` installs on Linux x86_64 and\n\naarch64, macOS arm64 and x86_64 and Windows x86_64; on any other platform no wheel of it exists and a constrained\n\nrequest is refused by name (`501 constrained_decoding_not_installed`).\n\nThe full list is in the [CHANGELOG](https://github.com/NeuroBrix/neurobrix/blob/v0.6.5/CHANGELOG.md#065---2026-11-10).", "url": "https://wpnews.pro/news/show-hn-neurobrix-pytorch-free-local-inference-with-a-rust-grammar-engine", "canonical_source": "https://github.com/NeuroBrix/neurobrix/releases/tag/v0.6.5", "published_at": "2026-11-10 13:03:32+00:00", "updated_at": "2026-11-10 13:24:28.944820+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "developer-tools", "ai-agents"], "entities": ["NeuroBrix", "nbxtensor", "nbxgrammar", "Qwen3-Coder-30B-A3B", "PyTorch", "Rust", "Nvidia V100-32GB", "OpenAI API"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-neurobrix-pytorch-free-local-inference-with-a-rust-grammar-engine", "markdown": "https://wpnews.pro/news/show-hn-neurobrix-pytorch-free-local-inference-with-a-rust-grammar-engine.md", "text": "https://wpnews.pro/news/show-hn-neurobrix-pytorch-free-local-inference-with-a-rust-grammar-engine.txt", "jsonld": "https://wpnews.pro/news/show-hn-neurobrix-pytorch-free-local-inference-with-a-rust-grammar-engine.jsonld"}}