Show HN: NeuroBrix – PyTorch-free local inference with a Rust grammar engine

NeuroBrix released version 0.6.5, adding constrained decoding enforced by its own Rust-based grammar engine (packaged separately as nbxgrammar 0.1.0) for the OpenAI API's response_format json_schema, tool_choice: required or named function, and parallel_tool_calls: false with a forced call. The release also moves the oldest KV-cache pages to host memory so long conversations continue past the card's cache ceiling, refusing requests beyond the stated limit with the OpenAI error 400 context_length_exceeded, and on two V100-32GB cards Qwen3-Coder-30B-A3B dropped from 6.7 s to 0.33 s per token through run and a served 64-token answer from about 3 minutes to about 30 s. The library ships as nbxtensor 0.1.0, installs via pip install neurobrix without PyTorch, and adds /v1/embeddings; a 79,931-token single prompt took 39 minutes on two V100-32GB.

Repository navigation NeuroBrix 0.6.5 NeuroBrix 0.6.5 brings constrained decoding and a faster, longer-reaching serving daemon, with nbxtensor 0.1.0 and nbxgrammar 0.1.0. - Constrained decoding on the OpenAI API. response format: json schema a subset of JSON Schema; a constraining keyword outside it is refused by name , tool choice: required or a named function, and parallel tool calls: false with a forced call are enforced by NeuroBrix's own grammar engine written in Rust : the model can only write what the schema or the template's tool-call format allows. A reasoning model answers after its reasoning. - Long conversations on a serving daemon. serve states its context ceiling when it starts; past the card's KV cache the oldest pages move to host memory and decoding goes on, exactly; a conversation that leaves the cache comes back from host memory and its next turn prefills only what is new; a request longer than the ceiling is refused by /v1/chat/completions with the OpenAI error 400 context length exceeded . A very long single prompt is answered slowly: 79 931 tokens took 39 minutes on two V100-32GB. - Mixture-of-experts models on several cards run at the cards' speed. On two V100-32GB, Qwen3-Coder-30B-A3B went from 6.7 s to 0.33 s a token through run , and a served 64-token answer from about 3 minutes to about 30 s; the daemon keeps such a model loaded between requests. - /v1/embeddings and an embedding model family , ready for the first embedding model of the catalogue. - A served request re-plans after a device out-of-memory , as run does; laptops with a hybrid CPU or an integrated GPU are described correctly; neurobrix import recognises a model already installed. - The hub is at neurobrix.org https://neurobrix.org and the documentation at docs.neurobrix.org https://docs.neurobrix.org . Contributions are made under the NeuroBrix Contributor Assignment Agreement https://github.com/NeuroBrix/neurobrix/blob/main/docs/legal/contributor-assignment-agreement.md . Install: pip install neurobrix the Triton engine, no PyTorch or pip install "neurobrix pytorch " ; on a Mac, python -m nbxtensor install-metal . The library is nbxtensor 0.1.0. The grammar engine is its own package, nbxgrammar 0.1.0, which pip install neurobrix installs on Linux x86 64 and aarch64, macOS arm64 and x86 64 and Windows x86 64; on any other platform no wheel of it exists and a constrained request is refused by name 501 constrained decoding not installed . The full list is in the CHANGELOG https://github.com/NeuroBrix/neurobrix/blob/v0.6.5/CHANGELOG.md 065---2026-11-10 .