Show HN: NeuroBrix – PyTorch-free local inference with a Rust grammar engine

Repository navigation #

NeuroBrix 0.6.5 brings constrained decoding and a faster, longer-reaching serving daemon, with nbxtensor 0.1.0 and nbxgrammar 0.1.0.

• Constrained decoding on the OpenAI API.response_format: json_schema (a subset of JSON Schema; a constraining keyword outside it is refused by name),tool_choice: required or a named function, andparallel_tool_calls: false with a forced call are enforced by NeuroBrix's own grammar engine (written in Rust): the model can only write what the schema or the template's tool-call format allows. A reasoning model answers after its reasoning.
• Long conversations on a serving daemon.serve states its context ceiling when it starts; past the card's KV cache the oldest pages move to host memory and decoding goes on, exactly; a conversation that leaves the cache comes back from host memory and its next turn prefills only what is new; a request longer than the ceiling is refused by /v1/chat/completions with the OpenAI error400 context_length_exceeded . A very long single prompt is answered slowly: 79 931 tokens took 39 minutes on two V100-32GB.

- **Mixture-of-experts models on several cards run at the cards' speed.** On two V100-32GB, Qwen3-Coder-30B-A3B went
 from 6.7 s to 0.33 s a token through`run` , and a served 64-token answer from about 3 minutes to about 30 s; the

daemon keeps such a model loaded between requests.

• /v1/embeddings and an embedding model family , ready for the first embedding model of the catalogue.
• A served request re-plans after a device out-of-memory , asrun does; laptops with a hybrid CPU or an integrated GPU are described correctly;neurobrix import recognises a model already installed.

- **The hub is at [neurobrix.org](https://neurobrix.org)** and the documentation at
 [docs.neurobrix.org](https://docs.neurobrix.org) . Contributions are made under the

NeuroBrix Contributor Assignment Agreement .

Install: pip install neurobrix (the Triton engine, no PyTorch) or pip install "neurobrix[pytorch]"; on a Mac, python -m nbxtensor install-metal. The library is nbxtensor 0.1.0.

The grammar engine is its own package, nbxgrammar 0.1.0, which pip install neurobrix installs on Linux x86_64 and

aarch64, macOS arm64 and x86_64 and Windows x86_64; on any other platform no wheel of it exists and a constrained

request is refused by name (501 constrained_decoding_not_installed).

The full list is in the CHANGELOG.