Run a private OpenAI-compatible LLM endpoint on Apple Silicon with one Rust binary

A developer has released Ferrum, an MIT-licensed Rust-based local LLM inference server that provides a private OpenAI-compatible endpoint on Apple Silicon. The tool, installable via Homebrew, supports models like Qwen3.5 and includes a 'doctor' command to check setup prerequisites. The developer is seeking feedback from Apple Silicon users on potential issues.

Disclosure: I maintain Ferrum https://github.com/sizzlecar/ferrum-infer-rs , an MIT-licensed local LLM inference server written in Rust. If you want a private OpenAI-compatible endpoint on an M-series Mac without setting up Python or a container, this is the shortest path I currently recommend. The commands below were tested end to end with the published Ferrum v0.8.3 Homebrew build on an M1 Max. brew tap sizzlecar/ferrum brew install ferrum ferrum doctor qwen3.5:4b-q4 k m ferrum doctor checks the local setup before inference. In one real install it caught missing Xcode Command Line Tools before the user reached a harder-to-diagnose failure. The first run downloads about 2.55 GiB. The terminal can look quiet during that download, so give it time before assuming it has hung. ferrum run qwen3.5:4b-q4 k m --disable-thinking Qwen3.5 emits verbose reasoning by default. --disable-thinking gives a more conventional first-chat experience. Omit the flag when you want the reasoning behavior. ferrum serve \ --model qwen3.5:4b-q4 k m \ --served-model-name ferrum \ --disable-thinking \ --port 8000 In another terminal: curl http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "ferrum", "messages": {"role": "user", "content": "Reply exactly: ferrum-ok"} , "max tokens": 32 }' A working setup returns HTTP 200 with a non-empty assistant response. I am looking for failure-oriented feedback from Apple Silicon users: doctor , the model download, or the first API request fail?Repository and issue tracker: github.com/sizzlecar/ferrum-infer-rs https://github.com/sizzlecar/ferrum-infer-rs This post was drafted with assistance from OpenAI Codex. The commands and stated behavior were checked against the published build; no comparative performance claim is being made here.