Disclosure: I maintain Ferrum, an MIT-licensed local LLM inference server written in Rust.
If you want a private OpenAI-compatible endpoint on an M-series Mac without setting up Python or a container, this is the shortest path I currently recommend.
The commands below were tested end to end with the published Ferrum v0.8.3 Homebrew build on an M1 Max.
brew tap sizzlecar/ferrum
brew install ferrum
ferrum doctor qwen3.5:4b-q4_k_m
ferrum doctor
checks the local setup before inference. In one real install it caught missing Xcode Command Line Tools before the user reached a harder-to-diagnose failure.
The first run downloads about 2.55 GiB. The terminal can look quiet during that download, so give it time before assuming it has hung.
ferrum run qwen3.5:4b-q4_k_m --disable-thinking
Qwen3.5 emits verbose reasoning by default. --disable-thinking
gives a more conventional first-chat experience. Omit the flag when you want the reasoning behavior.
ferrum serve \
--model qwen3.5:4b-q4_k_m \
--served-model-name ferrum \
--disable-thinking \
--port 8000
In another terminal:
curl http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "ferrum",
"messages": [{"role": "user", "content": "Reply exactly: ferrum-ok"}],
"max_tokens": 32
}'
A working setup returns HTTP 200 with a non-empty assistant response.
I am looking for failure-oriented feedback from Apple Silicon users:
doctor
, the model download, or the first API request fail?Repository and issue tracker: github.com/sizzlecar/ferrum-infer-rs
This post was drafted with assistance from OpenAI Codex. The commands and stated behavior were checked against the published build; no comparative performance claim is being made here.