{"slug": "run-a-private-openai-compatible-llm-endpoint-on-apple-silicon-with-one-rust", "title": "Run a private OpenAI-compatible LLM endpoint on Apple Silicon with one Rust binary", "summary": "A developer has released Ferrum, an MIT-licensed Rust-based local LLM inference server that provides a private OpenAI-compatible endpoint on Apple Silicon. The tool, installable via Homebrew, supports models like Qwen3.5 and includes a 'doctor' command to check setup prerequisites. The developer is seeking feedback from Apple Silicon users on potential issues.", "body_md": "Disclosure: I maintain [Ferrum](https://github.com/sizzlecar/ferrum-infer-rs), an MIT-licensed local LLM inference server written in Rust.\n\nIf you want a private OpenAI-compatible endpoint on an M-series Mac without setting up Python or a container, this is the shortest path I currently recommend.\n\nThe commands below were tested end to end with the published Ferrum v0.8.3 Homebrew build on an M1 Max.\n\n```\nbrew tap sizzlecar/ferrum\nbrew install ferrum\nferrum doctor qwen3.5:4b-q4_k_m\n```\n\n`ferrum doctor`\n\nchecks the local setup before inference. In one real install it caught missing Xcode Command Line Tools before the user reached a harder-to-diagnose failure.\n\n**The first run downloads about 2.55 GiB.** The terminal can look quiet during that download, so give it time before assuming it has hung.\n\n```\nferrum run qwen3.5:4b-q4_k_m --disable-thinking\n```\n\nQwen3.5 emits verbose reasoning by default. `--disable-thinking`\n\ngives a more conventional first-chat experience. Omit the flag when you want the reasoning behavior.\n\n```\nferrum serve \\\n  --model qwen3.5:4b-q4_k_m \\\n  --served-model-name ferrum \\\n  --disable-thinking \\\n  --port 8000\n```\n\nIn another terminal:\n\n```\ncurl http://localhost:8000/v1/chat/completions \\\n  -H 'Content-Type: application/json' \\\n  -d '{\n    \"model\": \"ferrum\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Reply exactly: ferrum-ok\"}],\n    \"max_tokens\": 32\n  }'\n```\n\nA working setup returns HTTP 200 with a non-empty assistant response.\n\nI am looking for failure-oriented feedback from Apple Silicon users:\n\n`doctor`\n\n, the model download, or the first API request fail?Repository and issue tracker: [github.com/sizzlecar/ferrum-infer-rs](https://github.com/sizzlecar/ferrum-infer-rs)\n\n*This post was drafted with assistance from OpenAI Codex. The commands and stated behavior were checked against the published build; no comparative performance claim is being made here.*", "url": "https://wpnews.pro/news/run-a-private-openai-compatible-llm-endpoint-on-apple-silicon-with-one-rust", "canonical_source": "https://dev.to/jinxuan_ai/run-a-private-openai-compatible-llm-endpoint-on-apple-silicon-with-one-rust-binary-29h0", "published_at": "2026-09-02 03:12:46+00:00", "updated_at": "2026-09-02 03:52:49.900102+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure", "large-language-models"], "entities": ["Ferrum", "Qwen3.5", "Apple Silicon", "Homebrew", "OpenAI", "Codex"], "alternates": {"html": "https://wpnews.pro/news/run-a-private-openai-compatible-llm-endpoint-on-apple-silicon-with-one-rust", "markdown": "https://wpnews.pro/news/run-a-private-openai-compatible-llm-endpoint-on-apple-silicon-with-one-rust.md", "text": "https://wpnews.pro/news/run-a-private-openai-compatible-llm-endpoint-on-apple-silicon-with-one-rust.txt", "jsonld": "https://wpnews.pro/news/run-a-private-openai-compatible-llm-endpoint-on-apple-silicon-with-one-rust.jsonld"}}