Run a 4B LLM model locally on your mac

A developer has published a step-by-step guide for running a 4B parameter LLM locally on a Mac, using Mozilla's llamafile and a fine-tuned Qwen3 model from Hugging Face. The process involves downloading llamafile v0.10.5 and the Qwen3-4B GGUF file, creating a shell script, and removing macOS quarantine attributes to run the model as a local server on port 8080.

| Run a 4B LLM model locally on your mac, this has been tested on MacOS Tahoe | | | Step 1: | | | Lookup "llamafile github" or go to https://github.com/mozilla-ai/llamafile | | | Download the latest release, as of this it is llamafile v0.10.5 | | | Step 2: | | | Go to https://huggingface.co/models | | | Search for qween3 | | | Click on pramodlohra/Qween3 4B thinking finetune | | | Go to Files and versions | | | Download qwen3-4b-thinking-2507.Q4 K M.gguf | | | Step 3: | | | Create a folder called qwen | | | Copy llamafile v0.10.5 and qwen3-4b-thinking-2507.Q4 K M.gguf into the folder | | | Step 4: | | | In this folder, create a file called run qwen.command | | | Write the following lines in it | | | /bin/bash | | | cd "$ dirname "$0" " | | | ./llamafile-0.10.5 --server --model ./qwen3-4b-thinking-2507.Q4 K M.gguf | | | Save the run qwen.command file | | | Step 5: | | | Run chmod for llamafile llamafile-0.10.5 and run qwen.command | | | chmod 755 llamafile-0.10.5 | | | chmod 755 run qwen.command | | | Step 6: | | | Remove the quarantine flag from running this in the terminal | | | xattr -d com.apple.quarantine run qwen.command | | | xattr -d com.apple.quarantine llamafile-0.10.5 | | | You can check whether quarantine is still present with: | | | xattr -l run qwen.command | | | xattr -l llamafile-0.10.5 | | | If com.apple.quarantine doesn't appear, you're good. | | | NOTE: Don't disable Gatekeeper globally with commands such as sudo spctl --master-disable. | | | There's absolutely no need for that here; removing the quarantine attribute from these specific files is much safer. | | | Now you are ready to click on the run qwen.command file and run the model locally | | | If everything is done correctly, this is what would show up in the terminal. | | | % /Users/santa/Downloads/qwen/run qwen.command ; exit; | | | 0.00.000.673 I cmn common param: common params print info: verbosity = 3 adjust with the -lv N CLI arg | | | 0.00.000.921 W srv llama server: ----------------- | | | 0.00.000.926 W srv llama server: CORS is set to allow all origins ' ' and no API key is set | | | 0.00.000.926 W srv llama server: this can be a security risk cross-origin attacks | | | 0.00.000.927 W srv llama server: more info: https://github.com/ggml-org/llama.cpp/pull/25655 | | | 0.00.000.927 W srv llama server: ----------------- | | | 0.00.001.045 W srv llama server: sandbox: disabled in GPU mode | | | 0.00.002.429 I srv load model: loading model './qwen3-4b-thinking-2507.Q4 K M.gguf' | | | 0.00.429.537 W load: control-looking token: 128247 '</s ' was not control-type; this is probably a bug in the model. its type will be overridden | | | 0.04.590.135 I srv load model: initializing, n slots = 4, n ctx slot = 14080, kv unified = 'true' | | | 0.04.600.756 I srv llama server: model loaded | | | 0.04.600.778 I srv llama server: listening on http://127.0.0.1:8080 | | | You can start using the model on 127.0.0.1:8080, Enjoy |