Running Local LLMs with RamaLama and Docker on a Mac: A Hands-On Guide A developer tested RamaLama, an open-source CLI that runs large language models as OCI containers, on an Apple Silicon Mac with Docker. The tool simplifies local LLM deployment by pulling container images with llama.cpp and models from Hugging Face, and it includes an OpenAI-compatible server. The test revealed that on macOS, models inside Linux containers cannot access the GPU via Metal, so acceleration is unavailable. RamaLama runs large language models as OCI containers, so a single command ramalama run smollm:135m pulls a model and starts talking to it, with no Python environment to babysit. I spent an afternoon putting it through its paces on an Apple Silicon Mac Apple M4 Pro, 48 GB RAM, macOS 26.6 with Docker 29.4 provided by OrbStack. This guide is what I actually saw: the install, the first model, an OpenAI-compatible server, and the one macOS-specific catch that isn't obvious from the docs. Every command and number below is from that run, on RamaLama 0.24.0. RamaLama is an open-source CLI from the container-tooling community that treats models like container images. Instead of assembling an inference stack yourself, it pulls a hardened OCI image containing llama.cpp or vLLM/MLX plus your chosen model and runs it with Podman or Docker. If you've used Ollama the ergonomics feel familiar run , serve , list , pull , but the runtime and model live inside containers you can inspect and sign, and weights come straight from Hugging Face, Ollama, or any OCI registry. With Homebrew it's one command: brew install ramalama That pulled RamaLama 0.24.0 and, notably, its own copy of llama.cpp , ggml , and libomp as dependencies. Hold onto that detail; it matters for GPU acceleration later. Confirm the install: ramalama version ramalama version 0.24.0 You also need a container engine running. I used Docker through OrbStack; Podman works too and is RamaLama's default on Linux. The headline command: ramalama run smollm:135m "In one sentence, what is a Linux container?" Passing a prompt as an argument gives you one-shot output instead of dropping into a chat REPL. On first run this pulled the RamaLama container image, downloaded the model, and answered. smollm:135m resolves to hf://HuggingFaceTB/smollm-135M-instruct-v0.2-Q8 0-GGUF , a 138 MB, 8-bit quantized GGUF from Hugging Face. First-run wall-clock was 2 minutes 56 seconds, but almost all of that was downloads the ~1 GB image plus the model ; the 135M model itself is near-instant on CPU. It is also not smart: asked about containers it invented "2048-bit containers" and a docker-compose up -v command that doesn't exist. That's expected at 135M parameters. Use a model this small to validate your setup, not to do real work; a 1B model like llama3.2:1b a 770 MB Q4 K M download answers the same question correctly. For which models are actually worth running today, see the open-weight coding leaderboard shake-up https://www.techdevmantra.com/news/open-weight-model-cracks-webdev-leaderboard-top-3 . Check what you've downloaded: ramalama list SHORTNAME NAME SIZE llama3.2:1b hf://bartowski/Llama-3.2-1B-Instruct-GGUF 770.28 MB smollm:135m hf://HuggingFaceTB/smollm-135M-instruct-v0.2-Q8 0-GGUF 138.1 MB Models live under ~/.local/share/ramalama , separate from your container images. Before running anything for real, --dryrun prints the exact command without executing it: ramalama --dryrun run smollm:135m "hi" On my Mac that expands to a hardened docker run : docker run ... --security-opt=label=disable --cap-drop=all \ --security-opt=no-new-privileges --pull always -d -p 8080:8080 \ --init quay.io/ramalama/ramalama:0.24 \ llama-server --host :: --port 8080 --model /path/to/model --threads 7 ... Note what it does by default: drops all Linux capabilities, disables privilege escalation, and starts llama-server , the same server that backs the OpenAI-compatible API below. The base image quay.io/ramalama/ramalama:0.24 is about 1 GB, downloaded once and reused. Here is the part that trips people up. On Apple Silicon, a model running inside a Linux container cannot reach the Mac's GPU, because Docker's Linux VM has no path to Metal. ramalama info reports the container engine's accelerator as none : { "Accelerator": "none", "Config": { "runtimes": { "llama cpp": {}, "mlx": {} } } } So the default containerized run is CPU-only. Fine for a 135M toy, painful for anything larger. The fix is --nocontainer , which runs the host's llama.cpp the copy Homebrew installed directly: ramalama --nocontainer serve -p 8081 llama3.2:1b I benchmarked the difference on the same model and prompt. Served natively, Llama-3.2-1B Q4 K M loads straight onto the Apple GPU. Its startup log shows: load tensors: offloaded 17/17 layers to GPU ggml metal init: found device: Apple M4 Pro and it generated at ~206 tokens/sec. The same model served in the default container has no GPU to offload to and ran at ~102 tokens/sec on the CPU, about half the speed on this M4 Pro. RamaLama also exposes an mlx runtime if you'd rather use Apple's own inference framework than llama.cpp. | Mode | Command | Isolation | Acceleration | Llama-3.2-1B | |---|---|---|---|---| | Container default | ramalama run | Full OCI, cap-drop | CPU only | ~102 tok/s | | Native | ramalama --nocontainer run | None | Apple GPU Metal / MLX | ~206 tok/s | The trade-off is genuine: containers give you isolation and reproducibility; native gives you the GPU. On a Mac doing real work, --nocontainer is usually what you want. On Linux with an NVIDIA GPU, the container path keeps both. This is where RamaLama earns its place. serve starts the same llama-server as a local endpoint: ramalama serve -d --name tdm-lab -p 8080 smollm:135m It speaks the OpenAI API, so anything that talks to OpenAI can point at it: curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"smollm","messages": {"role":"user","content":"Say hello."} }' The response is standard OpenAI JSON: choices .message.content , a usage block, and timings . Swap the base URL in your existing OpenAI client and your app runs locally with no code changes. Stop it when done: ramalama stop tdm-lab If you'd rather wire a local endpoint into your editor, the same idea powers our guide on connecting Copilot Chat to a local API https://www.techdevmantra.com/guides/vs-codes-github-copilot-chat-lm-studio-local-api-for-offline-coding . If you already live in containers, yes. Its strengths are the security defaults cap-drop, no-new-privileges, signed OCI images , pulling from Hugging Face, Ollama, and OCI registries interchangeably, and the zero-friction OpenAI server. If you just want the fastest local chat on a Mac with a GUI, LM Studio https://www.techdevmantra.com/guides/lm-studio-guide-run-local-llms-on-macs is gentler. The two aren't mutually exclusive: I keep LM Studio for exploring and RamaLama for scripting reproducible, servable model runs. Plan for two things before you graduate from the toy model. Pick a real quantized model that fits your RAM a 7–8B Q4 model wants roughly 6–8 GB free , and on a Mac decide up front whether you're optimizing for isolation container, CPU or speed native, Apple GPU . brew install ramalama bundles llama.cpp ; needs Docker or Podman running. ramalama run