{"slug": "running-local-llms-with-ramalama-and-docker-on-a-mac-a-hands-on-guide", "title": "Running Local LLMs with RamaLama and Docker on a Mac: A Hands-On Guide", "summary": "A developer tested RamaLama, an open-source CLI that runs large language models as OCI containers, on an Apple Silicon Mac with Docker. The tool simplifies local LLM deployment by pulling container images with llama.cpp and models from Hugging Face, and it includes an OpenAI-compatible server. The test revealed that on macOS, models inside Linux containers cannot access the GPU via Metal, so acceleration is unavailable.", "body_md": "RamaLama runs large language models as OCI containers, so a single command (`ramalama run smollm:135m`\n\n) pulls a model and starts talking to it, with no Python environment to babysit. I spent an afternoon putting it through its paces on an Apple Silicon Mac (Apple M4 Pro, 48 GB RAM, macOS 26.6) with Docker 29.4 provided by OrbStack. This guide is what I actually saw: the install, the first model, an OpenAI-compatible server, and the one macOS-specific catch that isn't obvious from the docs. Every command and number below is from that run, on RamaLama 0.24.0.\n\nRamaLama is an open-source CLI from the container-tooling community that treats models like container images. Instead of assembling an inference stack yourself, it pulls a hardened OCI image containing llama.cpp (or vLLM/MLX) plus your chosen model and runs it with Podman or Docker. If you've used Ollama the ergonomics feel familiar (`run`\n\n, `serve`\n\n, `list`\n\n, `pull`\n\n), but the runtime and model live inside containers you can inspect and sign, and weights come straight from Hugging Face, Ollama, or any OCI registry.\n\nWith Homebrew it's one command:\n\n```\nbrew install ramalama\n```\n\nThat pulled RamaLama 0.24.0 and, notably, its own copy of `llama.cpp`\n\n, `ggml`\n\n, and `libomp`\n\nas dependencies. Hold onto that detail; it matters for GPU acceleration later. Confirm the install:\n\n```\nramalama version\n# ramalama version 0.24.0\n```\n\nYou also need a container engine running. I used Docker through OrbStack; Podman works too and is RamaLama's default on Linux.\n\nThe headline command:\n\n```\nramalama run smollm:135m \"In one sentence, what is a Linux container?\"\n```\n\nPassing a prompt as an argument gives you one-shot output instead of dropping into a chat REPL. On first run this pulled the RamaLama container image, downloaded the model, and answered. `smollm:135m`\n\nresolves to `hf://HuggingFaceTB/smollm-135M-instruct-v0.2-Q8_0-GGUF`\n\n, a 138 MB, 8-bit quantized GGUF from Hugging Face.\n\nFirst-run wall-clock was 2 minutes 56 seconds, but almost all of that was downloads (the ~1 GB image plus the model); the 135M model itself is near-instant on CPU. It is also not smart: asked about containers it invented \"2048-bit containers\" and a `docker-compose up -v`\n\ncommand that doesn't exist. That's expected at 135M parameters. Use a model this small to validate your setup, not to do real work; a 1B model like `llama3.2:1b`\n\n(a 770 MB Q4_K_M download) answers the same question correctly. (For which models are actually worth running today, see the [open-weight coding leaderboard shake-up](https://www.techdevmantra.com/news/open-weight-model-cracks-webdev-leaderboard-top-3).)\n\nCheck what you've downloaded:\n\n```\nramalama list\n# SHORTNAME    NAME                                                    SIZE\n# llama3.2:1b  hf://bartowski/Llama-3.2-1B-Instruct-GGUF               770.28 MB\n# smollm:135m  hf://HuggingFaceTB/smollm-135M-instruct-v0.2-Q8_0-GGUF  138.1 MB\n```\n\nModels live under `~/.local/share/ramalama`\n\n, separate from your container images.\n\nBefore running anything for real, `--dryrun`\n\nprints the exact command without executing it:\n\n```\nramalama --dryrun run smollm:135m \"hi\"\n```\n\nOn my Mac that expands to a hardened `docker run`\n\n:\n\n```\ndocker run ... --security-opt=label=disable --cap-drop=all \\\n  --security-opt=no-new-privileges --pull always -d -p 8080:8080 \\\n  --init quay.io/ramalama/ramalama:0.24 \\\n  llama-server --host :: --port 8080 --model /path/to/model --threads 7 ...\n```\n\nNote what it does by default: drops all Linux capabilities, disables privilege escalation, and starts `llama-server`\n\n, the same server that backs the OpenAI-compatible API below. The base image (`quay.io/ramalama/ramalama:0.24`\n\n) is about 1 GB, downloaded once and reused.\n\nHere is the part that trips people up. On Apple Silicon, a model running inside a Linux container cannot reach the Mac's GPU, because Docker's Linux VM has no path to Metal. `ramalama info`\n\nreports the container engine's accelerator as `none`\n\n:\n\n```\n{ \"Accelerator\": \"none\", \"Config\": { \"runtimes\": { \"llama_cpp\": {}, \"mlx\": {} } } }\n```\n\nSo the default containerized run is CPU-only. Fine for a 135M toy, painful for anything larger. The fix is `--nocontainer`\n\n, which runs the host's llama.cpp (the copy Homebrew installed) directly:\n\n```\nramalama --nocontainer serve -p 8081 llama3.2:1b\n```\n\nI benchmarked the difference on the same model and prompt. Served natively, Llama-3.2-1B (Q4_K_M) loads straight onto the Apple GPU. Its startup log shows:\n\n```\nload_tensors: offloaded 17/17 layers to GPU\nggml_metal_init: found device: Apple M4 Pro\n```\n\nand it generated at ~206 tokens/sec. The same model served in the default container has no GPU to offload to and ran at ~102 tokens/sec on the CPU, about half the speed on this M4 Pro. RamaLama also exposes an `mlx`\n\nruntime if you'd rather use Apple's own inference framework than llama.cpp.\n\n| Mode | Command | Isolation | Acceleration | Llama-3.2-1B |\n|---|---|---|---|---|\n| Container (default) | `ramalama run` |\nFull (OCI, cap-drop) | CPU only | ~102 tok/s |\n| Native | `ramalama --nocontainer run` |\nNone | Apple GPU (Metal) / MLX | ~206 tok/s |\n\nThe trade-off is genuine: containers give you isolation and reproducibility; native gives you the GPU. On a Mac doing real work, `--nocontainer`\n\nis usually what you want. On Linux with an NVIDIA GPU, the container path keeps both.\n\nThis is where RamaLama earns its place. `serve`\n\nstarts the same `llama-server`\n\nas a local endpoint:\n\n```\nramalama serve -d --name tdm-lab -p 8080 smollm:135m\n```\n\nIt speaks the OpenAI API, so anything that talks to OpenAI can point at it:\n\n```\ncurl http://localhost:8080/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"model\":\"smollm\",\"messages\":[{\"role\":\"user\",\"content\":\"Say hello.\"}]}'\n```\n\nThe response is standard OpenAI JSON: `choices[].message.content`\n\n, a `usage`\n\nblock, and `timings`\n\n. Swap the base URL in your existing OpenAI client and your app runs locally with no code changes. Stop it when done:\n\n```\nramalama stop tdm-lab\n```\n\nIf you'd rather wire a local endpoint into your editor, the same idea powers our guide on [connecting Copilot Chat to a local API](https://www.techdevmantra.com/guides/vs-codes-github-copilot-chat-lm-studio-local-api-for-offline-coding).\n\nIf you already live in containers, yes. Its strengths are the security defaults (cap-drop, no-new-privileges, signed OCI images), pulling from Hugging Face, Ollama, and OCI registries interchangeably, and the zero-friction OpenAI server. If you just want the fastest local chat on a Mac with a GUI, [LM Studio](https://www.techdevmantra.com/guides/lm-studio-guide-run-local-llms-on-macs) is gentler. The two aren't mutually exclusive: I keep LM Studio for exploring and RamaLama for scripting reproducible, servable model runs.\n\nPlan for two things before you graduate from the toy model. Pick a real quantized model that fits your RAM (a 7–8B Q4 model wants roughly 6–8 GB free), and on a Mac decide up front whether you're optimizing for isolation (container, CPU) or speed (native, Apple GPU).\n\n`brew install ramalama`\n\n(bundles llama.cpp); needs Docker or Podman running.`ramalama run <model> \"prompt\"`\n\nfor one-shot output; models come from Hugging Face, Ollama, or OCI registries.`ramalama --dryrun run <model>`\n\nprints the exact hardened `docker run`\n\n.`--nocontainer`\n\noffloads to the Apple GPU (Metal). On this M4 Pro, Llama-3.2-1B ran ~206 tok/s native versus ~102 tok/s in-container, about 2x faster.`ramalama serve`\n\nexposes a drop-in OpenAI-compatible API on port 8080.", "url": "https://wpnews.pro/news/running-local-llms-with-ramalama-and-docker-on-a-mac-a-hands-on-guide", "canonical_source": "https://dev.to/shubham_sharma_94/running-local-llms-with-ramalama-and-docker-on-a-mac-a-hands-on-guide-140p", "published_at": "2026-08-31 12:24:50+00:00", "updated_at": "2026-08-31 12:52:27.817586+00:00", "lang": "en", "topics": ["developer-tools", "large-language-models", "ai-tools", "mlops"], "entities": ["RamaLama", "Docker", "OrbStack", "Hugging Face", "llama.cpp", "Apple Silicon", "Homebrew", "Podman"], "alternates": {"html": "https://wpnews.pro/news/running-local-llms-with-ramalama-and-docker-on-a-mac-a-hands-on-guide", "markdown": "https://wpnews.pro/news/running-local-llms-with-ramalama-and-docker-on-a-mac-a-hands-on-guide.md", "text": "https://wpnews.pro/news/running-local-llms-with-ramalama-and-docker-on-a-mac-a-hands-on-guide.txt", "jsonld": "https://wpnews.pro/news/running-local-llms-with-ramalama-and-docker-on-a-mac-a-hands-on-guide.jsonld"}}