RamaLama runs large language models as OCI containers, so a single command (ramalama run smollm:135m
) pulls a model and starts talking to it, with no Python environment to babysit. I spent an afternoon putting it through its paces on an Apple Silicon Mac (Apple M4 Pro, 48 GB RAM, macOS 26.6) with Docker 29.4 provided by OrbStack. This guide is what I actually saw: the install, the first model, an OpenAI-compatible server, and the one macOS-specific catch that isn't obvious from the docs. Every command and number below is from that run, on RamaLama 0.24.0.
RamaLama is an open-source CLI from the container-tooling community that treats models like container images. Instead of assembling an inference stack yourself, it pulls a hardened OCI image containing llama.cpp (or vLLM/MLX) plus your chosen model and runs it with Podman or Docker. If you've used Ollama the ergonomics feel familiar (run
, serve
, list
, pull
), but the runtime and model live inside containers you can inspect and sign, and weights come straight from Hugging Face, Ollama, or any OCI registry.
With Homebrew it's one command:
brew install ramalama
That pulled RamaLama 0.24.0 and, notably, its own copy of llama.cpp
, ggml
, and libomp
as dependencies. Hold onto that detail; it matters for GPU acceleration later. Confirm the install:
ramalama version
You also need a container engine running. I used Docker through OrbStack; Podman works too and is RamaLama's default on Linux.
The headline command:
ramalama run smollm:135m "In one sentence, what is a Linux container?"
Passing a prompt as an argument gives you one-shot output instead of dropping into a chat REPL. On first run this pulled the RamaLama container image, downloaded the model, and answered. smollm:135m
resolves to hf://HuggingFaceTB/smollm-135M-instruct-v0.2-Q8_0-GGUF
, a 138 MB, 8-bit quantized GGUF from Hugging Face.
First-run wall-clock was 2 minutes 56 seconds, but almost all of that was downloads (the ~1 GB image plus the model); the 135M model itself is near-instant on CPU. It is also not smart: asked about containers it invented "2048-bit containers" and a docker-compose up -v
command that doesn't exist. That's expected at 135M parameters. Use a model this small to validate your setup, not to do real work; a 1B model like llama3.2:1b
(a 770 MB Q4_K_M download) answers the same question correctly. (For which models are actually worth running today, see the open-weight coding leaderboard shake-up.)
Check what you've downloaded:
ramalama list
Models live under ~/.local/share/ramalama
, separate from your container images.
Before running anything for real, --dryrun
prints the exact command without executing it:
ramalama --dryrun run smollm:135m "hi"
On my Mac that expands to a hardened docker run
:
docker run ... --security-opt=label=disable --cap-drop=all \
--security-opt=no-new-privileges --pull always -d -p 8080:8080 \
--init quay.io/ramalama/ramalama:0.24 \
llama-server --host :: --port 8080 --model /path/to/model --threads 7 ...
Note what it does by default: drops all Linux capabilities, disables privilege escalation, and starts llama-server
, the same server that backs the OpenAI-compatible API below. The base image (quay.io/ramalama/ramalama:0.24
) is about 1 GB, downloaded once and reused.
Here is the part that trips people up. On Apple Silicon, a model running inside a Linux container cannot reach the Mac's GPU, because Docker's Linux VM has no path to Metal. ramalama info
reports the container engine's accelerator as none
:
{ "Accelerator": "none", "Config": { "runtimes": { "llama_cpp": {}, "mlx": {} } } }
So the default containerized run is CPU-only. Fine for a 135M toy, painful for anything larger. The fix is --nocontainer
, which runs the host's llama.cpp (the copy Homebrew installed) directly:
ramalama --nocontainer serve -p 8081 llama3.2:1b
I benchmarked the difference on the same model and prompt. Served natively, Llama-3.2-1B (Q4_K_M) loads straight onto the Apple GPU. Its startup log shows:
load_tensors: offloaded 17/17 layers to GPU
ggml_metal_init: found device: Apple M4 Pro
and it generated at ~206 tokens/sec. The same model served in the default container has no GPU to offload to and ran at ~102 tokens/sec on the CPU, about half the speed on this M4 Pro. RamaLama also exposes an mlx
runtime if you'd rather use Apple's own inference framework than llama.cpp.
| Mode | Command | Isolation | Acceleration | Llama-3.2-1B |
|---|---|---|---|---|
| Container (default) | ramalama run |
|||
| Full (OCI, cap-drop) | CPU only | ~102 tok/s | ||
| Native | ramalama --nocontainer run |
|||
| None | Apple GPU (Metal) / MLX | ~206 tok/s |
The trade-off is genuine: containers give you isolation and reproducibility; native gives you the GPU. On a Mac doing real work, --nocontainer
is usually what you want. On Linux with an NVIDIA GPU, the container path keeps both.
This is where RamaLama earns its place. serve
starts the same llama-server
as a local endpoint:
ramalama serve -d --name tdm-lab -p 8080 smollm:135m
It speaks the OpenAI API, so anything that talks to OpenAI can point at it:
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"smollm","messages":[{"role":"user","content":"Say hello."}]}'
The response is standard OpenAI JSON: choices[].message.content
, a usage
block, and timings
. Swap the base URL in your existing OpenAI client and your app runs locally with no code changes. Stop it when done:
ramalama stop tdm-lab
If you'd rather wire a local endpoint into your editor, the same idea powers our guide on connecting Copilot Chat to a local API.
If you already live in containers, yes. Its strengths are the security defaults (cap-drop, no-new-privileges, signed OCI images), pulling from Hugging Face, Ollama, and OCI registries interchangeably, and the zero-friction OpenAI server. If you just want the fastest local chat on a Mac with a GUI, LM Studio is gentler. The two aren't mutually exclusive: I keep LM Studio for exploring and RamaLama for scripting reproducible, servable model runs.
Plan for two things before you graduate from the toy model. Pick a real quantized model that fits your RAM (a 7–8B Q4 model wants roughly 6–8 GB free), and on a Mac decide up front whether you're optimizing for isolation (container, CPU) or speed (native, Apple GPU).
brew install ramalama
(bundles llama.cpp); needs Docker or Podman running.ramalama run <model> "prompt"
for one-shot output; models come from Hugging Face, Ollama, or OCI registries.ramalama --dryrun run <model>
prints the exact hardened docker run
.--nocontainer
offloads to the Apple GPU (Metal). On this M4 Pro, Llama-3.2-1B ran ~206 tok/s native versus ~102 tok/s in-container, about 2x faster.ramalama serve
exposes a drop-in OpenAI-compatible API on port 8080.