The trade-off between instruction following and raw reasoning is killing me. I tried Gemma 2 (which is fast and handles reasoning well), but the tool calling is hit-or-miss. Then I shifted to Qwen 2.5 7B/14B (MLX versions), and while the tool calling is actually reliable, the generic reasoning feels a step down from Gemma.
If you're targeting a 24GB footprint, you have to account for the OS overhead and the KV cache. You're effectively looking at about 16-18GB of usable VRAM for the model.
The Model Dilemma #
For a real-world AI workflow where you need both reasoning and strict tool adherence, I've found that Mistral NeMo 12B is usually the sweet spot for this specific hardware. It fits comfortably in memory and doesn't suffer from the "lobotomy" effect that happens when you squeeze a larger model into 4-bit quantization.
If you're still struggling with tool calling, I'd suggest tweaking the system prompt to be extremely explicit about the JSON schema. Instead of relying on the model's native tool-calling capability, force a structured output.
The Ollama Reranker Crash #
The bigger headache right now is the reranker. I've been trying to integrate a reranking step into my RAG pipeline using Ollama, but qllama/bge-reranker-v2-m3:Q4_K_M
keeps crashing my instance.
The error usually looks something like this in the logs:
sigkill: process terminated by signal 9 (out of memory)
Since rerankers are cross-encoders, they process pairs of documents, which can spike memory usage unexpectedly. On a 24GB Mac, if you already have a 12B-14B model loaded in the GPU, a reranker on top of it often pushes the system over the edge, triggering the macOS OOM killer.To fix this, I've had to implement a strict "unload" logic or use a smaller embedding model for initial retrieval and skip the reranker until I can find a more stable implementation.
Current Config Attempt #
Here is the basic setup I'm testing to see if I can stabilize the memory:
model: "mistral-nemo"
parameters:
num_ctx: 8192
num_gpu: 1
temperature: 0.2
top_p: 0.9
If anyone has a stable reranking setup on M4 Pro that doesn't nukes the Ollama process, I'm all ears. I need something that can handle the bge-reranker
logic without eating all 24GB of unified memory.
Next Open Weight Models: Why Big Tech is Fighting Restrictions →