Local LLM Selection for Mac Mini M4 Pro 24GB

A developer reports that Mistral NeMo 12B is the best local LLM for a Mac Mini M4 Pro with 24GB of unified memory, balancing reasoning and tool calling, but warns that adding a reranker like qllama/bge-reranker-v2-m3:Q4_K_M to an Ollama pipeline causes out-of-memory crashes due to macOS's OOM killer. The user recommends keeping context low and implementing strict unload logic to stabilize memory usage.

Local LLM Selection for Mac Mini M4 Pro 24GB The trade-off between instruction following and raw reasoning is killing me. I tried Gemma 2 which is fast and handles reasoning well , but the tool calling is hit-or-miss. Then I shifted to Qwen 2.5 7B/14B MLX versions , and while the tool calling is actually reliable, the generic reasoning feels a step down from Gemma. If you're targeting a 24GB footprint, you have to account for the OS overhead and the KV cache. You're effectively looking at about 16-18GB of usable VRAM for the model. The Model Dilemma For a real-world AI workflow where you need both reasoning and strict tool adherence, I've found that Mistral NeMo 12B is usually the sweet spot for this specific hardware. It fits comfortably in memory and doesn't suffer from the "lobotomy" effect that happens when you squeeze a larger model into 4-bit quantization. If you're still struggling with tool calling, I'd suggest tweaking the system prompt to be extremely explicit about the JSON schema. Instead of relying on the model's native tool-calling capability, force a structured output. The Ollama Reranker Crash The bigger headache right now is the reranker. I've been trying to integrate a reranking step into my RAG /en/tags/rag/ pipeline using Ollama, but qllama/bge-reranker-v2-m3:Q4 K M keeps crashing my instance. The error usually looks something like this in the logs: Typical Ollama crash signal during embedding/reranking sigkill: process terminated by signal 9 out of memory Since rerankers are cross-encoders, they process pairs of documents, which can spike memory usage unexpectedly. On a 24GB Mac, if you already have a 12B-14B model loaded in the GPU, loading a reranker on top of it often pushes the system over the edge, triggering the macOS OOM killer.To fix this, I've had to implement a strict "unload" logic or use a smaller embedding model for initial retrieval and skip the reranker until I can find a more stable implementation. Current Config Attempt Here is the basic setup I'm testing to see if I can stabilize the memory: My current local deployment attempt model: "mistral-nemo" parameters: num ctx: 8192 num gpu: 1 temperature: 0.2 top p: 0.9 Trying to keep ctx low to leave room for the reranker If anyone has a stable reranking setup on M4 Pro that doesn't nukes the Ollama process, I'm all ears. I need something that can handle the bge-reranker logic without eating all 24GB of unified memory. Next Open Weight Models: Why Big Tech is Fighting Restrictions → /en/threads/2859/