{"slug": "local-llm-selection-for-mac-mini-m4-pro-24gb", "title": "Local LLM Selection for Mac Mini M4 Pro 24GB", "summary": "A developer reports that Mistral NeMo 12B is the best local LLM for a Mac Mini M4 Pro with 24GB of unified memory, balancing reasoning and tool calling, but warns that adding a reranker like qllama/bge-reranker-v2-m3:Q4_K_M to an Ollama pipeline causes out-of-memory crashes due to macOS's OOM killer. The user recommends keeping context low and implementing strict unload logic to stabilize memory usage.", "body_md": "# Local LLM Selection for Mac Mini M4 Pro 24GB\n\nThe trade-off between instruction following and raw reasoning is killing me. I tried Gemma 2 (which is fast and handles reasoning well), but the tool calling is hit-or-miss. Then I shifted to Qwen 2.5 7B/14B (MLX versions), and while the tool calling is actually reliable, the generic reasoning feels a step down from Gemma.\n\nIf you're targeting a 24GB footprint, you have to account for the OS overhead and the KV cache. You're effectively looking at about 16-18GB of usable VRAM for the model.\n\n## The Model Dilemma\n\nFor a real-world AI workflow where you need both reasoning and strict tool adherence, I've found that Mistral NeMo 12B is usually the sweet spot for this specific hardware. It fits comfortably in memory and doesn't suffer from the \"lobotomy\" effect that happens when you squeeze a larger model into 4-bit quantization.\n\nIf you're still struggling with tool calling, I'd suggest tweaking the system prompt to be extremely explicit about the JSON schema. Instead of relying on the model's native tool-calling capability, force a structured output.\n\n## The Ollama Reranker Crash\n\nThe bigger headache right now is the reranker. I've been trying to integrate a reranking step into my [RAG](/en/tags/rag/) pipeline using Ollama, but `qllama/bge-reranker-v2-m3:Q4_K_M`\n\nkeeps crashing my instance.\n\nThe error usually looks something like this in the logs:\n\n```\n# Typical Ollama crash signal during embedding/reranking\nsigkill: process terminated by signal 9 (out of memory)\n```\n\nSince rerankers are cross-encoders, they process pairs of documents, which can spike memory usage unexpectedly. On a 24GB Mac, if you already have a 12B-14B model loaded in the GPU, loading a reranker on top of it often pushes the system over the edge, triggering the macOS OOM killer.To fix this, I've had to implement a strict \"unload\" logic or use a smaller embedding model for initial retrieval and skip the reranker until I can find a more stable implementation.\n\n## Current Config Attempt\n\nHere is the basic setup I'm testing to see if I can stabilize the memory:\n\n```\n# My current local deployment attempt\nmodel: \"mistral-nemo\"\nparameters:\n  num_ctx: 8192\n  num_gpu: 1\n  temperature: 0.2\n  top_p: 0.9\n# Trying to keep ctx low to leave room for the reranker\n```\n\nIf anyone has a stable reranking setup on M4 Pro that doesn't nukes the Ollama process, I'm all ears. I need something that can handle the `bge-reranker`\n\nlogic without eating all 24GB of unified memory.\n\n[Next Open Weight Models: Why Big Tech is Fighting Restrictions →](/en/threads/2859/)", "url": "https://wpnews.pro/news/local-llm-selection-for-mac-mini-m4-pro-24gb", "canonical_source": "https://promptcube3.com/en/threads/2863/", "published_at": "2026-07-24 18:47:29+00:00", "updated_at": "2026-07-24 19:08:06.276940+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-infrastructure"], "entities": ["Mac Mini M4 Pro", "Mistral NeMo 12B", "Gemma 2", "Qwen 2.5", "Ollama", "qllama/bge-reranker-v2-m3"], "alternates": {"html": "https://wpnews.pro/news/local-llm-selection-for-mac-mini-m4-pro-24gb", "markdown": "https://wpnews.pro/news/local-llm-selection-for-mac-mini-m4-pro-24gb.md", "text": "https://wpnews.pro/news/local-llm-selection-for-mac-mini-m4-pro-24gb.txt", "jsonld": "https://wpnews.pro/news/local-llm-selection-for-mac-mini-m4-pro-24gb.jsonld"}}