{"slug": "running-local-ai-models", "title": "Running local AI models", "summary": "A practitioner running Ollama on an 8GB M1 MacBook Pro reported that quantized 1B–2B parameter models such as Qwen 0.6B and 1.7B remain usable for narrow tasks like classification, information extraction and structured JSON output, while 7B–8B models are a poor fit for the hardware. The practitioner measured roughly 1.5 to 1.9 seconds on some simple tasks, depending on prompt and output length, and recommended monitoring Memory Pressure and Swap Used in Activity Monitor because 8GB fills quickly with browsers and other apps running. The account concludes that local inference on 8GB is possible if expectations are kept realistic, and that hosted inference is less frustrating for more demanding work.", "body_md": "I’m running Ollama on an 8GB M1 MacBook Pro, so pretty similar constraints to yours!\n\nI’ve experimented with smaller Qwen models (0.6B and 1.7B), mainly for classification, information extraction and structured JSON outputs.\n\nA few things I’ve learned:\n\nSmaller models are surprisingly usable if you give them specific, narrow tasks. I wouldn’t expect them to perform like Claude or ChatGPT, though.\n\nI use Activity Monitor to keep an eye on Memory Pressure and Swap Used. The 8GB fills up pretty quickly, especially with browsers and other apps running.\n\nQuantized models help considerably. I’d start with something around 1B to 2B parameters rather than jumping straight into 7B or 8B models.\n\nFor more demanding tasks, hosted inference is definitely less frustrating.\n\nMy smaller models have managed around 1.5 to 1.9 seconds on some simple tasks, although that obviously depends on the prompt and output length.\n\nI’m still learning and experimenting myself, but my biggest takeaway is that local inference on 8GB is absolutely possible. You just have to be realistic about what you’re asking the model to do.\n\nCurious what models you’re planning to try!", "url": "https://wpnews.pro/news/running-local-ai-models", "canonical_source": "https://discuss.huggingface.co/t/running-local-ai-models/192189#post_3", "published_at": "2026-11-20 09:25:14+00:00", "updated_at": "2026-11-20 09:43:32.732583+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools"], "entities": ["Ollama", "Apple M1 MacBook Pro", "Qwen", "Claude", "ChatGPT", "Activity Monitor"], "also_reported_by": [], "takeaway": "On an 8GB M1 MacBook Pro, quantized 1B–2B models such as Qwen 0.6B and 1.7B handled narrow classification, extraction and JSON tasks in about 1.5–1.9 seconds, while 7B–8B models were not recommended for the hardware.", "recommendation": {"advice": "On 8GB machines, start with quantized 1B–2B models for narrow tasks and watch Memory Pressure and Swap Used in Activity Monitor before attempting 7B–8B models.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/running-local-ai-models", "markdown": "https://wpnews.pro/news/running-local-ai-models.md", "text": "https://wpnews.pro/news/running-local-ai-models.txt", "jsonld": "https://wpnews.pro/news/running-local-ai-models.jsonld"}}