{"slug": "what-s-your-setup-for-running-models-locally", "title": "What's your setup for running models locally?", "summary": "A Hugging Face forum discussion on local model inference drew advice that a 7B or 8B model at 4-bit quantization runs comfortably on 16GB of unified memory on a MacBook Pro, while anything larger starts swapping. The respondent attributed laptop limits mainly to memory bandwidth and unified memory capacity rather than the runner, and recommended keeping the laptop for small models and prompt testing while renting a GPU box by the hour for larger models or batch jobs, calling a used desktop with a 24GB GPU the sweet spot for daily use.", "body_md": "[ashalwi](https://discuss.huggingface.co/u/ashalwi)\n1\n \nDo you run your models locally on your laptop, or do you use a dedicated machine/server for inference?\n\nI’ve experimented with running several small models on my MacBook Pro using Ollama, but found the performance rather limiting.\n\nCurious to know what setups others are using, especially for experimenting with open-weight models.\n\n \n \nOn a MacBook Pro the limit is mostly memory bandwidth and how much unified memory you have, so quantization matters more than the runner. A 7B or 8B model at 4 bit runs comfortably on 16GB, anything bigger starts swapping and gets painful. What worked for me was keeping the laptop for small models and testing prompts there, then renting a GPU box by the hour when I needed a bigger model or batch jobs. A used desktop with a 24GB GPU is the sweet spot if you do this daily. Also check your context length setting, a huge default context eats memory fast and looks like slow inference.", "url": "https://wpnews.pro/news/what-s-your-setup-for-running-models-locally", "canonical_source": "https://discuss.huggingface.co/t/whats-your-setup-for-running-models-locally/192128#post_2", "published_at": "2026-11-19 11:21:23+00:00", "updated_at": "2026-11-19 11:41:40.733572+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools"], "entities": ["Hugging Face", "Ollama", "MacBook Pro"], "also_reported_by": [], "takeaway": "On a 16GB MacBook Pro, run 7B-8B models at 4-bit quantization and keep the default context length small, since larger models swap and oversized context eats memory and looks like slow inference.", "recommendation": {"advice": "Quantize to 4-bit and cap context length to fit 7B-8B models in 16GB of unified memory; rent a GPU by the hour for larger models or batch jobs.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/what-s-your-setup-for-running-models-locally", "markdown": "https://wpnews.pro/news/what-s-your-setup-for-running-models-locally.md", "text": "https://wpnews.pro/news/what-s-your-setup-for-running-models-locally.txt", "jsonld": "https://wpnews.pro/news/what-s-your-setup-for-running-models-locally.jsonld"}}