cd /news/ai-infrastructure/what-s-your-setup-for-running-models… · home › topics › ai-infrastructure › article
[ARTICLE · art-172494] src=discuss.huggingface.co ↗ pub= topic=ai-infrastructure verified=true sentiment=· neutral

What's your setup for running models locally?

A Hugging Face forum discussion on local model inference drew advice that a 7B or 8B model at 4-bit quantization runs comfortably on 16GB of unified memory on a MacBook Pro, while anything larger starts swapping. The respondent attributed laptop limits mainly to memory bandwidth and unified memory capacity rather than the runner, and recommended keeping the laptop for small models and prompt testing while renting a GPU box by the hour for larger models or batch jobs, calling a used desktop with a 24GB GPU the sweet spot for daily use.

recommendation

Quantize to 4-bit and cap context length to fit 7B-8B models in 16GB of unified memory; rent a GPU by the hour for larger models or batch jobs.

more in playbook: Inference and self-hosting
read1 min views1 publishedNov 19, 2026
What's your setup for running models locally?
Image: Discuss (auto-discovered)

ashalwi 1

Do you run your models locally on your laptop, or do you use a dedicated machine/server for inference?

I’ve experimented with running several small models on my MacBook Pro using Ollama, but found the performance rather limiting.

Curious to know what setups others are using, especially for experimenting with open-weight models.

On a MacBook Pro the limit is mostly memory bandwidth and how much unified memory you have, so quantization matters more than the runner. A 7B or 8B model at 4 bit runs comfortably on 16GB, anything bigger starts swapping and gets painful. What worked for me was keeping the laptop for small models and testing prompts there, then renting a GPU box by the hour when I needed a bigger model or batch jobs. A used desktop with a 24GB GPU is the sweet spot if you do this daily. Also check your context length setting, a huge default context eats memory fast and looks like slow inference.

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @hugging face 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/what-s-your-setup-fo…] indexed:0 read:1min 2026-11-19 · —