cd /news/large-language-models/running-local-ai-models · home › topics › large-language-models › article
[ARTICLE · art-173153] src=discuss.huggingface.co ↗ pub= topic=large-language-models verified=true sentiment=· neutral

Running local AI models

A practitioner running Ollama on an 8GB M1 MacBook Pro reported that quantized 1B–2B parameter models such as Qwen 0.6B and 1.7B remain usable for narrow tasks like classification, information extraction and structured JSON output, while 7B–8B models are a poor fit for the hardware. The practitioner measured roughly 1.5 to 1.9 seconds on some simple tasks, depending on prompt and output length, and recommended monitoring Memory Pressure and Swap Used in Activity Monitor because 8GB fills quickly with browsers and other apps running. The account concludes that local inference on 8GB is possible if expectations are kept realistic, and that hosted inference is less frustrating for more demanding work.

recommendation

On 8GB machines, start with quantized 1B–2B models for narrow tasks and watch Memory Pressure and Swap Used in Activity Monitor before attempting 7B–8B models.

more in playbook: Inference and self-hosting
read1 min views8 publishedNov 20, 2026

I’m running Ollama on an 8GB M1 MacBook Pro, so pretty similar constraints to yours!

I’ve experimented with smaller Qwen models (0.6B and 1.7B), mainly for classification, information extraction and structured JSON outputs.

A few things I’ve learned:

Smaller models are surprisingly usable if you give them specific, narrow tasks. I wouldn’t expect them to perform like Claude or ChatGPT, though.

I use Activity Monitor to keep an eye on Memory Pressure and Swap Used. The 8GB fills up pretty quickly, especially with browsers and other apps running.

Quantized models help considerably. I’d start with something around 1B to 2B parameters rather than jumping straight into 7B or 8B models.

For more demanding tasks, hosted inference is definitely less frustrating. My smaller models have managed around 1.5 to 1.9 seconds on some simple tasks, although that obviously depends on the prompt and output length.

I’m still learning and experimenting myself, but my biggest takeaway is that local inference on 8GB is absolutely possible. You just have to be realistic about what you’re asking the model to do.

Curious what models you’re planning to try!

── more in #large-language-models 4 stories · sorted by recency
── more on @ollama 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/running-local-ai-mod…] indexed:0 read:1min 2026-11-20 · —