cd /news/artificial-intelligence/local-ai-on-a-pc-what-actually-works… · home › topics › artificial-intelligence › article
[ARTICLE · art-172234] src=dev.to ↗ pub= topic=artificial-intelligence verified=true sentiment=· neutral

Local AI on a PC: what actually works (Whisper, translation, LLMs)

A developer published a hands-on account of running local AI models on consumer hardware, reporting that faster-whisper medium with int8 quantization uses about 4.4 GB of VRAM on an RTX 2070 and transcribes a 30-second clip in 0.36–1.62 seconds, while a 7–8B LLM in Q4 quantization takes 8–12 GB of VRAM and yields 20–40 tokens per second. The writeup recommends Helsinki-NLP OPUS models for subtitle translation, warns that NLLB's CC BY-NC license rules out commercial use, and argues local versus API is an economics and privacy decision, with hybrid setups best for sensitive data.

recommendation

Gate speech recognition with a VAD filter to suppress Whisper hallucinations on silence, and check model licenses like NLLB's CC BY-NC before shipping commercially.

more in playbook: Inference and self-hosting
by read2 min views2 publishedNov 19, 2026

Local AI is the most overpromised topic in dev tooling. "Run GPT on your laptop" — sure, if your laptop has 12 GB of VRAM and you enjoy 3 tokens per second.

I've been running local models in production-ish settings for a while (speech recognition, translation, TTS, document analysis). Here is what actually works on consumer hardware, and what doesn't.

#

Speech recognition: faster-whisper The best value-per-watt local model I use. CTranslate2 backend, int8 quantization.

medium int8 on an RTX 2070: ~4.4 GB of VRAM , a 30-second fragment in0.36–1.62 s depending on speech density;

  • CPU works too, several times slower — fine for batch transcription, too slow for real-time subtitles;
  • hallucination filter is mandatory: silence and noise produce plausible-looking text that nobody said. A VAD gate before recognition fixes most of it.

#

Translation

Helsinki-NLP OPUS models : ~74 MB, fast, quality is decent for short, well-formed sentences — exactly what subtitles are; #

NLLB / M2M100 : better quality, but check the licence. NLLB isCC BY-NC — fine for personal use, not for a commercial product. I learned that the hard way and swapped it;

  • for anything legal or medical, local NMT is not good enough and you should say so rather than ship it.

#

TTS

Piper : tiny, fast, runs on CPU, sounds robotic but intelligible — good for notifications and accessibility; #

XTTS : much better quality, needs a GPU and a licence read.

#

LLMs

A 7–8B model in Q4 quantization takes 8–12 GB of VRAM and produces 20–40 tokens/sec on a decent GPU. That is enough for:

  • summarising long documents;
  • extracting fields from PDFs and invoices;
  • classification and routing;
  • "explain this error" against your own logs.

It is not enough to replace a frontier model for open-ended reasoning — and if you're paying per token, the API is cheaper until you cross a volume threshold.

#

The decision that actually matters

Local vs API is an economics and privacy decision, not a religious one:

API is cheaper at the start — dozens of operations a month don't justify hardware; #

local is cheaper at volume — thousands of documents a month amortise the GPU in months; #

local is mandatory for sensitive data — medical, personal, trade secrets; #

hybrid is often best — sensitive data locally, human-facing drafts through an API.

Start with a prototype on your own data. It tells you both quality and real speed, and the decision becomes arithmetic instead of vibes.

Longer write-up with the hardware table, model sizes and real benchmarks: Local AI on a PC: what actually works. A real product built this way: Stream-Subs.

── more in #artificial-intelligence 4 stories · sorted by recency
── more on @faster-whisper 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/local-ai-on-a-pc-wha…] indexed:0 read:2min 2026-11-19 · —