Local AI on a PC: what actually works (Whisper, translation, LLMs) A developer published a hands-on account of running local AI models on consumer hardware, reporting that faster-whisper medium with int8 quantization uses about 4.4 GB of VRAM on an RTX 2070 and transcribes a 30-second clip in 0.36–1.62 seconds, while a 7–8B LLM in Q4 quantization takes 8–12 GB of VRAM and yields 20–40 tokens per second. The writeup recommends Helsinki-NLP OPUS models for subtitle translation, warns that NLLB's CC BY-NC license rules out commercial use, and argues local versus API is an economics and privacy decision, with hybrid setups best for sensitive data. Local AI is the most overpromised topic in dev tooling. "Run GPT on your laptop" — sure, if your laptop has 12 GB of VRAM and you enjoy 3 tokens per second. I've been running local models in production-ish settings for a while speech recognition, translation, TTS, document analysis . Here is what actually works on consumer hardware, and what doesn't. Speech recognition: faster-whisper The best value-per-watt local model I use. CTranslate2 backend, int8 quantization. - medium int8 on an RTX 2070: ~ 4.4 GB of VRAM , a 30-second fragment in 0.36–1.62 s depending on speech density; - CPU works too, several times slower — fine for batch transcription, too slow for real-time subtitles; - hallucination filter is mandatory: silence and noise produce plausible-looking text that nobody said. A VAD gate before recognition fixes most of it. Translation - Helsinki-NLP OPUS models : ~74 MB, fast, quality is decent for short, well-formed sentences — exactly what subtitles are; - NLLB / M2M100 : better quality, but check the licence. NLLB is CC BY-NC — fine for personal use, not for a commercial product. I learned that the hard way and swapped it; - for anything legal or medical, local NMT is not good enough and you should say so rather than ship it. TTS - Piper : tiny, fast, runs on CPU, sounds robotic but intelligible — good for notifications and accessibility; - XTTS : much better quality, needs a GPU and a licence read. LLMs A 7–8B model in Q4 quantization takes 8–12 GB of VRAM and produces 20–40 tokens/sec on a decent GPU. That is enough for: - summarising long documents; - extracting fields from PDFs and invoices; - classification and routing; - "explain this error" against your own logs. It is not enough to replace a frontier model for open-ended reasoning — and if you're paying per token, the API is cheaper until you cross a volume threshold. The decision that actually matters Local vs API is an economics and privacy decision, not a religious one: - API is cheaper at the start — dozens of operations a month don't justify hardware; - local is cheaper at volume — thousands of documents a month amortise the GPU in months; - local is mandatory for sensitive data — medical, personal, trade secrets; - hybrid is often best — sensitive data locally, human-facing drafts through an API. Start with a prototype on your own data. It tells you both quality and real speed, and the decision becomes arithmetic instead of vibes. Longer write-up with the hardware table, model sizes and real benchmarks: Local AI on a PC: what actually works https://tenlux.tech/en/blog/local-ai-on-a-pc.html . A real product built this way: Stream-Subs https://tenlux.tech/stream-subs/en/ .