{"slug": "local-ai-on-a-pc-what-actually-works-whisper-translation-llms", "title": "Local AI on a PC: what actually works (Whisper, translation, LLMs)", "summary": "A developer published a hands-on account of running local AI models on consumer hardware, reporting that faster-whisper medium with int8 quantization uses about 4.4 GB of VRAM on an RTX 2070 and transcribes a 30-second clip in 0.36–1.62 seconds, while a 7–8B LLM in Q4 quantization takes 8–12 GB of VRAM and yields 20–40 tokens per second. The writeup recommends Helsinki-NLP OPUS models for subtitle translation, warns that NLLB's CC BY-NC license rules out commercial use, and argues local versus API is an economics and privacy decision, with hybrid setups best for sensitive data.", "body_md": "Local AI is the most overpromised topic in dev tooling. \"Run GPT on your laptop\" — sure, if your laptop has 12 GB of VRAM and you enjoy 3 tokens per second.\n\nI've been running local models in production-ish settings for a while (speech recognition, translation, TTS, document analysis). Here is what actually works on consumer hardware, and what doesn't.\n\n## \n  \n  \n  Speech recognition: faster-whisper\n\nThe best value-per-watt local model I use. CTranslate2 backend, int8 quantization.\n\n- \n**medium int8** on an RTX 2070: ~**4.4 GB of VRAM** , a 30-second fragment in**0.36–1.62 s** depending on speech density;\n- CPU works too, several times slower — fine for batch transcription, too slow for real-time subtitles;\n- hallucination filter is mandatory: silence and noise produce plausible-looking text that nobody said. A VAD gate before recognition fixes most of it.\n\n## \n  \n  \n  Translation\n\n- \n**Helsinki-NLP OPUS models** : ~74 MB, fast, quality is decent for short, well-formed sentences — exactly what subtitles are;\n- \n**NLLB / M2M100** : better quality, but check the licence. NLLB is**CC BY-NC** — fine for personal use, not for a commercial product. I learned that the hard way and swapped it;\n- for anything legal or medical, local NMT is not good enough and you should say so rather than ship it.\n\n## \n  \n  \n  TTS\n\n- \n**Piper** : tiny, fast, runs on CPU, sounds robotic but intelligible — good for notifications and accessibility;\n- \n**XTTS** : much better quality, needs a GPU and a licence read.\n\n## \n  \n  \n  LLMs\n\nA 7–8B model in Q4 quantization takes 8–12 GB of VRAM and produces **20–40 tokens/sec** on a decent GPU. That is enough for:\n\n- summarising long documents;\n- extracting fields from PDFs and invoices;\n- classification and routing;\n- \"explain this error\" against your own logs.\n\nIt is not enough to replace a frontier model for open-ended reasoning — and if you're paying per token, the API is cheaper until you cross a volume threshold.\n\n## \n  \n  \n  The decision that actually matters\n\nLocal vs API is an economics and privacy decision, not a religious one:\n\n- \n**API is cheaper at the start** — dozens of operations a month don't justify hardware;\n- \n**local is cheaper at volume** — thousands of documents a month amortise the GPU in months;\n- \n**local is mandatory for sensitive data** — medical, personal, trade secrets;\n- \n**hybrid is often best** — sensitive data locally, human-facing drafts through an API.\n\nStart with a prototype on your own data. It tells you both quality and real speed, and the decision becomes arithmetic instead of vibes.\n\n*Longer write-up with the hardware table, model sizes and real benchmarks: [Local AI on a PC: what actually works](https://tenlux.tech/en/blog/local-ai-on-a-pc.html). A real product built this way: [Stream-Subs](https://tenlux.tech/stream-subs/en/).*", "url": "https://wpnews.pro/news/local-ai-on-a-pc-what-actually-works-whisper-translation-llms", "canonical_source": "https://dev.to/cjdarklord/local-ai-on-a-pc-what-actually-works-whisper-translation-llms-1323", "published_at": "2026-11-19 01:10:06+00:00", "updated_at": "2026-11-19 01:11:46.387272+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools", "natural-language-processing"], "entities": ["faster-whisper", "CTranslate2", "Helsinki-NLP", "NLLB", "M2M100", "Piper", "XTTS", "Stream-Subs"], "also_reported_by": [], "takeaway": "On an RTX 2070, faster-whisper medium int8 uses ~4.4 GB VRAM and transcribes 30 seconds in 0.36–1.62 s, while a 7–8B Q4 LLM needs 8–12 GB VRAM for 20–40 tokens/sec.", "recommendation": {"advice": "Gate speech recognition with a VAD filter to suppress Whisper hallucinations on silence, and check model licenses like NLLB's CC BY-NC before shipping commercially.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/local-ai-on-a-pc-what-actually-works-whisper-translation-llms", "markdown": "https://wpnews.pro/news/local-ai-on-a-pc-what-actually-works-whisper-translation-llms.md", "text": "https://wpnews.pro/news/local-ai-on-a-pc-what-actually-works-whisper-translation-llms.txt", "jsonld": "https://wpnews.pro/news/local-ai-on-a-pc-what-actually-works-whisper-translation-llms.jsonld"}}