Local AI is the most overpromised topic in dev tooling. "Run GPT on your laptop" — sure, if your laptop has 12 GB of VRAM and you enjoy 3 tokens per second.
I've been running local models in production-ish settings for a while (speech recognition, translation, TTS, document analysis). Here is what actually works on consumer hardware, and what doesn't.
#
Speech recognition: faster-whisper The best value-per-watt local model I use. CTranslate2 backend, int8 quantization.
medium int8 on an RTX 2070: ~4.4 GB of VRAM , a 30-second fragment in0.36–1.62 s depending on speech density;
- CPU works too, several times slower — fine for batch transcription, too slow for real-time subtitles;
- hallucination filter is mandatory: silence and noise produce plausible-looking text that nobody said. A VAD gate before recognition fixes most of it.
#
Translation
Helsinki-NLP OPUS models : ~74 MB, fast, quality is decent for short, well-formed sentences — exactly what subtitles are; #
NLLB / M2M100 : better quality, but check the licence. NLLB isCC BY-NC — fine for personal use, not for a commercial product. I learned that the hard way and swapped it;
- for anything legal or medical, local NMT is not good enough and you should say so rather than ship it.
#
TTS
Piper : tiny, fast, runs on CPU, sounds robotic but intelligible — good for notifications and accessibility; #
XTTS : much better quality, needs a GPU and a licence read.
#
LLMs
A 7–8B model in Q4 quantization takes 8–12 GB of VRAM and produces 20–40 tokens/sec on a decent GPU. That is enough for:
- summarising long documents;
- extracting fields from PDFs and invoices;
- classification and routing;
- "explain this error" against your own logs.
It is not enough to replace a frontier model for open-ended reasoning — and if you're paying per token, the API is cheaper until you cross a volume threshold.
#
The decision that actually matters
Local vs API is an economics and privacy decision, not a religious one:
API is cheaper at the start — dozens of operations a month don't justify hardware; #
local is cheaper at volume — thousands of documents a month amortise the GPU in months; #
local is mandatory for sensitive data — medical, personal, trade secrets; #
hybrid is often best — sensitive data locally, human-facing drafts through an API.
Start with a prototype on your own data. It tells you both quality and real speed, and the decision becomes arithmetic instead of vibes.
Longer write-up with the hardware table, model sizes and real benchmarks: Local AI on a PC: what actually works. A real product built this way: Stream-Subs.