{"slug": "why-i-run-speech-to-text-locally-instead-of-calling-a-cloud-api", "title": "Why I run speech-to-text locally instead of calling a cloud API", "summary": "A developer building a phone assistant that transcribes work calls has chosen to run speech-to-text locally with faster-whisper instead of using cloud APIs, citing data privacy concerns. The local setup, which uses a Whisper medium model in int8 quantization on a shared GPU, keeps audio files on the machine and avoids sending sensitive call content to vendor servers. The developer acknowledges trade-offs, including slower transcription under GPU pressure and accuracy limitations, but considers the privacy benefit worth the cost.", "body_md": "[Yesterday I wrote about deploying gemma, bge-m3, and whisper on a single server without enough VRAM for all three.](https://dev.to/hannune/running-three-ai-models-on-one-local-server-when-your-vram-doesnt-cover-all-of-them-b7g) This post is about why whisper is one of those three.\n\nWhen you call a cloud speech-to-text API—OpenAI's Whisper endpoint, Google Speech-to-Text, Amazon Transcribe—the audio travels to their servers. For most use cases that's fine. For mine it isn't.\n\nThe project I'm building transcribes work calls. Work calls contain client names, project specifics, sometimes pricing discussions. Sending that audio to a vendor's inference endpoint means it travels over the network and gets processed on hardware I don't control.\n\nRunning whisper locally means the audio file stays on the machine.\n\n`faster-whisper`\n\nin `int8`\n\nquantized modeA transcription call looks like this:\n\n``` python\nfrom faster_whisper import WhisperModel\n\nmodel = WhisperModel(\"medium\", device=\"cuda\", compute_type=\"int8\")\nsegments, info = model.transcribe(\"recording.m4a\", beam_size=5)\nfor segment in segments:\n    print(f\"[{segment.start:.1f}s] {segment.text}\")\n```\n\nNo API call. No auth header. No request log on a vendor's side. The audio file doesn't leave the machine.\n\nThe 12 GB card is shared with two other models. Whisper `medium`\n\nin int8 uses roughly 2.5–3 GB of VRAM when warm. That's workable as long as the models don't run at the same time.\n\nThe three models load sequentially in my pipeline: transcribe → embed → analyze. Whisper runs first, which means it has the most headroom before the other two have loaded. When gemma is handling a VLM task and the card is under pressure, whisper drops to CPU. Slower—30–60 seconds per minute of audio instead of 8–12—but it completes without crashing.\n\nI didn't anticipate needing that fallback. It showed up during the first few real recordings when I hadn't fully worked out the load order yet.\n\nLocal STT doesn't fix transcription accuracy automatically. Whisper `medium`\n\nis good but not perfect. Names, domain-specific terms, and cross-talk all degrade it. I haven't run it on enough actual work calls to have a reliable accuracy number. On clean audio, 90%+. On a speakerphone with background noise, noticeably worse.\n\nIt also doesn't solve the downstream pipeline. Right now I can transcribe a call and get a text file. The next piece—extracting structured information from that transcript, linking it to what came in over text and email—is what I'm working on now. That part isn't done.\n\nHosted Whisper endpoints exist and some are cheap. If data residency doesn't matter, they're probably the right call: faster, no hardware to manage, no VRAM budgeting.\n\nThe asymmetry is the point. A slower local run that keeps audio in-house is worth the tradeoff when the content of the calls is the thing you're trying to protect. The cost of audio leaving is not symmetric with the cost of running it locally.\n\nI'm building a phone assistant that collects what comes in over calls, texts, and emails—and turns it into something useful instead of leaving it scattered across three different apps. Local transcription is the piece that's working. Everything past the transcript is still being built.\n\nIf you've dealt with local Whisper deployment on shared GPU, curious what you found: do you bother with beam_size tuning, or does the accuracy delta not matter enough at inference time to be worth it?", "url": "https://wpnews.pro/news/why-i-run-speech-to-text-locally-instead-of-calling-a-cloud-api", "canonical_source": "https://dev.to/hannune/why-i-run-speech-to-text-locally-instead-of-calling-a-cloud-api-59j7", "published_at": "2026-08-18 02:34:34+00:00", "updated_at": "2026-08-18 03:12:14.149445+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "developer-tools"], "entities": ["OpenAI", "Google Speech-to-Text", "Amazon Transcribe", "faster-whisper", "Whisper"], "alternates": {"html": "https://wpnews.pro/news/why-i-run-speech-to-text-locally-instead-of-calling-a-cloud-api", "markdown": "https://wpnews.pro/news/why-i-run-speech-to-text-locally-instead-of-calling-a-cloud-api.md", "text": "https://wpnews.pro/news/why-i-run-speech-to-text-locally-instead-of-calling-a-cloud-api.txt", "jsonld": "https://wpnews.pro/news/why-i-run-speech-to-text-locally-instead-of-calling-a-cloud-api.jsonld"}}