# Shaving a second off a real-time speech-to-LLM pipeline in Electron > Source: > Published: 2026-08-14 23:18:07+00:00 Every part of a speech→LLM pipeline is fast enough on its own. Put them in a row and you get three seconds, which is far too slow when a human is waiting for you to say something. I build a desktop overlay that listens to the other side of a video call, transcribes it, and streams an answer. The budget I care about is time from the speaker finishing a sentence to the first token on screen. Here is where that second and a half went, and what actually moved it. The naive pipeline mic/loopback → PCM → WebSocket STT → final transcript → "is this a question?" classifier → LLM → stream Roughly 3.2s to first token. Four places to attack, and only two of them turned out to matter. So VAD runs client-side in an AudioWorklet, on the raw signal before normalization: this._vadThresh = 0.005; // RMS: above → speech this._hangoverSec = 1.0; // keep sending after level drops this._prerollMax = 14; // ~600ms @48k of buffered silence Three parameters, and each one exists because of a specific failure: Threshold on raw audio. I normalized first, and normalization amplifies room noise into "speech". Detect on the raw signal, normalize afterwards. Hangover of 1.0s. Cutting the stream the instant RMS drops chops the tail off every sentence. It also has to exceed the server's own silence threshold (0.6s) with margin, or the server never gets the trailing silence it needs to commit the segment. Pre-roll of ~600ms. A quiet sentence onset sits below the VAD threshold. By the time you detect speech, the first syllable is gone. So keep a rolling buffer of the last 14 chunks and flush it when VAD opens. That last one is the difference between "what's a database index" and "at's a database index" — and the LLM answers the second one confidently and wrongly. commit_strategy: 'vad', vad_silence_threshold_secs: '0.6', Combined with client-side gating, the transcript arrives while the person is still drawing breath. One sharp edge: keyterm biasing caps at 50. Send 51 and the socket closes with code 1008 and a message you will not see unless you log close reasons. I lost an evening to that. The failure mode: a reconnect fires, and while it is in flight the user switches language, which triggers another reconnect. Now two sockets exist. The older one still delivers events, the newer one is the real connection, and the transcript interleaves garbage from both. Or worse, the stale one wins and the live one is silently ignored — audio flows, nothing appears, no error anywhere. The fix is a generation counter: const myGen = ++this.gen; // this connection // ...later, in every handler: if (myGen !== this.gen) return; // a newer connection superseded us Every async continuation checks whether it is still the current generation. Anything from a superseded socket is dropped on the floor. It is four lines and it removed a whole class of "it just stops working after twenty minutes" reports. It was the single worst component in the system. Not because of latency — because of what it got wrong. Real conversation is full of follow-ups that are not questions in isolation: "What is SOLID?" — answer — "And the second letter?" "And the second letter?" scores as not-a-question. The classifier stayed silent exactly when context made the intent obvious. Precision was fine; the failures were catastrophic and clustered in the most valuable moments. I deleted it. The main model now sees every completed utterance plus recent context and decides for itself — it has the context the classifier never had. That removed 200ms and fixed the follow-up problem. Two wins from deleting code. What replaced it is much dumber and works better: a client-side completeness gate. const AUTO_SILENCE_MS = 900; // no "?" — pause mid-sentence const AUTO_UTTEREND_MS = 120; // ends with "?" — react almost immediately If the buffer ends in a question mark, the sentence is over — fire in 120ms. Otherwise wait 900ms in case they are just thinking. No model call, no network hop. Two blocks, both at a 1-hour TTL: const CACHE_1H = { type: "ephemeral", ttl: "1h" } as const; // block 1: static persona — shared across every session and user // block 2: session data (CV card, role) — stable for one interview Splitting static from per-session matters: block 1 is identical for everyone, so it is warm before the user's first question. Get greedy and interpolate anything variable into it — the answer language, say — and you shatter one cache entry into sixteen. const [, accessRes] = await Promise.all([ enforceRateLimit(db, userId, "ask", 15, 60), assertAccess(db, userId, "ask"), assertDailyCap(db, userId), req.json().then((b) => { body = b; }), ]); ~92ms, for free, with identical guarantees. Unglamorous and the best ratio of effort to result in the whole list. cost / 500 requests full answer Sonnet-class $3.54 8.6s Haiku-class $1.03 5.2s 3.4× cheaper and 3.4 seconds faster. The price is roughly one factual slip every 5–6 questions and slightly rougher prose. For a real-time assistant where a late answer is worth zero, that trade is not close. The enforcement point matters too: the client asks for a model, but the server owns the allow-list. Updates ship manually, so a client-side default would mean every already-installed copy keeps requesting the expensive model forever. Where it landed ~1.1s from end-of-sentence to first token, from ~3.2s. The breakdown of what got it there is not what I expected going in: deleting the classifier: −200ms and a correctness fix VAD gating + server-side endpointing: −1.2s parallel gatekeeping: −92ms cheaper, faster model: −3.4s on full answer The two biggest wins came from removing a component and from parameters in a 40-line audio worklet. Zero came from the LLM call itself, which is where I spent the first week. What I would do differently Log socket close codes from day one. The 1008 keyterm limit was invisible for a day because the close reason was never surfaced. Treat "it silently stopped" as the default failure. Every quiet path — a superseded socket, an expired cache, a swallowed catch — looks identical to "working" from the outside. The generation counter and an idle watchdog on the stream exist because both failed silently first. Measure before assuming the model is the bottleneck. It was 200ms of the 3.2s. The overlay is [Interview Copilot](https://theinterviewcopilot.com/) — Electron on the client, Supabase Edge Functions on the server so the STT and model keys never reach the desktop app. Happy to go deeper on any part of the pipeline in the comments.