{"slug": "voiceloop-the-fastest-voice-agent-loop-in-the-browser-is-now-open-source", "title": "voiceloop: the fastest voice agent loop in the browser is now open source", "summary": "TODOforAI has open-sourced voiceloop, a zero-dependency JavaScript library that runs a full browser voice-agent loop (VAD → STT → LLM → TTS) with pluggable LLM, STT, and TTS providers. The team also published voice-agent-bench, a black-box benchmark rig that scores agents from recorded audio alone, reporting voiceloop with Deepgram Flux and ElevenLabs flash at 862ms median voice-to-voice latency and a 1067ms p95, versus 866ms/1644ms for OpenAI Realtime and 1046ms/3573ms for Pipecat 1.8.1 on the same providers. The benchmark also found echo handling to be the main differentiator, with voiceloop cutting itself off in 0 of 30 runs while OpenAI Realtime did so in 17 of 30.", "body_md": "*Everybody can now build the best voice agent into their own product. The #1 loop is open source, and the benchmark that says so is public.*\n\n`npm i @todoforai/voiceloop`\nWe wanted the fluid JARVIS feel in the browser for TODOforAI — you talk, it answers within a\n\nsecond, you interrupt it mid-sentence and it just stops. We could not find a stack that did\n\nthis properly. Closed APIs were close but not ours; the open frameworks talked over the user,\n\nor worse, heard their own voice through the speakers and cut themselves off.\n\nIt is 2026. This should be a solved problem. So we solved it and published the whole thing:\n\nthe loop, the numbers, and the rig that produced the numbers.\n\nA zero-dependency JavaScript library that runs the full loop in the browser:\n\n**VAD → STT → LLM → TTS**, with the hard parts already handled.\n\nEverything is pluggable: any OpenAI-compatible LLM, four STT providers (Web Speech,\n\nElevenLabs Scribe, Deepgram Flux, Speechmatics), swappable TTS (Piper local, ElevenLabs\n\ncloud, or your own).\n\n``` js\nimport { VoiceAgent, unlockAudio } from '@todoforai/voiceloop';\n\nconst agent = new VoiceAgent({\n  llmUrl: '/api/chat/completions',   // any OpenAI-compatible endpoint, behind your proxy\n  model: 'claude-haiku-4-5',\n  persona: 'You are a friendly cooking assistant.',\n  onEvent: (e) => { if (e.type === 'assistant') render(e.text); },\n});\n\nbutton.onclick = async () => { unlockAudio(); await agent.start(); };\n```\n\nThat is the entire integration.\n\nLatency claims in voice AI are usually self-reported and unreproducible. We did not want to\n\nadd another one, so we built [voice-agent-bench](https://github.com/todoforai/voice-agent-bench):\n\na **black-box** rig. A scripted \"person\" (byte-identical pre-generated speech) talks into a\n\nvirtual mic, the agent's speaker output is recorded, and every score is derived from the\n\naudio alone. No integration needed — any agent that makes sound can be measured, including\n\nclosed ones.\n\nEvery system gets the same scripted conversations and, where the system allows it, the same\n\nfixed mock LLM (300ms TTFT), so the comparison isolates the voice loop from the model. 5\n\nconversations × 6 turns pooled, n=30, median and p95 — single runs jitter by ±300ms and are\n\nnot worth printing.\n\n| configuration | voice→voice | p95 | barge-in stop | stalls | \n|---|---|---|---|---|\n| OpenAI Realtime (speech-to-speech, own LLM) * | 866ms | 1644 | 429ms | 20 | \n| **voiceloop** · deepgram + ElevenLabs flash | **862ms** | **1067** | 944ms | 16 | \n| **voiceloop** · deepgram + Piper (free, local TTS) | 974ms | 1287 | 1463ms | 19 | \n| Pipecat 1.8.1 · deepgram + EL flash | 1046ms | 3573 | 542ms | 14 | \n| ElevenLabs ConvAI | 1454ms | 1632 | 1042ms | 8 | \n| voiceloop · EL Scribe + EL flash | 1562ms | 1855 | 1566ms | 12 | \n| voiceloop · Speechmatics + EL flash | 1706ms | 2069 | 1046ms | 17 | \n| voiceloop · webspeech + Piper (zero-key) | 2113ms | 2607 | 1257ms | 30 | \n\n| system | clean | hesitation | talked through user | echo | cut itself | \n|---|---|---|---|---|---|\n| OpenAI Realtime * | 870 | 1290 | 0 (yields 130ms) | 790 | **17/30** | \n| **voiceloop** · deepgram + EL flash | **860** | 1400 | 0 (420ms) | 930 | **0** | \n| **voiceloop** · deepgram + Piper | 970 | 1400 | 0 | — | — | \n| Pipecat | 1050 | 1290 | 2 (200ms) | 1320 | **20/30** | \n| ElevenLabs ConvAI | 1450 | 1810 | 0 (490ms) | 1410 | 0 | \n\n* Realtime is speech-to-speech and can't use the fixed mock LLM, so its row isn't fully\n\napples-to-apples.\n\n**Clean audio:** voiceloop with Deepgram Flux + ElevenLabs flash is the fastest configuration\n\nwe measured, at 862ms median — and its p95 (1067ms) is the tightest in the table by a wide\n\nmargin. Pipecat's p95 of 3573ms on the same providers means one turn in twenty takes over\n\nthree seconds. The free, fully local Piper path lands at 974ms with no cloud TTS at all.\n\n**Echo is the failure that separates the stacks.** Feed each system its own voice back\n\nthrough the mic (−15dB, 30ms delay, no AEC — what a laptop with the speakers on actually\n\ndoes) and the other fast stacks hear themselves as the user and cut their own replies:\n\nPipecat on 20 of 30 turns, OpenAI Realtime on 17. voiceloop cut itself **zero** times and\n\nran echo-coupled turns at 930ms — parity with clean. Word-level echo filtering costs no\n\nlatency once it classifies correctly.\n\n**Hesitation:** a user who pauses mid-sentence should not be talked over. Every stack except\n\nPipecat backs off; voiceloop enters 2 of 30 hesitation turns and yields within 420ms.\n\n**The zero-key default is honest about its cost.** Browser Web Speech + Piper needs no\n\naccount anywhere and runs the demo, but it is ~1.2s slower to close a turn than cloud STT\n\n(2113ms). Pick a pipeline STT provider for the numbers above.\n\nFull per-scenario tables, methodology and reproduction steps:\n\n[results/RESULTS.md](https://github.com/todoforai/voice-agent-bench/blob/master/results/RESULTS.md).\n\nThe rows above are the survivors. Behind them are hundreds of runs across STT providers, TTS\n\nengines, VAD thresholds, end-of-turn debounces, barge-in minimum lengths, prefetch stability\n\nwindows and echo-match thresholds. Every knob that mattered is exposed in\n\n[`src/tuning.js`](https://github.com/todoforai/voiceloop/blob/master/src/tuning.js) with the\n\ndefault set to what won on the bench. The edge cases you would otherwise discover one\n\nproduction bug at a time — the agent interrupting itself, tool calls firing on a sentence\n\nthe user was still amending, a hung tool stalling the next turn — are already handled and\n\nregression-tested.\n\nThis is the voice loop inside TODOforAI's JARVIS; the integration overhead between the\n\nlibrary and the product is nil, which is exactly the point. Everybody should have the best\n\nvoice loop. Star it, share it, contribute — let's keep the best one open source.", "url": "https://wpnews.pro/news/voiceloop-the-fastest-voice-agent-loop-in-the-browser-is-now-open-source", "canonical_source": "https://dev.to/todoforai/voiceloop-the-fastest-voice-agent-loop-in-the-browser-is-now-open-source-l36", "published_at": "2026-09-22 09:46:15+00:00", "updated_at": "2026-09-22 09:52:51.485126+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "natural-language-processing", "ai-products"], "entities": ["TODOforAI", "voiceloop", "voice-agent-bench", "OpenAI Realtime", "Pipecat", "ElevenLabs", "Deepgram", "Speechmatics"], "alternates": {"html": "https://wpnews.pro/news/voiceloop-the-fastest-voice-agent-loop-in-the-browser-is-now-open-source", "markdown": "https://wpnews.pro/news/voiceloop-the-fastest-voice-agent-loop-in-the-browser-is-now-open-source.md", "text": "https://wpnews.pro/news/voiceloop-the-fastest-voice-agent-loop-in-the-browser-is-now-open-source.txt", "jsonld": "https://wpnews.pro/news/voiceloop-the-fastest-voice-agent-loop-in-the-browser-is-now-open-source.jsonld"}}