# voiceloop: the fastest voice agent loop in the browser is now open source

> Source: <https://dev.to/todoforai/voiceloop-the-fastest-voice-agent-loop-in-the-browser-is-now-open-source-l36>
> Published: 2026-09-22 09:46:15+00:00

*Everybody can now build the best voice agent into their own product. The #1 loop is open source, and the benchmark that says so is public.*

`npm i @todoforai/voiceloop`
We wanted the fluid JARVIS feel in the browser for TODOforAI — you talk, it answers within a

second, you interrupt it mid-sentence and it just stops. We could not find a stack that did

this properly. Closed APIs were close but not ours; the open frameworks talked over the user,

or worse, heard their own voice through the speakers and cut themselves off.

It is 2026. This should be a solved problem. So we solved it and published the whole thing:

the loop, the numbers, and the rig that produced the numbers.

A zero-dependency JavaScript library that runs the full loop in the browser:

**VAD → STT → LLM → TTS**, with the hard parts already handled.

Everything is pluggable: any OpenAI-compatible LLM, four STT providers (Web Speech,

ElevenLabs Scribe, Deepgram Flux, Speechmatics), swappable TTS (Piper local, ElevenLabs

cloud, or your own).

``` js
import { VoiceAgent, unlockAudio } from '@todoforai/voiceloop';

const agent = new VoiceAgent({
  llmUrl: '/api/chat/completions',   // any OpenAI-compatible endpoint, behind your proxy
  model: 'claude-haiku-4-5',
  persona: 'You are a friendly cooking assistant.',
  onEvent: (e) => { if (e.type === 'assistant') render(e.text); },
});

button.onclick = async () => { unlockAudio(); await agent.start(); };
```

That is the entire integration.

Latency claims in voice AI are usually self-reported and unreproducible. We did not want to

add another one, so we built [voice-agent-bench](https://github.com/todoforai/voice-agent-bench):

a **black-box** rig. A scripted "person" (byte-identical pre-generated speech) talks into a

virtual mic, the agent's speaker output is recorded, and every score is derived from the

audio alone. No integration needed — any agent that makes sound can be measured, including

closed ones.

Every system gets the same scripted conversations and, where the system allows it, the same

fixed mock LLM (300ms TTFT), so the comparison isolates the voice loop from the model. 5

conversations × 6 turns pooled, n=30, median and p95 — single runs jitter by ±300ms and are

not worth printing.

| configuration | voice→voice | p95 | barge-in stop | stalls | 
|---|---|---|---|---|
| OpenAI Realtime (speech-to-speech, own LLM) * | 866ms | 1644 | 429ms | 20 | 
| **voiceloop** · deepgram + ElevenLabs flash | **862ms** | **1067** | 944ms | 16 | 
| **voiceloop** · deepgram + Piper (free, local TTS) | 974ms | 1287 | 1463ms | 19 | 
| Pipecat 1.8.1 · deepgram + EL flash | 1046ms | 3573 | 542ms | 14 | 
| ElevenLabs ConvAI | 1454ms | 1632 | 1042ms | 8 | 
| voiceloop · EL Scribe + EL flash | 1562ms | 1855 | 1566ms | 12 | 
| voiceloop · Speechmatics + EL flash | 1706ms | 2069 | 1046ms | 17 | 
| voiceloop · webspeech + Piper (zero-key) | 2113ms | 2607 | 1257ms | 30 | 

| system | clean | hesitation | talked through user | echo | cut itself | 
|---|---|---|---|---|---|
| OpenAI Realtime * | 870 | 1290 | 0 (yields 130ms) | 790 | **17/30** | 
| **voiceloop** · deepgram + EL flash | **860** | 1400 | 0 (420ms) | 930 | **0** | 
| **voiceloop** · deepgram + Piper | 970 | 1400 | 0 | — | — | 
| Pipecat | 1050 | 1290 | 2 (200ms) | 1320 | **20/30** | 
| ElevenLabs ConvAI | 1450 | 1810 | 0 (490ms) | 1410 | 0 | 

* Realtime is speech-to-speech and can't use the fixed mock LLM, so its row isn't fully

apples-to-apples.

**Clean audio:** voiceloop with Deepgram Flux + ElevenLabs flash is the fastest configuration

we measured, at 862ms median — and its p95 (1067ms) is the tightest in the table by a wide

margin. Pipecat's p95 of 3573ms on the same providers means one turn in twenty takes over

three seconds. The free, fully local Piper path lands at 974ms with no cloud TTS at all.

**Echo is the failure that separates the stacks.** Feed each system its own voice back

through the mic (−15dB, 30ms delay, no AEC — what a laptop with the speakers on actually

does) and the other fast stacks hear themselves as the user and cut their own replies:

Pipecat on 20 of 30 turns, OpenAI Realtime on 17. voiceloop cut itself **zero** times and

ran echo-coupled turns at 930ms — parity with clean. Word-level echo filtering costs no

latency once it classifies correctly.

**Hesitation:** a user who pauses mid-sentence should not be talked over. Every stack except

Pipecat backs off; voiceloop enters 2 of 30 hesitation turns and yields within 420ms.

**The zero-key default is honest about its cost.** Browser Web Speech + Piper needs no

account anywhere and runs the demo, but it is ~1.2s slower to close a turn than cloud STT

(2113ms). Pick a pipeline STT provider for the numbers above.

Full per-scenario tables, methodology and reproduction steps:

[results/RESULTS.md](https://github.com/todoforai/voice-agent-bench/blob/master/results/RESULTS.md).

The rows above are the survivors. Behind them are hundreds of runs across STT providers, TTS

engines, VAD thresholds, end-of-turn debounces, barge-in minimum lengths, prefetch stability

windows and echo-match thresholds. Every knob that mattered is exposed in

[`src/tuning.js`](https://github.com/todoforai/voiceloop/blob/master/src/tuning.js) with the

default set to what won on the bench. The edge cases you would otherwise discover one

production bug at a time — the agent interrupting itself, tool calls firing on a sentence

the user was still amending, a hung tool stalling the next turn — are already handled and

regression-tested.

This is the voice loop inside TODOforAI's JARVIS; the integration overhead between the

library and the product is nil, which is exactly the point. Everybody should have the best

voice loop. Star it, share it, contribute — let's keep the best one open source.