You say into your phone: "Explain quantum entanglement in one minute." A beat later, a voice answers β not the robot from your GPS, but almost a real conversation partner: it s, it has intonation, it even drops a little "hm" before the tricky part. This is voice-to-voice (V2V), also called speech-to-speech (S2S): a system that takes audio in and answers with audio. In this article we'll take the trick apart, from the wave hitting your microphone to the sound coming out of your speaker.
I spent a while at Newo.ai, deploying voice agents into real projects on the European market. In production, the marketing line "talks almost like a human" decomposes quickly into measurable problems: latency, false endpoints, mangled names, and a tone of voice that's wrong for the moment.
That's where I first collided with voice-to-voice systems for real β the ones that take speech in and answer with speech. From the outside it all looks simple. Inside there may be a cascade of several models, neural codecs, speech encoders, transformers, and months of training that the sales brochures never get around to mentioning. I wanted to know how the thing actually works and how V2V systems differ from one another β not only by their slide decks, but by their published mechanisms.
So I dug in, and this article is the result. It's for people who already have V2V in their work life and who β like me not so long ago β would rather see an understandable machine under the hood than a magic show.
If you know how large language models work (or read my book How LLMs Work: From Zero to Your Own Transformer), you know the principle of
The article is written for an IT professional: you know what an API is, what a GPU does, and roughly what a transformer looks like, but you may never have touched speech ML. We won't flatten it into "the neural net listens and talks." We'll walk the whole path: sound representations, ways to organize the system, training, streaming, and the places where it all breaks. And for closed models we'll keep observable product behavior carefully separated from guesses about their internals.
The main text explains the ideas in plain words, the code examples are in Swift (github.com/asaptf/v2v-models-article-en β every file is dependency-free and runs with a single command), and the callouts add engineering depth:
| Icon | Callout type |
|---|---|
| π‘ | The key idea of a section |
| π | An analogy |
| π | Engineering details (formulas and such) |
| β | A common mistake |
| π | A bit of history |
| π | Links to models and papers |
Why Swift, of all things, and not Python? Because the examples here illustrate ideas (bitrate arithmetic, RVQ mechanics, streaming logic), not real training code. If you have Swift installed, the scripts run with zero external libraries. Real speech ML mostly lives in Python and PyTorch; links to working repositories and papers are in Chapter 7 and in the reading list at the end.
From the outside, a V2V system could not look simpler:
Speech goes in β speech comes out. There's no intermediate text for you to read (although inside, it sometimes exists anyway β more on that in Chapter 2). This entire article is about what happens inside that box.
The big idea of this article: many modern V2V systems borrow the ideas of language models and apply them to sound β autoregression, transformers, discrete speech representations. But how the audio is represented and how the conversation is run are separate engineering choices. A token model can be turn-based or full-duplex, and a realtime API can quietly hide several models inside.
Analogy: a text LLM is a writer who types the answer letter by letter. A token-based speech model is a radio announcer who assembles the answer from several parallel streams of codes β while also tracking s, interruptions and intonation, live on air. Both learned their craft the same way: from an enormous corpus of examples of "what came next."
Speech synthesis (TTS, text-to-speech) has been around for decades: you provide text, you get a voice. V2V is a different beast:
| TTS | V2V | |
|---|---|---|
| Input | text | audio (the user's speech) |
| Understands meaning | doesn't have to | has to |
| Role in the dialogue | voices a ready-made text | understands the input and forms the reply |
| Latency | a single component's metric | measured end-to-end, for the whole dialogue |
At the product level, V2V combines the abilities to recognize speech, form a reply, and synthesize sound. Those abilities can live in three separate blocks, in a speech-native loop, or in a hybrid that delegates β and that classification is exactly where Chapter 2 picks up.
A terminology check: an assistant built as a "Whisper β GPT β ElevenLabs" pipeline is properly called a cascaded V2V system, not a native S2S model. A cascade is easier to debug; but unless you pass extra acoustic features along, it loses the intonation and non-verbal cues that never made it into the transcript.
Before a model can "hear" speech, sound has to become a representation it can work with. The raw wave, spectral features and codec tokens are measured in different units, so just counting elements tells you little β what matters is frame rate, the number of codebooks, and bitrate.
A microphone measures the amplitude of sound pressure many times per second. Wideband speech typically uses 16 or 24 kHz: one second of mono audio at 24 kHz is 24,000 samples. At 16 bits per sample, that's 384 kbit/s before any container overhead.
Feeding every sample into a vanilla transformer as its own position is expensive: attention cost grows quadratically with sequence length, and 24,000 positions per second is a lot of sequence. But it doesn't follow that neural networks can't handle waveforms at all: wav2vec 2.0, Whisper-style frontends and other encoders happily accept the raw wave β they just shrink its temporal resolution with convolutions or patches before anything reaches the attention blocks.
Sound is sliced into windows of roughly 20β30 ms, and for each window we compute how the energy is distributed across frequencies. With a 10 ms hop you get about 100 frames per second, each one a vector of, say, 80 mel channels.
A spectrogram is convenient for a speech encoder, but "24,000 samples versus 100 frames" is not a compression ratio: each frame holds dozens of floating-point numbers. A spectrogram can easily take more memory than the original PCM. Its job is to make the local structure of speech convenient for the model, not necessarily to make the file smaller.
By the way, if you've ever flipped Audacity into spectrogram view, you've already seen this representation with your own eyes.
A neural audio codec compresses the wave into a sequence of latent frames, and a quantizer replaces each frame with indices from codebooks. Two quantities matter here, and they're easy to conflate:
If frames/s and , the model receives not 50 but up to 200 code indices per second. These are what people informally call audio tokens. With a codebook of size , one index costs 10 bits, and the useful bitrate comes out to roughly:
That's about 192 times less than 16-bit PCM at 24 kHz β and it's a far more honest calculation than dividing the number of samples by the number of frames.
You can run the numbers yourself in ch01_audio_compression.swift:
let sampleRate = 24_000
let pcmBitrate = sampleRate * 16 // 384 kbit/s
let frameRate = 50
let codebooks = 4
let bitsPerCode = 10 // K = 1024
let codecBitrate = frameRate * codebooks * bitsPerCode // 2 kbit/s
Key idea: the codec's frame rate and the number of discrete tokens are not the same thing. One time frame can carry several parallel RVQ codes. It's these parallel streams β not "50 words of sound per second" β that the transformer actually has to model.
Engineering notes: audio parameters.
A low-bitrate codec is negotiating a trade between intelligibility, naturalness and compute. Depending on the data and the loss function, the casualties may include:
The decoder's quality caps the acoustic quality of a token-based system β but not its understanding or its dialogue behavior. And a token codec is only one path: some speech models work with continuous latents or generate spectral features instead.
Common mistake: treating a codec token as a phoneme or a syllable. An individual index describes one quantized component of a short audio frame. Its link to phonetics can be strong for a semantically trained codebook β but a fixed "token βοΈ speech sound" mapping usually doesn't exist.
Cascade, audio tokens and full-duplex are often presented as three competing "architectures." That mixes up different questions. It's cleaner to place a system on three independent axes: what components it's built from, how it represents sound, and how it runs a conversation.
The classic scheme wires up three blocks:
π€ audio β [ASR] β text β [LLM] β text β [TTS] β π audio
What the cascade gets you:
What it costs you:
A cascade is not doomed to sound bad. Streaming ASR, a fast LLM, an expressive TTS and separately passed acoustic features can bring it close to native systems. And its real advantage isn't only the quick prototype: it's control.
In a speech-native (or end-to-end) system, audio is an input and output modality of the model loop itself. There is no mandatory text interface in the middle that a developer gets to see. Inside, there may still be a text stream, several transformers, a codec and assorted helper models.
For open models you can verify the architecture against the paper and the code. For closed APIs, the words "native audio" describe a documented product capability. They do not prove that a single transformer lives inside.
A practical system often combines the two: a fast speech model keeps the conversation flowing and hands the hard questions to a text reasoning model or an agent. GPT-Live works this way: continuous voice interaction is separated from deeper reasoning, which may run on a different model (OpenAI announcement, July 8, 2026).
A codec turns each audio frame into several RVQ indices. The transformer predicts them sequentially or hierarchically, and a decoder reconstructs the wave. Moshi and the speech generator in Sesame CSM are built this way.
audio β [codec] β several code streams β [model] β codes β [decoder] β audio
Discretization is convenient for cross-entropy and autoregression. The price is quantization, plus a lot of parallel code streams to keep track of.
A model can also consume and predict continuous latents or mel features, with no codebook involved. That sidesteps part of the quantization loss but calls for a different loss function and decoder.
Voicebox and other generators shape an audio fragment iteratively or non-autoregressively. These methods are strong in TTS and audio editing; causal realtime streaming is harder for them. A hybrid can plan the semantics autoregressively and let a fast non-autoregressive decoder paint in the acoustics.
Key idea: "native" doesn't mean "necessarily token-based," and "token-based" doesn't mean "necessarily realtime." These are independent properties.
The system waits for your turn to end, then answers. The boundary comes from VAD, an endpointing model, the ASR, or some combination of them. Many cascades and speech generators work like this.
Audio is sent in chunks, the reply is synthesized while it's still being generated, and fresh user speech can stop the playback. Barge-in improves how the interface feels. By itself it doesn't mean the model keeps meaningfully listening while it talks.
The system models the incoming and outgoing channels all the time. It can listen, speak, stay silent, drop a quick "uh-huh," and handle overlap without a hard turn boundary. The open reference is Moshi: two audio streams represented in parallel, with a claimed practical latency of about 200 ms (DΓ©fossez et al., 2024). In 2026, full-duplex also reached mass-market ChatGPT Voice via GPT-Live.
| System | Organization | Representation | Mode | The fine print |
|---|---|---|---|---|
| ASR β LLM β TTS | cascade | text between blocks | turn-based or streaming | acoustic features can be passed alongside |
| AudioLM | audio generator | discrete tokens | audio continuation | not a dialogue assistant |
| VALL-E | TTS generator | codec tokens | turn-based | takes text plus a voice prompt |
| Sesame CSM-1B | speech generator | Mimi/RVQ | turn-based | needs an external LLM for the reply's meaning |
| SpeechGPT | speech-language model | discrete speech + text | turn-based | research model |
| Moshi | speech-native | Mimi, parallel streams | full-duplex | paper, weights and code are open |
| OpenAI Realtime / Gemini Live | speech-native by API contract | internals closed | streaming, barge-in | architecture can't be inferred from the API |
| GPT-Live | hybrid system | internals closed | full-duplex | deep tasks delegated to a frontier model |
AudioLM positions itself as an audio-continuation model (Borsos et al., 2022), VALL-E as text-to-speech (Microsoft Research), and the open CSM flat-out recommends a separate LLM for writing the reply (SesameAILabs/csm). All three matter for the history of codec language models. None of them is a ready-made V2V conversation partner.
| You need | Where to start |
|---|---|
| A rigid script, audits, compliance | a cascade with a transcript and deterministic checks |
| Expressive voicing of ready-made text | a specialized TTS or speech generator |
| A fast multimodal agent | a realtime API, measured on your network and your language |
| Overlap and natural turn-taking | a system whose full-duplex claim you've verified separately |
| Data that can't leave your infrastructure | self-hosted components, after checking quality, license and ops cost |
The trend is not an inevitable march from cascades toward one-model systems. Native systems keep improving naturalness and latency; cascades keep their edge in observability and control. What wins in production, more and more often, is not a pure class but a measured hybrid.
For many token-based speech models, the codec is the bridge between the waveform and the transformer. It reduces temporal resolution and turns each continuous latent frame into a handful of discrete indices. An important way to represent sound β but not the only one.
A typical neural codec has three parts:
The codec is trained as an autoencoder: the reconstruction should resemble the original by spectral and perceptual criteria. Adversarial and feature-matching losses are added so that the output doesn't drift into dull, averaged-out sound.
In residual vector quantization, each codebook refines the error left by the previous one:
A simplified demo lives in ch03_rvq_demo.swift:
let (t1, q1) = nearest(z, in: codebookL1)
let r1 = subtract(z, q1)
let (t2, q2) = nearest(r1, in: codebookL2)
let reconstructed = add(q1, q2) // z β q1 + q2
For one time frame, the formula is:
At frame rate with codebooks, the codec emits indices per second if you count every codebook token separately. A model can predict them as one flat sequence, with parallel heads, or with a hierarchy of temporal and depth transformers.
Analogy: the first layer names the rough color of a wall, the next ones refine the shade and the finish. Keep in mind, though, that all these refinements describe one moment in time β not successive slices of speech.
| Codec | Author | What matters |
|---|---|---|
| SoundStream | ||
| Google, 2021 | causal codec with RVQ and adversarial training | |
| EnCodec | ||
| Meta, 2022 | open implementation for speech and music | |
| DAC | ||
| Descript, 2023 | high quality, open weights | |
| Mimi | ||
| Kyutai, 2024 | 12.5 frames/s, 8 codebooks; the first distilled from WavLM |
Mimi outputs eight synchronized code streams at 12.5 Hz: up to 100 indices per second for a single audio channel. In Moshi, a large temporal transformer walks across time frames while a small depth transformer models the dependencies between codebooks inside a frame. That's how a second of sound avoids becoming a flat sequence of a hundred steps (Moshi, Β§3.3β3.4).
The codec is often trained separately and frozen before the speech LM learns on top of it. Often, not always: joint fine-tuning, continuous speech encoders and RVQ-free decoders all exist in the wild.
The customary split goes like this:
This is not a clean "what was said / how it was said" decomposition. A semantic token is not a ready-made piece of meaning, and an acoustic codebook can carry phonetic information of its own.
AudioLM first models discretized w2v-BERT features, then SoundStream codes. In Mimi, the first RVQ codebook is distilled from WavLM and the remaining ones encode the residual acoustics. So the accurate phrase is a semantically reinforced first codebook, not a stream where meaning is fully separated from voice.
Common mistake: contrasting neural codecs with MP3 as if one existed only for transformers and the other only for human ears. Both solve perceptual compression. The real differences are the learned representation, the latency, the bitrate, and how convenient the resulting codes are as targets for a generative model.
There is no such thing as "the" V2V architecture. This chapter walks through a common autoregressive variant built on a discrete codec, then looks at Moshi as a documented full-duplex example. Closed realtime APIs cannot be reconstructed from this scheme.
A token-based speech-to-speech system is convenient to picture as four blocks:
ββββββββββββ βββββββββββββββ ββββββββββββββββ ββββββββββββ
β Input β β β Transformer β β β heads / β β β audio β
β encoder β β backbone β β depth model β β decoder β
ββββββββββββ βββββββββββββββ ββββββββββββββββ ββββββββββββ
A terminology trap: "decoder" gets used both for the transformer decoder and for the audio decoder. "End-to-end" is slippery too: it may mean nothing more than the absence of a text boundary in the public interface, not that a single network does everything.
Autoregressive speech LMs run on familiar machinery:
For a single discrete stream, the cross-entropy looks like it always does:
With codebooks there are usually several output distributions and a summed loss:
Text and audio don't have to share one vocabulary. In Moshi they're synchronized parallel streams; other models use separate encoders, projectors and heads. So "special tokens simply separate the modalities in one string" is one possible implementation, not a definition.
You can poke at single-step cross-entropy in ch05_cross_entropy.swift.
At the level of an API or a training example, the context can be written like this:
[SYSTEM_TEXT] the instruction
[USER_AUDIO] input frames or embeddings
[TOOL_RESULT] application data, if a tool was called
[ASSISTANT_AUDIO] the reply being generated
This is a logical view, not a required physical layout of tensors. The system prompt may feed a text backbone, the audio may go through its own encoder, and the tool result may arrive via a separate model or a server-side orchestration layer.
Key idea: in V2V, audio becomes part of the model's context. But the word "multimodal" doesn't tell you how the modalities are wired together or where the reasoning happens. For a closed system, you can only assert what's in the contract and the published capabilities.
Moshi makes a useful reference precisely because its paper, weights and code are public. The main components:
Don't picture the streams as a flat braid of hundreds of interleaved tokens. At each time step the temporal model receives the sum of the embeddings of the corresponding streams, and the depth model unrolls the dependencies between codebooks. The text positions often hold padding: the model isn't obliged to speak, or to "think in words," on every frame.
This construction lets Moshi handle overlap and short backchannels without a strict change of turns. The authors report a theoretical latency of 160 ms and about 200 ms in practice (DΓ©fossez et al., 2024). That's a measurement of one specific system, not the universal latency of token models.
Full-duplex is not a synonym for barge-in. With barge-in, the client can simply stop the playback when new speech arrives. A full-duplex model keeps jointly representing both channels and decides whether to speak, listen or stay silent many times per second.
V2V has no mandatory five-stage recipe. Some systems start from a text LLM, others from a self-supervised speech encoder; some train the speech generator entirely separately. What follows is a map of possible modules, not a universal invoice of GPU hours.
| Component | Job | Data | Required? |
|---|---|---|---|
| Neural codec | reconstruct sound after compression | audio without transcripts | only for codec-based models |
| Speech pretraining | learn acoustic and linguistic structure | audio, sometimes auto-transcripts | almost always, but the objective varies |
| Text backbone | knowledge, instructions, reasoning | text corpora | often reused from an existing LLM |
| Speechβtext alignment | connect audio and text | recordingβtranscript pairs | needed for a text interface |
| Dialogue adaptation | answer, don't just continue the recording | conversations and synthetic dialogues | needed for an assistant |
| Preference/safety tuning | manner, safety, helpfulness | comparisons and red-team data | method varies by team |
Public descriptions rarely let you split the compute honestly across these stages. Pretraining is usually the expensive part, but a number like "80%" or "95%" quoted without a specific model's report creates false precision. (An earlier draft of this article had exactly such a table; it did not survive the technical review.)
Codec training is a reconstruction problem:
Schematically:
$$\mathcal{L}{codec} = \lambda{spec}\mathcal{L}_{spec}
The exact recipe depends on the codec. Mimi, for example, uses adversarial and feature-matching objectives and separately distills its first codebook from WavLM. You can't reduce every neural codec to one "GAN plus mel L1" formula.
The data may carry no manual labels, but "unlabeled" doesn't mean "no strings attached." You still need licenses, removal of personal data, consent for private conversations, and control over which languages, channels and sound types the corpus actually contains.
For a discrete speech LM, one training task is predicting the next code:
This teaches phonetics, voice, prosody and the statistics of conversational speech. But "knowledge and reasoning emerge from raw audio" would be too strong a claim. The factual knowledge of a modern V2V system mostly arrives from the text backbone, the transcripts and the instruction tuning.
Moshi shows one real hybrid path: Helium is first trained as a text LLM, then the model goes through audio pretraining on millions of hours with automatic Whisper transcripts, and after that through multi-stream and instruction post-training (Moshi, Β§4). That's one published recipe. It is not a template you can project onto OpenAI, Google or Amazon.
For cross-entropy intuition there's a small example, ch05_cross_entropy.swift. It shows a single classification step; it does not simulate training a network.
A different line of research learns continuous speech representations:
These models output embeddings. To get discrete semantic tokens out of them, you usually need k-means, vector quantization or a separately learned tokenizer. The phrase "wav2vec gives you tokens" skips that step, and the step matters.
Audio-plus-transcript pairs support several tasks:
This is not a discrete stage that every model passes through. Whisper, for instance, is trained end to end as an encoder-decoder ASR and translation system on 680,000 hours of labeled or weakly labeled audio. Its encoder makes a fine frontend, but that alone doesn't turn Whisper into a V2V model, and it doesn't guarantee dialogue understanding.
A generator that plausibly continues a recording is not yet a system that answers questions. Dialogue tuning sets the roles, the instructions and the shape of a reply. Possible data sources:
Synthetic data scales beautifully, and it inherits the teacher's habits just as easily, mistakes included. If every training answer is voiced by the same TTS, the model may learn that one voice's narrow prosody instead of natural variety.
Sesame CSM illustrates a different scale of the task: the open checkpoint is trained to generate speech for text that already exists, given audio context. It doesn't decide what to answer, and its README plainly asks you to bring an external LLM (CSM README).
For speech, preferences can be collected along several axes:
These comparisons can feed reward modeling, RL, DPO-style objectives, or plain supervised fine-tuning on the winning answers. Public detail about the post-training of closed V2V models is scarce, so the honest move is to list the possible methods rather than assert that every system goes through PPO or DPO.
Preference tuning doesn't replace knowledge training and doesn't guarantee safety. It shifts the distribution of behavior within the data and tests you have. A production system still needs runtime guardrails, tool controls and escalation.
Full-duplex needs data where the speakers' channels are separated and synchronized. The model has to see silence, overlap, backchannels and half-finished turns. Possible signals:
Moshi fine-tunes its multi-stream behavior on Fisher and on synthetic dialogues. In a closed realtime API, turn-taking may be split between the main model, VAD, endpointing and client logic. Watching the reply stop when you interrupt doesn't tell you where that behavior was trained.
Low latency is a property of the whole chain: audio capture, end-of-turn detection, the network, the model, the decoder and the client's playout buffer. One "model latency" number won't tell you how the voice UX actually feels.
In human conversation, the median gap between turns is often close to 200 ms (Stivers et al., PNAS 2009). But a human can plan the reply while the other person is still talking; a server first has to recognize that the turn is over. So treat 200 ms as context, not as a universal SLA for your API.
It pays to take the chain apart:
For natural dialogue, teams usually aim at the first few hundred milliseconds, but the tolerable range depends on the language, the scenario and the accuracy of your endpointing. An aggressive endpoint buys you latency and pays for it with false interruptions.
Instead of waiting for the whole recording, the system receives fragments, say 20β240 ms each:
In a token system, one chunk becomes several time frames with several codebook codes per frame. In a system with a continuous speech encoder, there may be no discrete input tokens at all.
A simulation of VAD, endpointing and barge-in lives in ch06_streaming_chunks.swift. It's a state machine over a made-up energy value, not a production VAD.
In autoregressive decoding, the keys and values of past positions are kept around. A new token doesn't recompute their projections. Its query, however, still attends over the entire accumulated context.
For standard causal attention:
For a chunk of length over a past context of , the attention work is on the order of . Sliding windows, chunked attention, state-space layers and compaction can bound the growth, at the price of changing what context the model can see.
The memory of a standard multi-head KV-cache:
where is the number of layers, the number of KV heads, the context length, and bytes per element. For MHA, ; with GQA/MQA there are fewer KV heads. An upper-bound estimate lives in ch06_kv_cache_size.swift.
Key idea: the KV-cache removes repeated work. It does not make a long conversation free: both the memory and the attention over accumulated history stay on the bill.
VAD answers the question "is speech happening right now?" Endpointing answers "is the turn finished, may I speak?" These are different questions. A can sit in the middle of a sentence, and an "uh-huh" can be a brief reaction after which the main speaker carries on.
The cascade doesn't escape this either: a streaming ASR typically emits its final result exactly after endpointing. In a native system, the decision may come from a learned turn detector, from the main model, or from a combination.
Practical systems stack several signals:
What you evaluate is the trade-off: endpointing delay versus premature endpoints versus missed ones.
With plain barge-in, a system can run half-duplex:
A full-duplex model keeps consuming input even while producing output, and keeps deciding whether to continue talking. Moshi implements this in an open multi-stream architecture. GPT-Live, rolled out for ChatGPT Voice in July 2026, is also officially described as full-duplex; API access was still on the roadmap at the time of writing (OpenAI).
OpenAI Realtime and Gemini Live support streaming and interruption. That by itself doesn't prove a Moshi-like architecture inside. For closed APIs, describe the observed behavior and the documented events. Don't invent the model graph.
Under the "voice AI" label you'll be sold realtime agents, full-duplex models, TTS engines, audio generators and speech encoders. Comparing them in one table without naming the task is like comparing a browser, a database and a video codec. Technically, all three are software.
Product names and API status in this chapter were checked on July 18, 2026. For closed products we describe the interface and the published behavior, not the unknown architecture of the weights.
Job |
realtime voice agent with text, audio and tool calls |
Models |
gpt-realtime-2.1 , gpt-realtime-2.1-mini ,
plus specialized realtime models |
Transport |
WebRTC, WebSocket, SIP |
Mode |
streaming S2S with interruption; don't call it full-duplex by default |
GPT-Realtime-2.1 has a 128k context, configurable reasoning effort
and tool use. In the WebSocket API, a session runs on events like
input_audio_buffer.append
and
response.output_audio.delta
; in the browser, WebRTC is usually the saner option. Official pages: the model and the Realtime guide.
The API takes audio in and returns audio within one session. From that, you cannot conclude that a single transformer or any particular neural codec sits inside. The "audio tokens" in the billing are a pricing unit of the API, not a public spec of some internal RVQ.
GPT-Live-1 and GPT-Live-1 mini began rolling out to ChatGPT Voice in July 2026. OpenAI describes them as full-duplex outright: the model continuously decides whether to listen, speak, stay silent or hand off to a tool. For deep search and reasoning, the voice loop can delegate to a GPT-5.5-class model. Which makes GPT-Live a prime example of a hybrid architecture rather than evidence that one audio model does everything (announcement). API access to GPT-Live was still listed as planned when we checked.
Job |
a continuous multimodal session: audio, text, images and video |
Current model |
gemini-3.1-flash-live-preview |
Mode |
low-latency audio-to-audio, interruption, tool use |
Specialty |
camera/screen plus search grounding in one session |
Gemini 3.1 Flash Live Preview supports thinking levels and uses a different event format than the 2.5 generation. Note that proactive audio and affective dialog are not supported in 3.1 yet; they belong to compatible versions of Gemini 2.5 Flash Live. A useful reminder that capabilities attach to a model ID, not to a brand: the 3.1 model page, Live API.
Google's phrase "native audio" describes a product capability. The detailed graph of the closed model isn't published, so this article won't ascribe Mimi-like codebooks or any particular loss function to it.
Class | full-duplex speech-text dialogue model | Size | about 7B in the temporal backbone | Codec | Mimi: 12.5 frames/s, 8 codebooks | Access | open weights and code |
Moshi models separate, synchronized user and assistant streams, and handles overlap and backchannels. A large temporal transformer and a small depth transformer split the work between time and codebook dependencies. Its strength is a transparent research implementation. Its weakness: knowledge, language coverage and instruction following trail the big closed products. Sources: the paper and the repository.
SpeechGPT was an early study of cross-modal instruction following on discrete speech. Newer omni models, Qwen3-Omni among them, unify audio with text, images and video. How realtime and how duplex they are is a separate question from the modality list; check it separately.
Self-hosting an open model buys you control over where the processing happens. It does not hand you compliance for free: license, data provenance, vulnerabilities, logging, infrastructure and quality on your target language all remain your problem.
CSM-1B uses a Llama backbone and a small audio decoder to generate Mimi codes from text plus audio context. It's a fine demonstration of context-aware prosody. The open checkpoint, however, doesn't understand your question and doesn't write the reply. The README says it plainly: bring your own LLM (SesameAILabs/csm).
So CSM is an expressive speech generator, a TTS component for a cascade. Its conversational context helps it pick the voice and prosody for a line that's already written.
These works gave V2V its codecs and its generation machinery. Their place is in the history of the technology and in the output speech stack, not in a catalog of ready-made voice agents.
| Component | Examples |
|---|---|
| Streaming ASR | Whisper-derived services, Deepgram, Google STT |
| Text/agent model | GPT-class, Claude, Gemini, a local LLM |
| TTS | ElevenLabs, Cartesia, OpenAI TTS, Google Cloud TTS |
A cascade lets you optimize WER, reasoning and the voice independently. Its latency isn't necessarily the plain sum of three completed requests: partial ASR, streaming LLM output and incremental TTS overlap each other. But endpointing, context handoff and the client buffer still show up in the end-to-end number.
For legal, financial and tightly scripted agents, a readable transcript with separate policy checks can matter more than the last drop of natural prosody.
| You need | Candidates | What to check |
|---|---|---|
| A tool-using voice agent over an API | OpenAI Realtime, Gemini Live, Nova Sonic | language, TTFA P95, tools, data policy, price |
| Full-duplex research | Moshi | hardware, English-centric coverage, reasoning quality |
| Expressive speech for ready-made text | CSM or a specialized TTS | streaming, voice, license, watermarking |
| Maximum observability | ASR β LLM β TTS | WER, endpointing, prosody handoff |
| Self-hosted multimodality | Qwen3-Omni and other open models | real VRAM, throughput, license, evals |
Choose by the measurable user journey, not by the word "native." One and the same product can combine a realtime speech loop, a text agent and an external TTS fallback.
The same words can sound completely different. But the acoustic signal gives no direct access to a person's emotions. What a model observes is tempo, s, intensity and the pitch contour; "irritation" or "joy" is already an interpretation in context.
| Level | What we observe | Example |
|---|---|---|
| Semantics | ||
| the content of the words | "My order is two days late" | |
| Prosody | ||
| tempo, s, pitch contour, intensity | speeding up, stress on "two" | |
| Pragmatics | ||
| the intent in this context | complaint, request, irony | |
| Affect | ||
| a hypothesis about the speaker's state | irritation, fatigue, joy |
Prosody can be measured. Pragmatics and affect have to be inferred from a mix of signal, words, conversation history and cultural context. People disagree on those judgments too. Which is why "the model feels the emotion" works as a metaphor and fails as a technical description.
A minimal cascade passes nothing but text between the ASR and the LLM:
audio β ASR β "the order is late" β LLM β "I'm sorry" β TTS β audio
In that setup, the sigh, the and the stress on words all vanish. But notice what this is: a property of the interface between components, not an iron law of cascades. A richer pipeline can pass along:
Such a cascade is more work and can add latency. In exchange, every transformation can be inspected. A bank, for example, may simply want to answer calmly and neutrally at all times, without trying to classify the customer's mental state in the first place.
Native audio input carries more of the original signal all the way to the model's backbone. The system can use s, pace and pitch contour when shaping the answer. Here's what the signal's availability does not guarantee:
In a codec-based model, the semantically reinforced first codebook tends to carry coarse linguistic structure, while the later RVQ codebooks add residual acoustics. You can't say "the lower levels handle emotion": the information is distributed, and where it lands depends on how the tokenizer and the generator were trained.
Key idea: audio-to-audio preserves the possibility of using prosody. An appropriate reaction only appears when the data, the objectives and the product policy all pull in that direction.
An ordinary next-token loss contains no built-in notion of "empathetic." It penalizes mismatches with the target audio, nothing more. If the training dialogues systematically answer tense speech with calm speech, the model can pick up the correlation. If such pairs are missing, or the labeling contradicts itself, cross-entropy alone won't save you.
Extra sources of signal:
An absolute threshold like "pitch above 180 Hz means irritation" is a poor baseline for teaching and for production alike. Pitch depends on the speaker. Relative changes against that person's local manner work better, and even those shouldn't be auto-promoted into a diagnosis.
For compatible versions of Gemini 2.5 Flash Live, Google documents:
In the API this was switched on with
enable_affective_dialog
. The current
gemini-3.1-flash-live-preview
doesn't support it yet, and Google's migration guide says to remove that configuration outright (model guide).
That's a product distinction worth internalizing: the capability belongs to a specific model version, not to the Gemini brand. And the flag guarantees neither a correct reading of the emotion nor a substitute for your application's communication policy.
Ranking them on a single "who understands emotions best" scale doesn't work: different inputs, different tasks, different levels of openness.
An expressive voice raises trust, and raises the damage of a confidently spoken error by the same amount. The ability to sound empathetic has to grow together with fact-checking and safety. Not instead of them.
A demo answers the question "can this thing hold a pleasant five-minute chat?" Production asks different questions: reproducible measurements, control over tools, a data policy, and a clear idea of where the model ends and the orchestration begins.
Reasoning and knowledge. The fast speech loop is not required to be the strongest reasoning model in the building. GPT-Live, for one, officially delegates hard tasks to a separate frontier model. So measure the whole system: a beautiful voice tells you nothing about the math, the search or the tool use.
Long sessions. Audio representations are denser than text, the KV-cache and the attention bill keep growing, and behavior can degrade after compaction. Test more than the advertised context limit: check that decisions, names, numbers and unfinished actions survive across dozens of turns.
Languages and code-switching. Quality depends on a specific model's data. A sentence like "Π·Π°Π±ΡΠΊΠ°ΠΉ meeting Π½Π° maΓ±ana" stresses the ASR, the meaning, the tool schema and the pronunciation of the reply, all in one go. An average "quality on Russian" score won't show you the failures on names, addresses and part numbers.
Noise and the channel. 8 kHz telephony, echo, Bluetooth, music and other people's speech affect more than recognition: they hit VAD, endpointing, speaker attribution and the safety of tool calls.
Voice makes an error harder to check: the user sees no formula, no link, no exact spelling of a name. Confident prosody adds unearned trust on top.
For critical scenarios, asking the model to say "I'm not sure" is not a plan. You want:
And remember that voice prompt injection can arrive from the background: a radio, another person in the room, or a played recording can pronounce an instruction. Tool policy should rest on session permissions and confirmations, not on how confident the command sounded.
The risks include voice cloning, replay attacks, employee impersonation and automated social engineering. Watermarking and synthetic-speech detectors are useful layers of defense, but they aren't authentication: a mark can be damaged, and detectors have false positives.
The legal wording needs care:
None of this is legal advice. Before launch, you separately need a lawful basis, a retention policy, data residency, access rules for raw recordings, clarity on the provider's use of your data, and a deletion process. Self-hosting moves the processing; the duties come along with it.
Audio usually costs more than text: a denser representation, streaming decode, fewer batching opportunities. But "the GPU is pinned to one session the whole time" is too crude; servers multiplex and use continuous batching, and the actual utilization depends on the implementation.
For gpt-realtime-2.1
, input and output audio tokens cost $32 and $64 per million respectively at the time of checking (official model page). These billable tokens can't be derived from a neural codec's "50 frames per second": the provider defines its own tokenization and billing rules.
Count the cost from real usage records:
Account separately for the share of silence, cache discounts, reasoning effort, tool calls, retries, and the capacity reserve for P95 load. For a self-hosted model, add GPU/CPU, autoscaling, egress, observability, and the engineers who keep it all alive.
| Area | Metrics | How to test |
|---|---|---|
| End-to-end latency | P50/P95/P99 glass-to-glass; endpoint, model TTFA, network and playout measured separately | real devices, regions and target load |
| Input speech | WER/CER, entity and numeric accuracy | languages, accents, code-switching, noise, telephony |
| The task | task success, tool argument accuracy, confirmation rate | scenarios with a verifiable end state |
| Output speech | listening preference, MOS with confidence intervals, intelligibility | a blind panel; automatic audio judges only as a supporting signal |
| Content fidelity | omissions, repetitions, numbers and names | ASR round-trip for a given text; not a universal metric for open-ended replies |
| Turn-taking | premature/missed endpoints, interruption precision/recall, overlap quality | annotated two-channel conversations |
| Long sessions | retention of facts and unfinished actions after compaction | multi-turn tests at full length |
| Safety | unauthorized tool actions, replay, prompt injection, voice impersonation | red-team audio and background commands |
| Cost | gross and per-successful-session $/min, tokens/session, infrastructure utilization | real usage logs plus a load test |
A few rules make the comparison honest:
Key idea: "the most native" and "the fastest in the demo" are not product metrics. The system that wins is the one with the best task success at acceptable latency, risk and price on your traffic.
Likely directions for 2026β2027:
That's a forecast, not a guaranteed route. And cascades won't disappear: observability and independently replaceable components are durable advantages.
A reliable mental model of V2V looks like this:
If you understand transformers and LLMs, most of the machinery here will feel familiar. The new difficulty lives in multi-stream audio representations, endpointing, decoding, acoustic evaluation and continuous interaction.
Primary sources:
π Current documentation: