{"slug": "how-voice-to-voice-models-work-from-sound-wave-to-living-conversation", "title": "How voice-to-voice models work: from sound wave to living conversation", "summary": "Voice-to-voice (V2V) systems, also known as speech-to-speech (S2S), convert audio input to audio output using cascades of models including neural codecs, speech encoders, and transformers. The article explains the engineering behind V2V, covering sound representation, system architecture, training, streaming, and common pitfalls, aimed at IT professionals. It draws on the author's experience deploying voice agents at Newo.ai and provides code examples in Swift.", "body_md": "You say into your phone: \"Explain quantum entanglement in one minute.\" A beat later, a voice answers — not the robot from your GPS, but almost a real conversation partner: it pauses, it has intonation, it even drops a little \"hm\" before the tricky part. This is voice-to-voice (V2V), also called speech-to-speech (S2S): a system that takes audio in and answers with audio. In this article we'll take the trick apart, from the wave hitting your microphone to the sound coming out of your speaker.\n\nI spent a while at **Newo.ai**, deploying voice agents\ninto real projects on the European market. In production, the marketing\nline \"talks almost like a human\" decomposes quickly into measurable\nproblems: latency, false endpoints, mangled names, and a tone of voice\nthat's wrong for the moment.\n\nThat's where I first collided with **voice-to-voice**\nsystems for real — the ones that take speech in and answer with speech.\nFrom the outside it all looks simple. Inside there may be a cascade of\nseveral models, neural codecs, speech encoders, transformers, and months\nof training that the sales brochures never get around to mentioning. I\nwanted to know how the thing actually works and how V2V systems differ\nfrom one another — not only by their slide decks, but by their published\nmechanisms.\n\nSo I dug in, and this article is the result. It's for people who already have V2V in their work life and who — like me not so long ago — would rather see an understandable machine under the hood than a magic show.\n\nIf you know how large language models work (or read my book [ How LLMs Work: From Zero\nto Your Own Transformer](https://www.amazon.com/dp/B0H7BPJLHC)), you know the principle of\n\nThe article is written for an **IT professional**: you\nknow what an API is, what a GPU does, and roughly what a transformer\nlooks like, but you may never have touched speech ML. We won't flatten\nit into \"the neural net listens and talks.\" We'll walk the whole path:\nsound representations, ways to organize the system, training, streaming,\nand the places where it all breaks. And for closed models we'll keep\nobservable product behavior carefully separated from guesses about their\ninternals.\n\nThe main text explains the ideas in plain words, the **code\nexamples are in Swift** ([github.com/asaptf/v2v-models-article-en](https://github.com/asaptf/v2v-models-article-en)\n— every file is dependency-free and runs with a single command), and the\n**callouts** add engineering depth:\n\n| Icon | Callout type |\n|---|---|\n| 💡 | The key idea of a section |\n| 📐 | An analogy |\n| 🛠 | Engineering details (formulas and such) |\n| ⚠ | A common mistake |\n| 📜 | A bit of history |\n| 📚 | Links to models and papers |\n\nWhy Swift, of all things, and not Python? Because the examples here\nillustrate **ideas** (bitrate arithmetic, RVQ mechanics,\nstreaming logic), not real training code. If you have Swift installed,\nthe scripts run with zero external libraries. Real speech ML mostly\nlives in Python and PyTorch; links to working repositories and papers\nare in Chapter 7 and in the reading list at the end.\n\nFrom the outside, a V2V system could not look simpler:\n\nSpeech goes in — speech comes out. There's no intermediate text for\nyou to read (although inside, it sometimes exists anyway — more on that\nin Chapter 2). This entire article is about what happens\n**inside** that box.\n\n**The big idea of this article:** many modern V2V\nsystems borrow the ideas of language models and apply them to sound —\nautoregression, transformers, discrete speech representations. But how\nthe audio is represented and how the conversation is run are\n**separate engineering choices**. A token model can be\nturn-based or full-duplex, and a realtime API can quietly hide several\nmodels inside.\n\n**Analogy:** a text LLM is a writer who types the answer\nletter by letter. A token-based speech model is a radio announcer who\nassembles the answer from several parallel streams of codes — while also\ntracking pauses, interruptions and intonation, live on air. Both learned\ntheir craft the same way: from an enormous corpus of examples of \"what\ncame next.\"\n\nSpeech synthesis (TTS, *text-to-speech*) has been around for\ndecades: you provide **text**, you get a\n**voice**. V2V is a different beast:\n\n| TTS | V2V | |\n|---|---|---|\n| Input | text | audio (the user's speech) |\n| Understands meaning | doesn't have to | has to |\n| Role in the dialogue | voices a ready-made text | understands the input and forms the reply |\n| Latency | a single component's metric | measured end-to-end, for the whole dialogue |\n\nAt the product level, V2V combines the abilities to recognize speech, form a reply, and synthesize sound. Those abilities can live in three separate blocks, in a speech-native loop, or in a hybrid that delegates — and that classification is exactly where Chapter 2 picks up.\n\n**A terminology check:** an assistant built as a\n\"Whisper → GPT → ElevenLabs\" pipeline is properly called a\n**cascaded V2V system**, not a native S2S model. A cascade\nis easier to debug; but unless you pass extra acoustic features along,\nit loses the intonation and non-verbal cues that never made it into the\ntranscript.\n\nBefore a model can \"hear\" speech, sound has to become a representation it can work with. The raw wave, spectral features and codec tokens are measured in different units, so just counting elements tells you little — what matters is frame rate, the number of codebooks, and bitrate.\n\nA microphone measures the amplitude of sound pressure many times per second. Wideband speech typically uses 16 or 24 kHz: one second of mono audio at 24 kHz is 24,000 samples. At 16 bits per sample, that's 384 kbit/s before any container overhead.\n\nFeeding every sample into a vanilla transformer as its own position is expensive: attention cost grows quadratically with sequence length, and 24,000 positions per second is a lot of sequence. But it doesn't follow that neural networks can't handle waveforms at all: wav2vec 2.0, Whisper-style frontends and other encoders happily accept the raw wave — they just shrink its temporal resolution with convolutions or patches before anything reaches the attention blocks.\n\nSound is sliced into windows of roughly 20–30 ms, and for each window we compute how the energy is distributed across frequencies. With a 10 ms hop you get about 100 frames per second, each one a vector of, say, 80 mel channels.\n\nA spectrogram is convenient for a speech encoder, but \"24,000 samples\nversus 100 frames\" is **not** a compression ratio: each\nframe holds dozens of floating-point numbers. A spectrogram can easily\ntake *more* memory than the original PCM. Its job is to make the\nlocal structure of speech convenient for the model, not necessarily to\nmake the file smaller.\n\nBy the way, if you've ever flipped Audacity into spectrogram view, you've already seen this representation with your own eyes.\n\nA neural audio codec compresses the wave into a sequence of latent frames, and a quantizer replaces each frame with indices from codebooks. Two quantities matter here, and they're easy to conflate:\n\nIf frames/s and , the model receives not\n50 but up to **200 code indices per second**. These are\nwhat people informally call audio tokens. With a codebook of size , one index costs\n10 bits, and the useful bitrate comes out to roughly:\n\nThat's about 192 times less than 16-bit PCM at 24 kHz — and it's a far more honest calculation than dividing the number of samples by the number of frames.\n\nYou can run the numbers yourself in [ ch01_audio_compression.swift](https://github.com/asaptf/v2v-models-article-en/blob/main/code/ch01_audio_compression.swift):\n\n``` js\nlet sampleRate = 24_000\nlet pcmBitrate = sampleRate * 16                    // 384 kbit/s\n\nlet frameRate = 50\nlet codebooks = 4\nlet bitsPerCode = 10                                // K = 1024\nlet codecBitrate = frameRate * codebooks * bitsPerCode // 2 kbit/s\n```\n\n**Key idea:** the codec's frame rate and the number of\ndiscrete tokens are not the same thing. One time frame can carry several\nparallel RVQ codes. It's these parallel streams — not \"50 words of sound\nper second\" — that the transformer actually has to model.\n\n**Engineering notes: audio parameters.**\n\nA low-bitrate codec is negotiating a trade between intelligibility, naturalness and compute. Depending on the data and the loss function, the casualties may include:\n\nThe decoder's quality caps the **acoustic** quality of a\ntoken-based system — but not its understanding or its dialogue behavior.\nAnd a token codec is only one path: some speech models work with\ncontinuous latents or generate spectral features instead.\n\n**Common mistake:** treating a codec token as a phoneme\nor a syllable. An individual index describes one quantized component of\na short audio frame. Its link to phonetics can be strong for a\nsemantically trained codebook — but a fixed \"token ↔︎ speech sound\"\nmapping usually doesn't exist.\n\nCascade, audio tokens and full-duplex are often presented as three competing \"architectures.\" That mixes up different questions. It's cleaner to place a system on three independent axes: what components it's built from, how it represents sound, and how it runs a conversation.\n\nThe classic scheme wires up three blocks:\n\n```\n🎤 audio → [ASR] → text → [LLM] → text → [TTS] → 🔊 audio\n```\n\nWhat the cascade gets you:\n\nWhat it costs you:\n\nA cascade is not doomed to sound bad. Streaming ASR, a fast LLM, an expressive TTS and separately passed acoustic features can bring it close to native systems. And its real advantage isn't only the quick prototype: it's control.\n\nIn a speech-native (or end-to-end) system, audio is an input and output modality of the model loop itself. There is no mandatory text interface in the middle that a developer gets to see. Inside, there may still be a text stream, several transformers, a codec and assorted helper models.\n\nFor open models you can verify the architecture against the paper and the code. For closed APIs, the words \"native audio\" describe a documented product capability. They do not prove that a single transformer lives inside.\n\nA practical system often combines the two: a fast speech model keeps\nthe conversation flowing and hands the hard questions to a text\nreasoning model or an agent. GPT-Live works this way: continuous voice\ninteraction is separated from deeper reasoning, which may run on a\ndifferent model ([OpenAI\nannouncement, July 8, 2026](https://openai.com/index/introducing-gpt-live/)).\n\nA codec turns each audio frame into several RVQ indices. The transformer predicts them sequentially or hierarchically, and a decoder reconstructs the wave. Moshi and the speech generator in Sesame CSM are built this way.\n\n```\naudio → [codec] → several code streams → [model] → codes → [decoder] → audio\n```\n\nDiscretization is convenient for cross-entropy and autoregression. The price is quantization, plus a lot of parallel code streams to keep track of.\n\nA model can also consume and predict continuous latents or mel features, with no codebook involved. That sidesteps part of the quantization loss but calls for a different loss function and decoder.\n\nVoicebox and other generators shape an audio fragment iteratively or non-autoregressively. These methods are strong in TTS and audio editing; causal realtime streaming is harder for them. A hybrid can plan the semantics autoregressively and let a fast non-autoregressive decoder paint in the acoustics.\n\n**Key idea:** \"native\" doesn't mean \"necessarily\ntoken-based,\" and \"token-based\" doesn't mean \"necessarily realtime.\"\nThese are independent properties.\n\nThe system waits for your turn to end, then answers. The boundary comes from VAD, an endpointing model, the ASR, or some combination of them. Many cascades and speech generators work like this.\n\nAudio is sent in chunks, the reply is synthesized while it's still being generated, and fresh user speech can stop the playback. Barge-in improves how the interface feels. By itself it doesn't mean the model keeps meaningfully listening while it talks.\n\nThe system models the incoming and outgoing channels all the time. It\ncan listen, speak, stay silent, drop a quick \"uh-huh,\" and handle\noverlap without a hard turn boundary. The open reference is Moshi: two\naudio streams represented in parallel, with a claimed practical latency\nof about 200 ms ([Défossez et\nal., 2024](https://arxiv.org/abs/2410.00037)). In 2026, full-duplex also reached mass-market ChatGPT\nVoice via GPT-Live.\n\n| System | Organization | Representation | Mode | The fine print |\n|---|---|---|---|---|\n| ASR → LLM → TTS | cascade | text between blocks | turn-based or streaming | acoustic features can be passed alongside |\n| AudioLM | audio generator | discrete tokens | audio continuation | not a dialogue assistant |\n| VALL-E | TTS generator | codec tokens | turn-based | takes text plus a voice prompt |\n| Sesame CSM-1B | speech generator | Mimi/RVQ | turn-based | needs an external LLM for the reply's meaning |\n| SpeechGPT | speech-language model | discrete speech + text | turn-based | research model |\n| Moshi | speech-native | Mimi, parallel streams | full-duplex | paper, weights and code are open |\n| OpenAI Realtime / Gemini Live | speech-native by API contract | internals closed | streaming, barge-in | architecture can't be inferred from the API |\n| GPT-Live | hybrid system | internals closed | full-duplex | deep tasks delegated to a frontier model |\n\nAudioLM positions itself as an audio-continuation model ([Borsos et al., 2022](https://arxiv.org/abs/2209.03143)), VALL-E\nas text-to-speech ([Microsoft\nResearch](https://www.microsoft.com/en-us/research/project/vall-e-x/overview/)), and the open CSM flat-out recommends a separate LLM for\nwriting the reply ([SesameAILabs/csm](https://github.com/SesameAILabs/csm)). All\nthree matter for the history of codec language models. None of them is a\nready-made V2V conversation partner.\n\n| You need | Where to start |\n|---|---|\n| A rigid script, audits, compliance | a cascade with a transcript and deterministic checks |\n| Expressive voicing of ready-made text | a specialized TTS or speech generator |\n| A fast multimodal agent | a realtime API, measured on your network and your language |\n| Overlap and natural turn-taking | a system whose full-duplex claim you've verified separately |\n| Data that can't leave your infrastructure | self-hosted components, after checking quality, license and ops cost |\n\nThe trend is not an inevitable march from cascades toward one-model systems. Native systems keep improving naturalness and latency; cascades keep their edge in observability and control. What wins in production, more and more often, is not a pure class but a measured hybrid.\n\nFor many token-based speech models, the codec is the bridge between the waveform and the transformer. It reduces temporal resolution and turns each continuous latent frame into a handful of discrete indices. An important way to represent sound — but not the only one.\n\nA typical neural codec has three parts:\n\nThe codec is trained as an autoencoder: the reconstruction should resemble the original by spectral and perceptual criteria. Adversarial and feature-matching losses are added so that the output doesn't drift into dull, averaged-out sound.\n\nIn residual vector quantization, each codebook refines the error left by the previous one:\n\nA simplified demo lives in [ ch03_rvq_demo.swift](https://github.com/asaptf/v2v-models-article-en/blob/main/code/ch03_rvq_demo.swift):\n\n``` js\nlet (t1, q1) = nearest(z, in: codebookL1)\nlet r1 = subtract(z, q1)\nlet (t2, q2) = nearest(r1, in: codebookL2)\nlet reconstructed = add(q1, q2) // z ≈ q1 + q2\n```\n\nFor one time frame, the formula is:\n\nAt frame rate with codebooks, the codec emits indices per second if you count every codebook token separately. A model can predict them as one flat sequence, with parallel heads, or with a hierarchy of temporal and depth transformers.\n\n**Analogy:** the first layer names the rough color of a\nwall, the next ones refine the shade and the finish. Keep in mind,\nthough, that all these refinements describe one moment in time — not\nsuccessive slices of speech.\n\n| Codec | Author | What matters |\n|---|---|---|\nSoundStream |\nGoogle, 2021 | causal codec with RVQ and adversarial training |\nEnCodec |\nMeta, 2022 | open implementation for speech and music |\nDAC |\nDescript, 2023 | high quality, open weights |\nMimi |\nKyutai, 2024 | 12.5 frames/s, 8 codebooks; the first distilled from WavLM |\n\nMimi outputs eight synchronized code streams at 12.5 Hz: up to 100\nindices per second for a single audio channel. In Moshi, a large\ntemporal transformer walks across time frames while a small depth\ntransformer models the dependencies between codebooks inside a frame.\nThat's how a second of sound avoids becoming a flat sequence of a\nhundred steps ([Moshi,\n§3.3–3.4](https://arxiv.org/pdf/2410.00037)).\n\nThe codec is often trained separately and frozen before the speech LM learns on top of it. Often, not always: joint fine-tuning, continuous speech encoders and RVQ-free decoders all exist in the wild.\n\nThe customary split goes like this:\n\nThis is not a clean \"what was said / how it was said\" decomposition. A semantic token is not a ready-made piece of meaning, and an acoustic codebook can carry phonetic information of its own.\n\nAudioLM first models discretized w2v-BERT features, then SoundStream\ncodes. In Mimi, the first RVQ codebook is distilled from WavLM and the\nremaining ones encode the residual acoustics. So the accurate phrase is\na **semantically reinforced first codebook**, not a stream\nwhere meaning is fully separated from voice.\n\n**Common mistake:** contrasting neural codecs with MP3\nas if one existed only for transformers and the other only for human\nears. Both solve perceptual compression. The real differences are the\nlearned representation, the latency, the bitrate, and how convenient the\nresulting codes are as targets for a generative model.\n\nThere is no such thing as \"the\" V2V architecture. This chapter walks through a common autoregressive variant built on a discrete codec, then looks at Moshi as a documented full-duplex example. Closed realtime APIs cannot be reconstructed from this scheme.\n\nA token-based speech-to-speech system is convenient to picture as four blocks:\n\n```\n┌──────────┐    ┌─────────────┐    ┌──────────────┐    ┌──────────┐\n│  Input   │ →  │ Transformer │ →  │   heads /    │ →  │  audio   │\n│  encoder │    │  backbone   │    │ depth model  │    │  decoder │\n└──────────┘    └─────────────┘    └──────────────┘    └──────────┘\n```\n\n**A terminology trap:** \"decoder\" gets used both for the\ntransformer decoder and for the audio decoder. \"End-to-end\" is slippery\ntoo: it may mean nothing more than the absence of a text boundary in the\npublic interface, not that a single network does everything.\n\nAutoregressive speech LMs run on familiar machinery:\n\nFor a single discrete stream, the cross-entropy looks like it always does:\n\nWith codebooks there are usually several output distributions and a summed loss:\n\nText and audio don't have to share one vocabulary. In Moshi they're synchronized parallel streams; other models use separate encoders, projectors and heads. So \"special tokens simply separate the modalities in one string\" is one possible implementation, not a definition.\n\nYou can poke at single-step cross-entropy in [ ch05_cross_entropy.swift](https://github.com/asaptf/v2v-models-article-en/blob/main/code/ch05_cross_entropy.swift).\n\nAt the level of an API or a training example, the context can be written like this:\n\n```\n[SYSTEM_TEXT] the instruction\n[USER_AUDIO] input frames or embeddings\n[TOOL_RESULT] application data, if a tool was called\n[ASSISTANT_AUDIO] the reply being generated\n```\n\nThis is a logical view, not a required physical layout of tensors. The system prompt may feed a text backbone, the audio may go through its own encoder, and the tool result may arrive via a separate model or a server-side orchestration layer.\n\n**Key idea:** in V2V, audio becomes part of the model's\ncontext. But the word \"multimodal\" doesn't tell you how the modalities\nare wired together or where the reasoning happens. For a closed system,\nyou can only assert what's in the contract and the published\ncapabilities.\n\nMoshi makes a useful reference precisely because its paper, weights and code are public. The main components:\n\nDon't picture the streams as a flat braid of hundreds of interleaved tokens. At each time step the temporal model receives the sum of the embeddings of the corresponding streams, and the depth model unrolls the dependencies between codebooks. The text positions often hold padding: the model isn't obliged to speak, or to \"think in words,\" on every frame.\n\nThis construction lets Moshi handle overlap and short backchannels\nwithout a strict change of turns. The authors report a theoretical\nlatency of 160 ms and about 200 ms in practice ([Défossez et al., 2024](https://arxiv.org/abs/2410.00037)).\nThat's a measurement of one specific system, not the universal latency\nof token models.\n\n**Full-duplex is not a synonym for barge-in.** With\nbarge-in, the client can simply stop the playback when new speech\narrives. A full-duplex model keeps jointly representing both channels\nand decides whether to speak, listen or stay silent many times per\nsecond.\n\nV2V has no mandatory five-stage recipe. Some systems start from a text LLM, others from a self-supervised speech encoder; some train the speech generator entirely separately. What follows is a map of possible modules, not a universal invoice of GPU hours.\n\n| Component | Job | Data | Required? |\n|---|---|---|---|\n| Neural codec | reconstruct sound after compression | audio without transcripts | only for codec-based models |\n| Speech pretraining | learn acoustic and linguistic structure | audio, sometimes auto-transcripts | almost always, but the objective varies |\n| Text backbone | knowledge, instructions, reasoning | text corpora | often reused from an existing LLM |\n| Speech–text alignment | connect audio and text | recording–transcript pairs | needed for a text interface |\n| Dialogue adaptation | answer, don't just continue the recording | conversations and synthetic dialogues | needed for an assistant |\n| Preference/safety tuning | manner, safety, helpfulness | comparisons and red-team data | method varies by team |\n\nPublic descriptions rarely let you split the compute honestly across these stages. Pretraining is usually the expensive part, but a number like \"80%\" or \"95%\" quoted without a specific model's report creates false precision. (An earlier draft of this article had exactly such a table; it did not survive the technical review.)\n\nCodec training is a reconstruction problem:\n\nSchematically:\n\n$$\\mathcal{L}*{codec} = \\lambda*{spec}\\mathcal{L}_{spec}\n\nThe exact recipe depends on the codec. Mimi, for example, uses adversarial and feature-matching objectives and separately distills its first codebook from WavLM. You can't reduce every neural codec to one \"GAN plus mel L1\" formula.\n\nThe data may carry no manual labels, but \"unlabeled\" doesn't mean \"no strings attached.\" You still need licenses, removal of personal data, consent for private conversations, and control over which languages, channels and sound types the corpus actually contains.\n\nFor a discrete speech LM, one training task is predicting the next code:\n\nThis teaches phonetics, voice, prosody and the statistics of conversational speech. But \"knowledge and reasoning emerge from raw audio\" would be too strong a claim. The factual knowledge of a modern V2V system mostly arrives from the text backbone, the transcripts and the instruction tuning.\n\nMoshi shows one real hybrid path: Helium is first trained as a text\nLLM, then the model goes through audio pretraining on millions of hours\nwith automatic Whisper transcripts, and after that through multi-stream\nand instruction post-training ([Moshi, §4](https://arxiv.org/pdf/2410.00037)). That's one\npublished recipe. It is not a template you can project onto OpenAI,\nGoogle or Amazon.\n\nFor cross-entropy intuition there's a small example, [ ch05_cross_entropy.swift](https://github.com/asaptf/v2v-models-article-en/blob/main/code/ch05_cross_entropy.swift).\nIt shows a single classification step; it does not simulate training a\nnetwork.\n\nA different line of research learns continuous speech representations:\n\nThese models output embeddings. To get **discrete semantic\ntokens** out of them, you usually need k-means, vector\nquantization or a separately learned tokenizer. The phrase \"wav2vec\ngives you tokens\" skips that step, and the step matters.\n\nAudio-plus-transcript pairs support several tasks:\n\nThis is not a discrete stage that every model passes through. Whisper, for instance, is trained end to end as an encoder-decoder ASR and translation system on 680,000 hours of labeled or weakly labeled audio. Its encoder makes a fine frontend, but that alone doesn't turn Whisper into a V2V model, and it doesn't guarantee dialogue understanding.\n\nA generator that plausibly continues a recording is not yet a system that answers questions. Dialogue tuning sets the roles, the instructions and the shape of a reply. Possible data sources:\n\nSynthetic data scales beautifully, and it inherits the teacher's habits just as easily, mistakes included. If every training answer is voiced by the same TTS, the model may learn that one voice's narrow prosody instead of natural variety.\n\nSesame CSM illustrates a different scale of the task: the open\ncheckpoint is trained to generate speech for **text that already\nexists**, given audio context. It doesn't decide what to answer,\nand its README plainly asks you to bring an external LLM ([CSM README](https://github.com/SesameAILabs/csm)).\n\nFor speech, preferences can be collected along several axes:\n\nThese comparisons can feed reward modeling, RL, DPO-style objectives, or plain supervised fine-tuning on the winning answers. Public detail about the post-training of closed V2V models is scarce, so the honest move is to list the possible methods rather than assert that every system goes through PPO or DPO.\n\nPreference tuning doesn't replace knowledge training and doesn't guarantee safety. It shifts the distribution of behavior within the data and tests you have. A production system still needs runtime guardrails, tool controls and escalation.\n\nFull-duplex needs data where the speakers' channels are separated and synchronized. The model has to see silence, overlap, backchannels and half-finished turns. Possible signals:\n\nMoshi fine-tunes its multi-stream behavior on Fisher and on synthetic dialogues. In a closed realtime API, turn-taking may be split between the main model, VAD, endpointing and client logic. Watching the reply stop when you interrupt doesn't tell you where that behavior was trained.\n\nLow latency is a property of the whole chain: audio capture, end-of-turn detection, the network, the model, the decoder and the client's playout buffer. One \"model latency\" number won't tell you how the voice UX actually feels.\n\nIn human conversation, the median gap between turns is often close to\n200 ms ([Stivers et al.,\nPNAS 2009](https://www.pnas.org/doi/10.1073/pnas.0903616106)). But a human can plan the reply while the other person is\nstill talking; a server first has to recognize that the turn is over. So\ntreat 200 ms as context, not as a universal SLA for your API.\n\nIt pays to take the chain apart:\n\nFor natural dialogue, teams usually aim at the first few hundred milliseconds, but the tolerable range depends on the language, the scenario and the accuracy of your endpointing. An aggressive endpoint buys you latency and pays for it with false interruptions.\n\nInstead of waiting for the whole recording, the system receives fragments, say 20–240 ms each:\n\nIn a token system, one chunk becomes several time frames with several codebook codes per frame. In a system with a continuous speech encoder, there may be no discrete input tokens at all.\n\nA simulation of VAD, endpointing and barge-in lives in [ ch06_streaming_chunks.swift](https://github.com/asaptf/v2v-models-article-en/blob/main/code/ch06_streaming_chunks.swift).\nIt's a state machine over a made-up energy value, not a production\nVAD.\n\nIn autoregressive decoding, the keys and values of past positions are kept around. A new token doesn't recompute their projections. Its query, however, still attends over the entire accumulated context.\n\nFor standard causal attention:\n\nFor a chunk of length over a past context of , the attention work is on the order of . Sliding windows, chunked attention, state-space layers and compaction can bound the growth, at the price of changing what context the model can see.\n\nThe memory of a standard multi-head KV-cache:\n\nwhere is the number of layers, the number of\n**KV heads**, the context length, and bytes per element. For MHA, ; with GQA/MQA\nthere are fewer KV heads. An upper-bound estimate lives in [ ch06_kv_cache_size.swift](https://github.com/asaptf/v2v-models-article-en/blob/main/code/ch06_kv_cache_size.swift).\n\n**Key idea:** the KV-cache removes repeated work. It\ndoes not make a long conversation free: both the memory and the\nattention over accumulated history stay on the bill.\n\n**VAD** answers the question \"is speech happening right\nnow?\" **Endpointing** answers \"is the turn finished, may I\nspeak?\" These are different questions. A pause can sit in the middle of\na sentence, and an \"uh-huh\" can be a brief reaction after which the main\nspeaker carries on.\n\nThe cascade doesn't escape this either: a streaming ASR typically emits its final result exactly after endpointing. In a native system, the decision may come from a learned turn detector, from the main model, or from a combination.\n\nPractical systems stack several signals:\n\nWhat you evaluate is the trade-off: endpointing delay versus premature endpoints versus missed ones.\n\nWith plain barge-in, a system can run half-duplex:\n\nA full-duplex model keeps consuming input even while producing\noutput, and keeps deciding whether to continue talking. Moshi implements\nthis in an open multi-stream architecture. GPT-Live, rolled out for\nChatGPT Voice in July 2026, is also officially described as full-duplex;\nAPI access was still on the roadmap at the time of writing ([OpenAI](https://openai.com/index/introducing-gpt-live/)).\n\nOpenAI Realtime and Gemini Live support streaming and interruption. That by itself doesn't prove a Moshi-like architecture inside. For closed APIs, describe the observed behavior and the documented events. Don't invent the model graph.\n\nUnder the \"voice AI\" label you'll be sold realtime agents, full-duplex models, TTS engines, audio generators and speech encoders. Comparing them in one table without naming the task is like comparing a browser, a database and a video codec. Technically, all three are software.\n\nProduct names and API status in this chapter were checked on\n**July 18, 2026**. For closed products we describe the\ninterface and the published behavior, not the unknown architecture of\nthe weights.\n\nJob |\nrealtime voice agent with text, audio and tool calls |\nModels |\n`gpt-realtime-2.1` , `gpt-realtime-2.1-mini` ,\nplus specialized realtime models |\nTransport |\nWebRTC, WebSocket, SIP |\nMode |\nstreaming S2S with interruption; don't call it full-duplex by default |\n\nGPT-Realtime-2.1 has a 128k context, configurable reasoning effort\nand tool use. In the WebSocket API, a session runs on events like\n`input_audio_buffer.append`\n\nand\n`response.output_audio.delta`\n\n; in the browser, WebRTC is\nusually the saner option. Official pages: [the\nmodel](https://developers.openai.com/api/docs/models/gpt-realtime-2.1) and the [Realtime\nguide](https://developers.openai.com/api/docs/guides/realtime).\n\nThe API takes audio in and returns audio within one session. From that, you cannot conclude that a single transformer or any particular neural codec sits inside. The \"audio tokens\" in the billing are a pricing unit of the API, not a public spec of some internal RVQ.\n\nGPT-Live-1 and GPT-Live-1 mini began rolling out to ChatGPT Voice in\nJuly 2026. OpenAI describes them as full-duplex outright: the model\ncontinuously decides whether to listen, speak, stay silent or hand off\nto a tool. For deep search and reasoning, the voice loop can delegate to\na GPT-5.5-class model. Which makes GPT-Live a prime example of a\n**hybrid** architecture rather than evidence that one audio\nmodel does everything ([announcement](https://openai.com/index/introducing-gpt-live/)).\nAPI access to GPT-Live was still listed as planned when we checked.\n\nJob |\na continuous multimodal session: audio, text, images and video |\nCurrent model |\n`gemini-3.1-flash-live-preview` |\nMode |\nlow-latency audio-to-audio, interruption, tool use |\nSpecialty |\ncamera/screen plus search grounding in one session |\n\nGemini 3.1 Flash Live Preview supports thinking levels and uses a\ndifferent event format than the 2.5 generation. Note that\n**proactive audio and affective dialog are not supported in 3.1\nyet**; they belong to compatible versions of Gemini 2.5 Flash\nLive. A useful reminder that capabilities attach to a model ID, not to a\nbrand: [the\n3.1 model page](https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-live-preview), [Live API](https://ai.google.dev/gemini-api/docs/live-api).\n\nGoogle's phrase \"native audio\" describes a product capability. The detailed graph of the closed model isn't published, so this article won't ascribe Mimi-like codebooks or any particular loss function to it.\n\nClass |\nfull-duplex speech-text dialogue model |\nSize |\nabout 7B in the temporal backbone |\nCodec |\nMimi: 12.5 frames/s, 8 codebooks |\nAccess |\nopen weights and code |\n\nMoshi models separate, synchronized user and assistant streams, and\nhandles overlap and backchannels. A large temporal transformer and a\nsmall depth transformer split the work between time and codebook\ndependencies. Its strength is a transparent research implementation. Its\nweakness: knowledge, language coverage and instruction following trail\nthe big closed products. Sources: [the paper](https://arxiv.org/abs/2410.00037) and [the repository](https://github.com/kyutai-labs/moshi).\n\nSpeechGPT was an early study of cross-modal instruction following on discrete speech. Newer omni models, Qwen3-Omni among them, unify audio with text, images and video. How realtime and how duplex they are is a separate question from the modality list; check it separately.\n\nSelf-hosting an open model buys you control over where the processing happens. It does not hand you compliance for free: license, data provenance, vulnerabilities, logging, infrastructure and quality on your target language all remain your problem.\n\nCSM-1B uses a Llama backbone and a small audio decoder to generate\nMimi codes from **text plus audio context**. It's a fine\ndemonstration of context-aware prosody. The open checkpoint, however,\ndoesn't understand your question and doesn't write the reply. The README\nsays it plainly: bring your own LLM ([SesameAILabs/csm](https://github.com/SesameAILabs/csm)).\n\nSo CSM is an expressive speech generator, a TTS component for a cascade. Its conversational context helps it pick the voice and prosody for a line that's already written.\n\nThese works gave V2V its codecs and its generation machinery. Their place is in the history of the technology and in the output speech stack, not in a catalog of ready-made voice agents.\n\n| Component | Examples |\n|---|---|\n| Streaming ASR | Whisper-derived services, Deepgram, Google STT |\n| Text/agent model | GPT-class, Claude, Gemini, a local LLM |\n| TTS | ElevenLabs, Cartesia, OpenAI TTS, Google Cloud TTS |\n\nA cascade lets you optimize WER, reasoning and the voice independently. Its latency isn't necessarily the plain sum of three completed requests: partial ASR, streaming LLM output and incremental TTS overlap each other. But endpointing, context handoff and the client buffer still show up in the end-to-end number.\n\nFor legal, financial and tightly scripted agents, a readable transcript with separate policy checks can matter more than the last drop of natural prosody.\n\n| You need | Candidates | What to check |\n|---|---|---|\n| A tool-using voice agent over an API | OpenAI Realtime, Gemini Live, Nova Sonic | language, TTFA P95, tools, data policy, price |\n| Full-duplex research | Moshi | hardware, English-centric coverage, reasoning quality |\n| Expressive speech for ready-made text | CSM or a specialized TTS | streaming, voice, license, watermarking |\n| Maximum observability | ASR → LLM → TTS | WER, endpointing, prosody handoff |\n| Self-hosted multimodality | Qwen3-Omni and other open models | real VRAM, throughput, license, evals |\n\nChoose by the measurable user journey, not by the word \"native.\" One and the same product can combine a realtime speech loop, a text agent and an external TTS fallback.\n\nThe same words can sound completely different. But the acoustic signal gives no direct access to a person's emotions. What a model observes is tempo, pauses, intensity and the pitch contour; \"irritation\" or \"joy\" is already an interpretation in context.\n\n| Level | What we observe | Example |\n|---|---|---|\nSemantics |\nthe content of the words | \"My order is two days late\" |\nProsody |\ntempo, pauses, pitch contour, intensity | speeding up, stress on \"two\" |\nPragmatics |\nthe intent in this context | complaint, request, irony |\nAffect |\na hypothesis about the speaker's state | irritation, fatigue, joy |\n\nProsody can be measured. Pragmatics and affect have to be inferred from a mix of signal, words, conversation history and cultural context. People disagree on those judgments too. Which is why \"the model feels the emotion\" works as a metaphor and fails as a technical description.\n\nA minimal cascade passes nothing but text between the ASR and the LLM:\n\n```\naudio → ASR → \"the order is late\" → LLM → \"I'm sorry\" → TTS → audio\n```\n\nIn that setup, the sigh, the pause and the stress on words all\nvanish. But notice what this is: a property of the **interface\nbetween components**, not an iron law of cascades. A richer\npipeline can pass along:\n\nSuch a cascade is more work and can add latency. In exchange, every transformation can be inspected. A bank, for example, may simply want to answer calmly and neutrally at all times, without trying to classify the customer's mental state in the first place.\n\nNative audio input carries more of the original signal all the way to\nthe model's backbone. The system **can** use pauses, pace\nand pitch contour when shaping the answer. Here's what the signal's\navailability does not guarantee:\n\nIn a codec-based model, the semantically reinforced first codebook tends to carry coarse linguistic structure, while the later RVQ codebooks add residual acoustics. You can't say \"the lower levels handle emotion\": the information is distributed, and where it lands depends on how the tokenizer and the generator were trained.\n\n**Key idea:** audio-to-audio preserves the\n*possibility* of using prosody. An appropriate reaction only\nappears when the data, the objectives and the product policy all pull in\nthat direction.\n\nAn ordinary next-token loss contains no built-in notion of \"empathetic.\" It penalizes mismatches with the target audio, nothing more. If the training dialogues systematically answer tense speech with calm speech, the model can pick up the correlation. If such pairs are missing, or the labeling contradicts itself, cross-entropy alone won't save you.\n\nExtra sources of signal:\n\nAn absolute threshold like \"pitch above 180 Hz means irritation\" is a poor baseline for teaching and for production alike. Pitch depends on the speaker. Relative changes against that person's local manner work better, and even those shouldn't be auto-promoted into a diagnosis.\n\nFor compatible versions of Gemini 2.5 Flash Live, Google documents:\n\nIn the API this was switched on with\n`enable_affective_dialog`\n\n. The current\n`gemini-3.1-flash-live-preview`\n\ndoesn't support it yet, and\nGoogle's migration guide says to remove that configuration outright ([model\nguide](https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-live-preview)).\n\nThat's a product distinction worth internalizing: the capability belongs to a specific model version, not to the Gemini brand. And the flag guarantees neither a correct reading of the emotion nor a substitute for your application's communication policy.\n\nRanking them on a single \"who understands emotions best\" scale doesn't work: different inputs, different tasks, different levels of openness.\n\nAn expressive voice raises trust, and raises the damage of a confidently spoken error by the same amount. The ability to sound empathetic has to grow together with fact-checking and safety. Not instead of them.\n\nA demo answers the question \"can this thing hold a pleasant five-minute chat?\" Production asks different questions: reproducible measurements, control over tools, a data policy, and a clear idea of where the model ends and the orchestration begins.\n\n**Reasoning and knowledge.** The fast speech loop is not\nrequired to be the strongest reasoning model in the building. GPT-Live,\nfor one, officially delegates hard tasks to a separate frontier model.\nSo measure the whole system: a beautiful voice tells you nothing about\nthe math, the search or the tool use.\n\n**Long sessions.** Audio representations are denser than\ntext, the KV-cache and the attention bill keep growing, and behavior can\ndegrade after compaction. Test more than the advertised context limit:\ncheck that decisions, names, numbers and unfinished actions survive\nacross dozens of turns.\n\n**Languages and code-switching.** Quality depends on a\nspecific model's data. A sentence like \"забукай meeting на mañana\"\nstresses the ASR, the meaning, the tool schema and the pronunciation of\nthe reply, all in one go. An average \"quality on Russian\" score won't\nshow you the failures on names, addresses and part numbers.\n\n**Noise and the channel.** 8 kHz telephony, echo,\nBluetooth, music and other people's speech affect more than recognition:\nthey hit VAD, endpointing, speaker attribution and the safety of tool\ncalls.\n\nVoice makes an error harder to check: the user sees no formula, no link, no exact spelling of a name. Confident prosody adds unearned trust on top.\n\nFor critical scenarios, asking the model to say \"I'm not sure\" is not a plan. You want:\n\nAnd remember that voice prompt injection can arrive from the background: a radio, another person in the room, or a played recording can pronounce an instruction. Tool policy should rest on session permissions and confirmations, not on how confident the command sounded.\n\nThe risks include voice cloning, replay attacks, employee impersonation and automated social engineering. Watermarking and synthetic-speech detectors are useful layers of defense, but they aren't authentication: a mark can be damaged, and detectors have false positives.\n\nThe legal wording needs care:\n\nNone of this is legal advice. Before launch, you separately need a lawful basis, a retention policy, data residency, access rules for raw recordings, clarity on the provider's use of your data, and a deletion process. Self-hosting moves the processing; the duties come along with it.\n\nAudio usually costs more than text: a denser representation, streaming decode, fewer batching opportunities. But \"the GPU is pinned to one session the whole time\" is too crude; servers multiplex and use continuous batching, and the actual utilization depends on the implementation.\n\nFor `gpt-realtime-2.1`\n\n, input and output audio tokens cost\n$32 and $64 per million respectively at the time of checking ([official\nmodel page](https://developers.openai.com/api/docs/models/gpt-realtime-2.1)). These billable tokens can't be derived from a neural\ncodec's \"50 frames per second\": the provider defines its own\ntokenization and billing rules.\n\nCount the cost from real usage records:\n\nAccount separately for the share of silence, cache discounts, reasoning effort, tool calls, retries, and the capacity reserve for P95 load. For a self-hosted model, add GPU/CPU, autoscaling, egress, observability, and the engineers who keep it all alive.\n\n| Area | Metrics | How to test |\n|---|---|---|\n| End-to-end latency | P50/P95/P99 glass-to-glass; endpoint, model TTFA, network and playout measured separately | real devices, regions and target load |\n| Input speech | WER/CER, entity and numeric accuracy | languages, accents, code-switching, noise, telephony |\n| The task | task success, tool argument accuracy, confirmation rate | scenarios with a verifiable end state |\n| Output speech | listening preference, MOS with confidence intervals, intelligibility | a blind panel; automatic audio judges only as a supporting signal |\n| Content fidelity | omissions, repetitions, numbers and names | ASR round-trip for a given text; not a universal metric for open-ended replies |\n| Turn-taking | premature/missed endpoints, interruption precision/recall, overlap quality | annotated two-channel conversations |\n| Long sessions | retention of facts and unfinished actions after compaction | multi-turn tests at full length |\n| Safety | unauthorized tool actions, replay, prompt injection, voice impersonation | red-team audio and background commands |\n| Cost | gross and per-successful-session $/min, tokens/session, infrastructure utilization | real usage logs plus a load test |\n\nA few rules make the comparison honest:\n\n**Key idea:** \"the most native\" and \"the fastest in the\ndemo\" are not product metrics. The system that wins is the one with the\nbest task success at acceptable latency, risk and price on your\ntraffic.\n\nLikely directions for 2026–2027:\n\nThat's a forecast, not a guaranteed route. And cascades won't disappear: observability and independently replaceable components are durable advantages.\n\nA reliable mental model of V2V looks like this:\n\nIf you understand transformers and LLMs, most of the machinery here will feel familiar. The new difficulty lives in multi-stream audio representations, endpointing, decoding, acoustic evaluation and continuous interaction.\n\n**Primary sources:**\n\n📚 **Current documentation:**", "url": "https://wpnews.pro/news/how-voice-to-voice-models-work-from-sound-wave-to-living-conversation", "canonical_source": "https://asaptf.github.io/v2v-models-article-en/", "published_at": "2026-07-18 15:30:02+00:00", "updated_at": "2026-07-18 15:51:14.482563+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-products", "ai-tools"], "entities": ["Newo.ai", "Swift", "GitHub", "Python", "PyTorch"], "alternates": {"html": "https://wpnews.pro/news/how-voice-to-voice-models-work-from-sound-wave-to-living-conversation", "markdown": "https://wpnews.pro/news/how-voice-to-voice-models-work-from-sound-wave-to-living-conversation.md", "text": "https://wpnews.pro/news/how-voice-to-voice-models-work-from-sound-wave-to-living-conversation.txt", "jsonld": "https://wpnews.pro/news/how-voice-to-voice-models-work-from-sound-wave-to-living-conversation.jsonld"}}