{"slug": "sophon-realtime-v1-open-full-duplex-voice-with-a-32b-brain-and-vllm-omni", "title": "Sophon-Realtime v1: Open Full-Duplex Voice With a 32B Brain and vLLM-Omni", "summary": "Adventists AI released Sophon-Realtime v1, an open full-duplex voice stack published on Hugging Face under Apache-2.0 weights and code, built on a shared 32B model and served via vLLM-Omni. The model card reports VoiceBench 82.8, equal to GPT-4o-mini-Audio and below GPT-4o-Audio's 86.8, and above open full-duplex entries MiniCPM-o 4.5 (76.1), Freeze-Omni (55.2) and Moshi (29.5), plus 81.8% on 1,000 Big Bench Audio spoken reasoning questions and 48% Pass@1 on Full-Duplex-Bench v3 tool calls. Adventists ships three builds — Tag (~0.6 s first audio, 16 users per 80 GB GPU), Flash (0.31 s, 40 users) and E2E (~350M talker, ~0.25 s, 16 users) — with an optional OpenAI-compatible GPT-Live gateway on port 8780.", "body_md": "**Sophon-Realtime v1** is a new **open full-duplex** voice stack from **Adventists AI**, published on [Hugging Face](https://huggingface.co/adventists-ai/Sophon-Realtime-v1?ref=explainx) with Apache-2.0 weights and a [GitHub serving repo](https://github.com/adventists-ai/sophon-realtime?ref=explainx). Unlike a speech-to-text → LLM → TTS pipeline, it **listens while you talk**, **barges in when you interrupt**, and **marks laughs, sighs, and stress** in the text before a speech model performs them.\n\nFor builders comparing 2026 voice agents, the release matters because it bundles **benchmark tables**, **one-GPU economics**, and a **GPT-Live-shaped API** in one package. explainx.ai recently covered [Haru-4B](https://explainx.ai/blog/haru-4b-open-full-duplex-speech-model-apache-qwen3-5-higurashi-2026) without public evals and [Shizuku-Duplex](https://explainx.ai/blog/shizuku-duplex-japanese-full-duplex-voice-80ms-interaction-background-2026) with Japanese-focused metrics. Sophon is the first open family in this wave to claim **VoiceBench 82.8** beside MiniCPM, Moshi, and closed GPT-4o-audio baselines on the same card.\n\n## TL;DR\n\n| Question | Answer | \n|---|---|\n| License | **Apache-2.0** weights and code (commercial use allowed on card) | \n| Brain | Shared **32B** model for listening, turn-taking, and reply drafting | \n| Builds | **Tag** (expressive NVV),**Flash** (density),**E2E** (fastest first audio) | \n| VoiceBench | **82.8** overall on card table (open + full duplex) | \n| Big Bench Audio | **81.8%** spoken reasoning accuracy (self-run, per card) | \n| Tool calls | **48% Pass@1** on Full-Duplex-Bench v3,**100%** response rate when tools needed | \n| Serving | **vLLM-Omni** for Tag/Flash;**16–40 users / 80 GB GPU** depending on build | \n| GPT-Live | Optional **`live/` gateway** on port 8780 with OpenAI-compatible WebSocket | \n| Try online | ModelScope Studio link on the Hugging Face page | \n\n## Three builds, one brain\n\nAdventists ships **one cognitive model** and three **speech renderers**. The [model card table](https://huggingface.co/adventists-ai/Sophon-Realtime-v1?ref=explainx) is the quickest map:\n\n| Build | How speech is made | Best when | First audio (one user) | Users / 80 GB GPU | \n|---|---|---|---|---|\n| **Tag** (default) | Reply text includes **sound tags** ; expressive speech model performs them | Companions, character demos, rich NVV | ~0.6 s | 16 | \n| **Flash** | Same tags, **lighter speech model** | Contact center scale, in-car | **0.31 s** | **40** | \n| **E2E** | **~350M talker** emits continuous voice vectors, no speech tokens | Fastest time-to-first-byte with NVV | ~0.25 s | 16 | \n\n**NVV-SuperBench** (non-verbal vocalizations) scores on the card: Tag hits **0.86 / 0.80** (zh / en F1), Flash **0.77 / 0.63**, E2E **0.85 / 0.75**. Tag and Flash share APIs so you can swap renderers without rewriting clients; E2E can run on its own engine or, since v1.1.4, on vLLM-Omni with the same surface.\n\nSophon-Realtime v1 open full-duplex voice model benchmark table on the model card\n*Image: [Adventists AI](https://huggingface.co/adventists-ai/Sophon-Realtime-v1?ref=explainx)*\n\n## Benchmarks: what the card actually claims\n\nAll numbers below come from the **November 2026 model card** unless noted. Treat vendor-run evals as directional until you reproduce them on your hardware and audio set.\n\n### Conversation quality (VoiceBench)\n\nThe card places **Sophon-Realtime v1 at 82.8**, equal to **GPT-4o-mini-Audio** and below **GPT-4o-Audio at 86.8**. Among **open full-duplex** entries in the same table, it ranks above **MiniCPM-o 4.5 (76.1)**, **Freeze-Omni (55.2)**, and **Moshi (29.5)**.\n\n### Spoken reasoning (Big Bench Audio)\n\nOn **1,000 spoken reasoning questions**, the card lists **81.8% accuracy** for Sophon with open weights, versus **58%** for Raon SpeechChat and **4%** for Moshi. Sophon transcripts answers and grades with **gpt-4.1**, per the footnote.\n\n### Tool use under disfluent speech (Full-Duplex-Bench v3)\n\nHere is the builder-relevant row:\n\n| Model | Pass@1 | Responds when it should | \n|---|---|---|\n| GPT-Realtime (closed) | 60% | 96% | \n| Gemini Live 3.1 | 54% | — | \n| **Sophon-Realtime** | **48%** | **100%** | \n| Grok | 43% | — | \n\nSophon does **not** beat the closed frontier on raw Pass@1, but the card stresses it **always responds** when a tool call is appropriate—a failure mode that breaks real agents.\n\n### Hearing tone (paralinguistic)\n\nListening uses **DuplexJev**, Adventists’ open speech-decision model. Combined **LISTEN + StressTest + MSPB** score: **48.6** for DuplexJev, ahead of **Gemini 3.8 Flash (46.9)** and **Qwen3-Omni (44.6)** on the card.\n\nTwo speech bubbles passing a turn-taking bead, illustrating full-duplex voice agent listening while speaking\n\n## How the stack works (mechanism, not marketing)\n\nThe card’s architecture section reads like a checklist of what [Fish Audio Live 3](https://explainx.ai/blog/fish-audio-live-3-full-duplex-voice-agents-november-2026) and [OpenAI GPT-Live](https://explainx.ai/blog/gpt-live-1-api-openai-voice-agents-september-2026) buyers already ask for:\n\n1. **Continuous listening** — at each pause the model emits a**single token** deciding whether you finished, continued, back-channeled, or paused to think.\n2. **Early drafting** — it starts a reply at your first pause and only releases audio when it takes the floor; if you keep talking, the draft is discarded silently.\n3. **Tagged paralinguistics** — the brain inserts**laughter, sigh, breath, cough, emphasis** markers; renderers perform them instead of reading tags aloud.\n4. **Coupled text and audio streams** — transcripts stay available for logging, tools, and downstream LLMs.\n5. **Interrupt-safe memory** — history keeps audio both sides heard; partial assistant audio is trimmed on barge-in.\n\nThe **E2E** build replaces discrete speech tokens with **continuous voice vectors** from a **~350M** talker—a design closer to research speech codecs than classic TTS chains.\n\n## Quick start: Tag on one GPU\n\nThe fastest path on the card:\n\n```\npip install \"sophon-realtime[trt]\"\nsophon-realtime serve\n```\n\nRequirements called out in the README path:\n\n- **Python 3.12** ,**Linux** ,**CUDA 13**\n- **`[trt]` extra required** — without TensorRT the project warns the voice path is about**3× slower under load**\n- First run downloads **~42 GB** Tag weights, then listens on**port 8000**\n\nGitHub scripts clone **`adventists-ai/sophon-realtime`**, run `tag/install.sh`, then `GPU=0 PORT=8000 bash start.sh`. Cold start can take **15–20 minutes** the first time, **5–8 minutes** later.\n\n**WebSocket API:** `ws://HOST:8000/v1/realtime` for duplex sessions; **REST:** `POST /v1/chat/completions` with audio in and text+audio out. Example client: `python client_example.py --wav my_question.wav --lang en --voice ryan_en`.\n\n### GPT-Live gateway\n\nFor teams already running OpenAI-shaped clients:\n\n```\ncd sophon-realtime/live && BUILD=tag PORT=8000 LIVE_PORT=8780 bash start_live.sh\n```\n\nClients point at `ws://HOST:8780/v1/live/sessions`. Sophon keeps voice; when a request needs lookup, the gateway can say a short filler, call your **backend LLM**, and speak results. The card claims pairing with **DeepSeek V4 Pro** lifts Big Bench Audio from **81.2 to ~83** in their configuration—verify on your stack before quoting to customers.\n\n## Fine-tuning without retraining the voice\n\nAdventists documents **LoRA fine-tuning** on the thinker backbone via **ms-swift**, keeping audio encoder and voice frozen:\n\n```\npip install \"ms-swift>=4.6\"\ngit clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/finetune\nmodelscope download --model AdventistsAI/Sophon-Realtime-v1 --include 'base/*' 'adapters/*' --local_dir Sophon-Realtime-v1\n```\n\nTraining continues the bundled **tag LoRA** so laughter and cough tags stay aligned. One **80 GB GPU** is enough per the finetune README.\n\nOpen-weight model pieces distributed to trays, echoing Apache-2.0 Sophon builds builders can self-host\n\n## Comparisons: where Sophon fits in your stack\n\n| Need | Sophon angle | explainx.ai alternate | \n|---|---|---|\n| Smallest open duplex experiment | **Haru-4B** on ~12 GB, no public benchmarks | [Haru-4B guide](https://explainx.ai/blog/haru-4b-open-full-duplex-speech-model-apache-qwen3-5-higurashi-2026) | \n| Japanese companion, 80 ms ticks | **Shizuku-Duplex** | [Shizuku post](https://explainx.ai/blog/shizuku-duplex-japanese-full-duplex-voice-80ms-interaction-background-2026) | \n| Telephony latency SLOs | Commercial **Tell-1** | [Retell Tell-1](https://explainx.ai/blog/retell-tell-1-voice-model-contact-center-latency-2026) | \n| Closed GPT-Live SLA | OpenAI hosted | [GPT-Live API](https://explainx.ai/blog/gpt-live-1-api-openai-voice-agents-september-2026) | \n| Duplex + fish-scale agent demos | Fish stack | [Fish Live 3](https://explainx.ai/blog/fish-audio-live-3-full-duplex-voice-agents-november-2026) | \n\n**When to pick Sophon:** you want **Apache-2.0**, **self-hosted** duplex on **one H100-class GPU**, with **published** VoiceBench and tool-call numbers and a **GPT-Live migration path**.\n\n**When to wait:** you need **sub-100 ms** Japanese ticks (Shizuku’s pitch), **Mac-first MLX** (Haru’s path), or you cannot run **CUDA 13 + TensorRT** in production.\n\n## What people are asking\n\n**Are the benchmarks independent?** VoiceBench and Full-Duplex-Bench rows mix **official leaderboards** with **self-run** Sophon jobs (Big Bench Audio, some NVV runs). Budget time to replicate on your audio before shipping SLAs.\n\n**Why three builds instead of one checkpoint?** Speech quality and **GPU multiplexing** trade off. Flash sacrifices some NVV F1 in English (**0.63** vs **0.80** on Tag) to hit **40 users** per GPU.\n\n**Is DuplexJev separate weights?** The card markets DuplexJev as the **listening** module inside Sophon; fine-tune docs focus on the **thinker LLM** only.\n\n**ModelScope vs Hugging Face?** Deploy weights are linked per build page; finetune downloads use **ModelScope** paths in the script while marketing lives on HF.\n\n**HN traction?** A [Hacker News thread](https://news.ycombinator.com/item?id=50434577&ref=explainx) exists but had **no substantive comments** when we checked—hands-on reports may come later.\n\n## Cost and capacity planning\n\nCapacity math on the card is blunt: one **80 GB** GPU hosts **16** Tag users or **40** Flash users at recommended concurrency. If your contact center assumes **P99 first audio under 500 ms**, Flash’s **0.31 s** cited first-audio figure is the build to model in load tests, not Tag’s **~0.6 s** expressive path.\n\nDownload and storage also belong in finance models:\n\n- **~42 GB** Tag weights on first`sophon-realtime serve`\n- **TensorRT** path effectively mandatory for production multiplexing per README warnings\n- **Finetune artifacts** via ModelScope`base/*` and`adapters/*` bundles\n\nPair Sophon with the same observability you would run on [GPT-Live API pilots](https://explainx.ai/blog/gpt-live-1-api-openai-voice-agents-september-2026): log **barge-in rate**, **tool-call success**, and **truncated assistant audio** after interrupts. The card’s **100% response rate** on Full-Duplex-Bench v3 matters only if your gateway preserves those semantics under real network jitter.\n\n### Security and abuse notes for self-hosted voice\n\nSelf-hosted duplex endpoints are attractive **prompt-injection and social-engineering surfaces** when wired to customer CRMs. Treat the **`live/` GPT-Live gateway** like any other externally reachable WebSocket: mutual TLS, per-session API keys, rate limits, and separate **tool sandboxes** for backend LLMs. explainx.ai’s [MCP explainer](https://explainx.ai/blog/what-is-mcp-model-context-protocol-guide) is a useful template for how tool handoffs should stay scoped when duplex models call backends mid-call.\n\n## Limitations and licensing footnotes\n\n- **Hardware gate:** Without**`sophon-realtime[trt]`** , expect much slower speech under concurrent load.\n- **Weight size:** Tag download**~42 GB** ; plan artifact mirrors and checksum policy like any frontier open model.\n- **Third-party licenses:** The card’s**NOTICE** file keeps component-level terms; read before redistributing bundles.\n- **Business contact:**`jiejin@adventists.ai` on the model page for commercial enquiries.\n- **Paper link:** The Hugging Face page references arXiv**2609.21967** (NemotronLabs VoiceChat lineage) for readers who want architecture context beyond the marketing tables.\n\n## DuplexJev and the listening stack\n\nSophon’s **listening** path is not a generic Whisper front-end. The card brands **DuplexJev** as a speech-decision model that feeds the 32B thinker. That split explains why **paralinguistic understanding** scores are reported separately from **VoiceBench** conversation quality: one measures **ears**, the other **dialogue**.\n\nIf you are benchmarking against [Qwen3-Omni](https://explainx.ai/blog/qwen-3-8-omni-flash-launch-2026) or Gemini Live, reproduce **LISTEN**, **StressTest**, and **MSPB** tasks on your audio before claiming superiority—vendor F1 tables used **Gemini 3 Flash** as judge on NVV clips.\n\n## Related reading\n\n*Benchmark figures and install commands reflect the Adventists AI model card and GitHub README as accessed November 15, 2026. Re-run VoiceBench and latency tests on your own GPUs before production commitments.*", "url": "https://wpnews.pro/news/sophon-realtime-v1-open-full-duplex-voice-with-a-32b-brain-and-vllm-omni", "canonical_source": "https://explainx.ai/blog/sophon-realtime-v1-open-full-duplex-voice-vllm-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog", "published_at": "2026-11-15 00:00:00+00:00", "updated_at": "2026-11-15 10:52:27.251587+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-infrastructure", "ai-tools"], "entities": ["Adventists AI", "Sophon-Realtime v1", "Hugging Face", "vLLM-Omni", "MiniCPM-o 4.5", "Moshi", "GPT-4o-Audio", "Full-Duplex-Bench v3"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/sophon-realtime-v1-open-full-duplex-voice-with-a-32b-brain-and-vllm-omni", "markdown": "https://wpnews.pro/news/sophon-realtime-v1-open-full-duplex-voice-with-a-32b-brain-and-vllm-omni.md", "text": "https://wpnews.pro/news/sophon-realtime-v1-open-full-duplex-voice-with-a-32b-brain-and-vllm-omni.txt", "jsonld": "https://wpnews.pro/news/sophon-realtime-v1-open-full-duplex-voice-with-a-32b-brain-and-vllm-omni.jsonld"}}