Sophon-Realtime v1 is a new open full-duplex voice stack from Adventists AI, published on Hugging Face with Apache-2.0 weights and a GitHub serving repo. Unlike a speech-to-text → LLM → TTS pipeline, it listens while you talk, barges in when you interrupt, and marks laughs, sighs, and stress in the text before a speech model performs them.
For builders comparing 2026 voice agents, the release matters because it bundles benchmark tables, one-GPU economics, and a GPT-Live-shaped API in one package. explainx.ai recently covered Haru-4B without public evals and Shizuku-Duplex with Japanese-focused metrics. Sophon is the first open family in this wave to claim VoiceBench 82.8 beside MiniCPM, Moshi, and closed GPT-4o-audio baselines on the same card.
TL;DR #
| Question | Answer |
|---|---|
| License | Apache-2.0 weights and code (commercial use allowed on card) |
| Brain | Shared 32B model for listening, turn-taking, and reply drafting |
| Builds | Tag (expressive NVV),Flash (density),E2E (fastest first audio) |
| VoiceBench | 82.8 overall on card table (open + full duplex) |
| Big Bench Audio | 81.8% spoken reasoning accuracy (self-run, per card) |
| Tool calls | 48% Pass@1 on Full-Duplex-Bench v3,100% response rate when tools needed |
| Serving | vLLM-Omni for Tag/Flash;16–40 users / 80 GB GPU depending on build |
| GPT-Live | Optional live/ gateway on port 8780 with OpenAI-compatible WebSocket |
| Try online | ModelScope Studio link on the Hugging Face page |
Three builds, one brain #
Adventists ships one cognitive model and three speech renderers. The model card table is the quickest map:
| Build | How speech is made | Best when | First audio (one user) | Users / 80 GB GPU |
|---|---|---|---|---|
| Tag (default) | Reply text includes sound tags ; expressive speech model performs them | Companions, character demos, rich NVV | ~0.6 s | 16 |
| Flash | Same tags, lighter speech model | Contact center scale, in-car | 0.31 s | 40 |
| E2E | ~350M talker emits continuous voice vectors, no speech tokens | Fastest time-to-first-byte with NVV | ~0.25 s | 16 |
NVV-SuperBench (non-verbal vocalizations) scores on the card: Tag hits 0.86 / 0.80 (zh / en F1), Flash 0.77 / 0.63, E2E 0.85 / 0.75. Tag and Flash share APIs so you can swap renderers without rewriting clients; E2E can run on its own engine or, since v1.1.4, on vLLM-Omni with the same surface.
Sophon-Realtime v1 open full-duplex voice model benchmark table on the model card Image: Adventists AI
Benchmarks: what the card actually claims #
All numbers below come from the November 2026 model card unless noted. Treat vendor-run evals as directional until you reproduce them on your hardware and audio set.
Conversation quality (VoiceBench)
The card places Sophon-Realtime v1 at 82.8, equal to GPT-4o-mini-Audio and below GPT-4o-Audio at 86.8. Among open full-duplex entries in the same table, it ranks above MiniCPM-o 4.5 (76.1), Freeze-Omni (55.2), and Moshi (29.5).
Spoken reasoning (Big Bench Audio)
On 1,000 spoken reasoning questions, the card lists 81.8% accuracy for Sophon with open weights, versus 58% for Raon SpeechChat and 4% for Moshi. Sophon transcripts answers and grades with gpt-4.1, per the footnote.
Tool use under disfluent speech (Full-Duplex-Bench v3)
Here is the builder-relevant row:
| Model | Pass@1 | Responds when it should |
|---|---|---|
| GPT-Realtime (closed) | 60% | 96% |
| Gemini Live 3.1 | 54% | — |
| Sophon-Realtime | 48% | 100% |
| Grok | 43% | — |
Sophon does not beat the closed frontier on raw Pass@1, but the card stresses it always responds when a tool call is appropriate—a failure mode that breaks real agents.
Hearing tone (paralinguistic)
Listening uses DuplexJev, Adventists’ open speech-decision model. Combined LISTEN + StressTest + MSPB score: 48.6 for DuplexJev, ahead of Gemini 3.8 Flash (46.9) and Qwen3-Omni (44.6) on the card.
Two speech bubbles passing a turn-taking bead, illustrating full-duplex voice agent listening while speaking
How the stack works (mechanism, not marketing) #
The card’s architecture section reads like a checklist of what Fish Audio Live 3 and OpenAI GPT-Live buyers already ask for:
- Continuous listening — at each the model emits asingle token deciding whether you finished, continued, back-channeled, or d to think.
- Early drafting — it starts a reply at your first and only releases audio when it takes the floor; if you keep talking, the draft is discarded silently.
- Tagged paralinguistics — the brain insertslaughter, sigh, breath, cough, emphasis markers; renderers perform them instead of reading tags aloud.
- Coupled text and audio streams — transcripts stay available for logging, tools, and downstream LLMs.
- Interrupt-safe memory — history keeps audio both sides heard; partial assistant audio is trimmed on barge-in.
The E2E build replaces discrete speech tokens with continuous voice vectors from a ~350M talker—a design closer to research speech codecs than classic TTS chains.
Quick start: Tag on one GPU #
The fastest path on the card:
pip install "sophon-realtime[trt]"
sophon-realtime serve
Requirements called out in the README path:
- Python 3.12 ,Linux ,CUDA 13
[trt]extra required — without TensorRT the project warns the voice path is about3× slower under load- First run downloads ~42 GB Tag weights, then listens onport 8000
GitHub scripts clone adventists-ai/sophon-realtime, run tag/install.sh, then GPU=0 PORT=8000 bash start.sh. Cold start can take 15–20 minutes the first time, 5–8 minutes later.
WebSocket API: ws://HOST:8000/v1/realtime for duplex sessions; REST: POST /v1/chat/completions with audio in and text+audio out. Example client: python client_example.py --wav my_question.wav --lang en --voice ryan_en.
GPT-Live gateway
For teams already running OpenAI-shaped clients:
cd sophon-realtime/live && BUILD=tag PORT=8000 LIVE_PORT=8780 bash start_live.sh
Clients point at ws://HOST:8780/v1/live/sessions. Sophon keeps voice; when a request needs lookup, the gateway can say a short filler, call your backend LLM, and speak results. The card claims pairing with DeepSeek V4 Pro lifts Big Bench Audio from 81.2 to ~83 in their configuration—verify on your stack before quoting to customers.
Fine-tuning without retraining the voice #
Adventists documents LoRA fine-tuning on the thinker backbone via ms-swift, keeping audio encoder and voice frozen:
pip install "ms-swift>=4.6"
git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/finetune
modelscope download --model AdventistsAI/Sophon-Realtime-v1 --include 'base/*' 'adapters/*' --local_dir Sophon-Realtime-v1
Training continues the bundled tag LoRA so laughter and cough tags stay aligned. One 80 GB GPU is enough per the finetune README.
Open-weight model pieces distributed to trays, echoing Apache-2.0 Sophon builds builders can self-host
Comparisons: where Sophon fits in your stack #
| Need | Sophon angle | explainx.ai alternate |
|---|---|---|
| Smallest open duplex experiment | Haru-4B on ~12 GB, no public benchmarks | Haru-4B guide |
| Japanese companion, 80 ms ticks | Shizuku-Duplex | Shizuku post |
| Telephony latency SLOs | Commercial Tell-1 | Retell Tell-1 |
| Closed GPT-Live SLA | OpenAI hosted | GPT-Live API |
| Duplex + fish-scale agent demos | Fish stack | Fish Live 3 |
When to pick Sophon: you want Apache-2.0, self-hosted duplex on one H100-class GPU, with published VoiceBench and tool-call numbers and a GPT-Live migration path.
When to wait: you need sub-100 ms Japanese ticks (Shizuku’s pitch), Mac-first MLX (Haru’s path), or you cannot run CUDA 13 + TensorRT in production.
What people are asking #
Are the benchmarks independent? VoiceBench and Full-Duplex-Bench rows mix official leaderboards with self-run Sophon jobs (Big Bench Audio, some NVV runs). Budget time to replicate on your audio before shipping SLAs.
Why three builds instead of one checkpoint? Speech quality and GPU multiplexing trade off. Flash sacrifices some NVV F1 in English (0.63 vs 0.80 on Tag) to hit 40 users per GPU.
Is DuplexJev separate weights? The card markets DuplexJev as the listening module inside Sophon; fine-tune docs focus on the thinker LLM only.
ModelScope vs Hugging Face? Deploy weights are linked per build page; finetune downloads use ModelScope paths in the script while marketing lives on HF.
HN traction? A Hacker News thread exists but had no substantive comments when we checked—hands-on reports may come later.
Cost and capacity planning #
Capacity math on the card is blunt: one 80 GB GPU hosts 16 Tag users or 40 Flash users at recommended concurrency. If your contact center assumes P99 first audio under 500 ms, Flash’s 0.31 s cited first-audio figure is the build to model in load tests, not Tag’s ~0.6 s expressive path.
Download and storage also belong in finance models:
- ~42 GB Tag weights on first
sophon-realtime serve - TensorRT path effectively mandatory for production multiplexing per README warnings
- Finetune artifacts via ModelScope
base/*andadapters/*bundles
Pair Sophon with the same observability you would run on GPT-Live API pilots: log barge-in rate, tool-call success, and truncated assistant audio after interrupts. The card’s 100% response rate on Full-Duplex-Bench v3 matters only if your gateway preserves those semantics under real network jitter.
Security and abuse notes for self-hosted voice
Self-hosted duplex endpoints are attractive prompt-injection and social-engineering surfaces when wired to customer CRMs. Treat the live/ GPT-Live gateway like any other externally reachable WebSocket: mutual TLS, per-session API keys, rate limits, and separate tool sandboxes for backend LLMs. explainx.ai’s MCP explainer is a useful template for how tool handoffs should stay scoped when duplex models call backends mid-call.
Limitations and licensing footnotes #
- Hardware gate: Without**
sophon-realtime[trt]** , expect much slower speech under concurrent load. - Weight size: Tag download**~42 GB** ; plan artifact mirrors and checksum policy like any frontier open model.
- Third-party licenses: The card’sNOTICE file keeps component-level terms; read before redistributing bundles.
- Business contact:
jiejin@adventists.aion the model page for commercial enquiries. - Paper link: The Hugging Face page references arXiv2609.21967 (NemotronLabs VoiceChat lineage) for readers who want architecture context beyond the marketing tables.
DuplexJev and the listening stack #
Sophon’s listening path is not a generic Whisper front-end. The card brands DuplexJev as a speech-decision model that feeds the 32B thinker. That split explains why paralinguistic understanding scores are reported separately from VoiceBench conversation quality: one measures ears, the other dialogue.
If you are benchmarking against Qwen3-Omni or Gemini Live, reproduce LISTEN, StressTest, and MSPB tasks on your audio before claiming superiority—vendor F1 tables used Gemini 3 Flash as judge on NVV clips.
Related reading #
Benchmark figures and install commands reflect the Adventists AI model card and GitHub README as accessed November 15, 2026. Re-run VoiceBench and latency tests on your own GPUs before production commitments.