Confucius4-R2T2 is Netease Youdao's answer to a pain point every voice-agent builder hits: streaming ASR that rewrites its own output. The Hugging Face paper page and arXiv:2611.06043 describe Real Real-Time Transcription — a model that never modifies earlier tokens while keeping accuracy close to a full-context pass on Qwen3-ASR-1.7B.
Weights and code are public: huggingface.co/netease-youdao/Confucius4-R2T2 and github.com/netease-youdao/Confucius4-R2T2. A live demo lets you hear the behavior.
TL;DR #
| Item | Detail |
|---|---|
| Base | **Qwen3-ASR-1.7B** (encoder + projector frozen) |
| Training data | ~1,500 h Chinese + English audio (per paper) | | Idea | Longest Stable Prefix (LSP) supervision | | Output | Append-only text; no retroactive edits | | Release | HF weights + GitHub 871★ when listed on HF paper page |
| Demo | r2t2.youdao.com/demo |
Why rewriting transcripts breaks agents #
Confucius4-R2T2 stable-prefix streaming ASR keeps earlier transcript tokens fixed
Humans tolerate caption backtracking. Machines do not.
If a streaming model emits "book a table at eight" and two seconds later revises "eight" to "ate", a voice agent may have already:
- Called a reservations API with the wrong time.
- Started translation that downstream TTS already spoke.
- Logged a compliance transcript that no longer matches audio.
The paper's opening argument is that voice agents and simultaneous translators consume tokens as they arrive. Unstable prefixes create wasted compute or wrong actions. That is why stable-prefix ASR belongs in the same planning doc as speech-to-speech agent guides and duplex voice research like Shizuku.
Tencent's HY translation line already cites R2T2-style metrics in our HY-SimulMT coverage; this release is the open ASR half of that ecosystem pressure.
What Longest Stable Prefix means #
Authors introduce LSP as a formal label: given partial audio, take the longest prefix of the eventual full transcript that will not change when more audio is revealed. The paper claims three properties:
- Monotone under the prefix order — as audio grows, the stable prefix only extends.
- Consistent with the full-audio transcript at the end.
- Maximal among labels that equal some earlier whole output (no unnecessary conservatism).
Training on LSP targets lets you start from a strong offline ASR and post-train a streaming variant without architectural surgery — important for teams that already ship Qwen-family encoders.
## Training recipe on Qwen3-ASR-1.7B
Per the abstract on [arXiv](https://arxiv.org/abs/2611.06043?ref=explainx):
- Start from **Qwen3-ASR-1.7B** full-context model.
- Freeze the pretrained audio encoder and projector.
- Post-train on 1,500 hours ofChinese and English speech with LSP labels.
- Evaluate emission lag (how soon text appears) against full-context quality.
The Hugging Face paper page lists 27 upvotes and author notes promising cross-lingual generalization beyond the post-training languages — treat cross-lingual numbers as paper claims until you replicate on your locale.
Voice agents need streaming ASR that does not revise words already emitted
Results the authors report #
We anchor on the paper abstract, not third-party leaderboard screenshots:
- Streaming quality approaches the base model's full-context performance.
- Low emission lag relative to stable output — the "real-time" in the name.
- Configurable streaming chunk sizes for latency tradeoffs (details in repo configs).
Compare mentally to Microsoft VibeVoice's streaming ASR variant in our VibeVoice guide — different org, same buyer question: Can I act on partial text safely?
How to try R2T2 this week #
Browser demo. Open r2t2.youdao.com/demo and watch whether earlier lines stay fixed while new words append.
Weights. Pull netease-youdao/Confucius4-R2T2 from Hugging Face and follow the GitHub README for inference scripts.
Integration sketch for agents.
- Pipe microphone audio in streaming chunks.
- Commit only the LSP-stable segment to your agent state machine.
- Keep a separate buffer for hypothesis text if you want UI preview — butdo not call tools on hypothesis text.
4. Pair with a [semantic cache](https://explainx.ai/blog/semantic-caching-llm-apps-cost-latency-guide-2026) if repeated partial triggers would hammer your LLM.
For desktop meeting capture without coding, see trending app [Cicada](https://explainx.ai/blog/cicada-open-source-live-subtitles-meeting-notes-interpreting-2026) — complementary product layer, different stability guarantees.
Conventional streaming vs stable-prefix: a builder table #
| Behavior | Typical streaming ASR | Confucius4-R2T2 (claimed) |
|---|---|---|
| Partial text | Often rewrites earlier words | Append-only committed prefix |
| Agent tool calls | Risky on partials | Safer if you gate on LSP commits |
| UI captions | Humans tolerate backtrack | Machines should not rely on color flicker |
| Training | Chunked CTC/RNN teachers | LSP post-training on Qwen3-ASR |
| Architecture change | Sometimes new heads/decoders | None required per paper |
The table summarizes author claims — run your own WER and tool-call error tests on your audio domain before production.
Paper details worth reading in arXiv:2611.06043 #
Beyond the abstract, builders should skim:
- Proof intuition for LSP — monotone extension as audio grows, so your state machine can treat commits as immutable events.
- Frozen encoder choice — keeping Qwen3-ASR audio front-end fixed reduces regression risk when you already deployed that stack for offline captions.
- Chunk size knobs — latency vs stability trade space; voice agents with aggressive endpointing need sweeps here.
- Cross-lingual generalization — promising for multilingual contact centers, but verify on your locales (accents, code-switching).
Authors link project pages at r2t2.ai and mirror weights on Hugging Face — treat GitHub stars on the paper card as social proof, not quality certification.
Wiring R2T2 into a speech-to-speech stack #
A minimal production pattern:
- Microphone → VAD — do not confuse LSP with endpoint detection.
- R2T2 stream — maintain
committed_textandpending_tailseparately in your session object. - LLM — pass only
committed_textdiffs to the model; renderpending_tailin UI as grey italics if you show previews. - TTS — start speaking only after a punctuation or silence rule you control, not on every LSP extension.
- Tool router — requiretwo consecutive identical commitments before high-risk tools if you lack a separate confidence score.
Our A2A protocol guide matters when the agent receiving text is remote — serialize committed prefixes as events, not full transcript snapshots, to save bandwidth.
Evaluation gaps you should close locally #
Open-source release does not ship your compliance pack. Before shipping voice agents:
- Measure word error rate against your call center audio.
- Log tool errors triggered by pre-final text with and without LSP gating.
- Stress barge-in : users interrupting TTS while ASR still extends prefixes.
- Test accent and domain jargon absent from the 1,500-hour post-training mix.
Compare against Microsoft VibeVoice streaming checkpoints on the same clips — different licenses, different serving stacks.
Who should adopt vs wait #
Adopt early if:
- You ship tool-using voice agents (calendar, CRM, smart home).
- You run simultaneous interpretation with incremental MT.
- You fine-tune Qwen3-ASR already and want streaming without a second architecture.
Wait if:
- Your stack only sends final utterances to the LLM today.
- You need languages outside the paper's post-training set without your own eval.
- You require vendor SLAs — this is open research code, not a hosted API.
Engineering limitations to plan for #
The paper does not replace endpointing (knowing when the user stopped). LSP stability is about not revising committed text, not about perfect segmentation.
You still need VAD, barge-in, and permission UX — topics that show up in multimodal vision token budgeting when agents combine mic and camera.
Noise, codec artifacts, and code-switching can shrink the stable prefix — budget human review for high-risk tool calls.
What Hugging Face discussion highlights #
The paper discussion emphasizes downstream consumers — translators and agents — rather than human caption comfort. Commenters upvoted the append-only property specifically for machine consumers, aligning with Youdao's positioning against revising partials. Treat discussion praise as community signal, not independent benchmarking.
When you test the demo, record whether punctuation commits align with your agent's tool grammar. Some teams require a stable prefix plus a sentence-final period before CRM writes.
Licensing and deployment notes #
Weights ship under the Qwen / model card license on Hugging Face — read netease-youdao/Confucius4-R2T2 before redistributing in a SaaS endpoint. Self-hosted inference likely pairs with existing Qwen3-ASR serving guides; if you already run vLLM vs SGLang for LLMs, budget separate GPU memory for streaming audio batches.
Youdao is not a US hyperscaler — procurement teams should run the same vendor review they apply to Tencent HY models: export jurisdiction, logging clauses, and support SLAs.
Security note for voice agents #
Stable text reduces accidental tool calls from revised words, not malicious ones. Prompt injection through spoken audio remains. If agents can place calls or move money, use approval gates — AgentBeam is one layer that blocks dangerous tool execution in coding agents; the same discipline applies to voice-triggered workflows.
Bottom line #
Confucius4-R2T2 is a practical open release: formal LSP training, Qwen3-ASR backbone, public weights, and a demo that makes the stability property visible. For explainx.ai readers building voice agents, it is the clearest 2026 statement that streaming ASR should be append-only if downstream automation is live.
We will update when the authors publish per-language benchmark tables or hosted API pricing on Youdao's side.
Related reading #
Model and paper details follow arXiv:2611.06043 and the Hugging Face model card as of November 10, 2026.