{"slug": "cresist-self-enhancing-explorative-policy-learning-for-simultaneous-speech", "title": "CRESiST: Self-Enhancing Explorative Policy Learning for Simultaneous Speech Translation", "summary": "Researchers posted arXiv paper 2611.06501v1 introducing CRESiST, an end-to-end interleaved read-write framework for simultaneous speech translation built on an omni-modal large language model. CRESiST interleaves speech chunks and text tokens to achieve full KV cache reuse and segmentation-free, infinite-length inference without external Voice Activity Detection, and adds an Actor-Critic self-enhancement pipeline in which an RNN-T Actor samples read-write trajectories and an LLM-based Critic scores them on intrinsic semantic and latency rewards. The authors report new state-of-the-art quality-latency Pareto frontiers on sentence-level benchmarks FLEURS and CoVoST2 and document-level benchmarks MuST-C and ACL 60/60, outperforming cascaded and static-alignment baselines.", "body_md": "arXiv:2611.06501v1 Announce Type: new \nAbstract: Simultaneous Speech Translation (SimulST) requires balancing translation quality and latency. While Large Language Models (LLMs) excel in offline translation, current LLM-based SimulST systems predominantly rely on prefix-incremental generation. This paradigm suffers from redundant Key-Value cache recomputation and necessitates external Voice Activity Detection to process unbounded streams. To overcome these bottlenecks, we propose an end-to-end interleaved read-write framework built upon an omni-modal LLM. By interleaving speech chunks and text tokens, our architecture achieves full KV cache reuse, natively supporting segmentation-free, infinite-length inference. Furthermore, to learn optimal streaming policies without rigid offline alignments, we introduce a novel Actor-Critic self-enhancement pipeline. An RNN-T Actor dynamically samples read-write trajectories, while an LLM-based Critic evaluates these paths via intrinsic semantic and latency rewards, distilling the optimal policy back to the Actor. Extensive experiments on sentence-level (FLEURS, CoVoST2) and document-level (MuST-C, ACL 60/60) benchmarks demonstrate that our model establishes new state-of-the-art quality-latency Pareto frontiers, substantially outperforming existing cascaded and static-alignment baselines.", "url": "https://wpnews.pro/news/cresist-self-enhancing-explorative-policy-learning-for-simultaneous-speech", "canonical_source": "https://www.machinebrief.com/news/cresist-self-enhancing-explorative-policy-learning-for-simul-0owe", "published_at": "2026-11-05 05:00:00+00:00", "updated_at": "2026-11-05 06:21:23.985273+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "natural-language-processing", "ai-research"], "entities": ["CRESiST", "arXiv", "FLEURS", "CoVoST2", "MuST-C", "ACL 60/60"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/cresist-self-enhancing-explorative-policy-learning-for-simultaneous-speech", "markdown": "https://wpnews.pro/news/cresist-self-enhancing-explorative-policy-learning-for-simultaneous-speech.md", "text": "https://wpnews.pro/news/cresist-self-enhancing-explorative-policy-learning-for-simultaneous-speech.txt", "jsonld": "https://wpnews.pro/news/cresist-self-enhancing-explorative-policy-learning-for-simultaneous-speech.jsonld"}}