CRESiST: Self-Enhancing Explorative Policy Learning for Simultaneous Speech Translation

Researchers posted arXiv paper 2611.06501v1 introducing CRESiST, an end-to-end interleaved read-write framework for simultaneous speech translation built on an omni-modal large language model. CRESiST interleaves speech chunks and text tokens to achieve full KV cache reuse and segmentation-free, infinite-length inference without external Voice Activity Detection, and adds an Actor-Critic self-enhancement pipeline in which an RNN-T Actor samples read-write trajectories and an LLM-based Critic scores them on intrinsic semantic and latency rewards. The authors report new state-of-the-art quality-latency Pareto frontiers on sentence-level benchmarks FLEURS and CoVoST2 and document-level benchmarks MuST-C and ACL 60/60, outperforming cascaded and static-alignment baselines.

arXiv:2611.06501v1 Announce Type: new Abstract: Simultaneous Speech Translation SimulST requires balancing translation quality and latency. While Large Language Models LLMs excel in offline translation, current LLM-based SimulST systems predominantly rely on prefix-incremental generation. This paradigm suffers from redundant Key-Value cache recomputation and necessitates external Voice Activity Detection to process unbounded streams. To overcome these bottlenecks, we propose an end-to-end interleaved read-write framework built upon an omni-modal LLM. By interleaving speech chunks and text tokens, our architecture achieves full KV cache reuse, natively supporting segmentation-free, infinite-length inference. Furthermore, to learn optimal streaming policies without rigid offline alignments, we introduce a novel Actor-Critic self-enhancement pipeline. An RNN-T Actor dynamically samples read-write trajectories, while an LLM-based Critic evaluates these paths via intrinsic semantic and latency rewards, distilling the optimal policy back to the Actor. Extensive experiments on sentence-level FLEURS, CoVoST2 and document-level MuST-C, ACL 60/60 benchmarks demonstrate that our model establishes new state-of-the-art quality-latency Pareto frontiers, substantially outperforming existing cascaded and static-alignment baselines.