PhoPunct: A Large-Scale Benchmark of Transformer Architectures for Vietnamese Punctuation Restoration

A controlled benchmark on 22.1 million labeled tokens (19.9M News, 2.2M Novels) found that adding Bi-LSTM and CRF layers to pretrained Transformer backbones does not reliably improve Vietnamese punctuation restoration, according to the arXiv paper 2611.09107v1 from the aisia-research-lab. Across 17 architectures trained for eight epochs at a fixed learning rate without architecture-specific tuning, a plain Transformer classifier matched or outperformed its Bi-LSTM-CRF counterpart in seven of eight matched domain-backbone comparisons. The PhoBERT-large-Bi-LSTM-CRF model PhoPunct posted the best News macro F1 at 61.35, while XLM-R with a CRF led Novels at 50.18, above PhoPunct's 48.97, leading the authors to call for separating architectural gains from gains due to architecture-specific tuning.

arXiv:2611.09107v1 Announce Type: new Abstract: Vietnamese punctuation restoration is often evaluated on relatively small benchmarks, limiting conclusions about how architectural choices scale with data volume. We present a controlled benchmark on News and Novels corpora containing 22.1 million labeled tokens 19.9M News and 2.2M Novels . We evaluate 17 architectures formed from four pretrained Transformer backbones and optional bidirectional long short-term memory Bi-LSTM and conditional random field CRF layers, together with a PhoBERT-large-Bi-LSTM-CRF model, PhoPunct. Every model is trained for eight epochs with the same learning rate and without architecture-specific tuning. Under this fixed budget, adding Bi-LSTM and CRF layers does not reliably improve macro F1: in seven of eight matched domain-backbone comparisons, a plain Transformer classifier matches or outperforms its Bi-LSTM-CRF counterpart. PhoPunct obtains the best News result 61.35 , whereas XLM-R with a CRF obtains the best Novels result 50.18 , exceeding PhoPunct 48.97 . These results show that the value of recurrent and structured-prediction heads depends on their optimization regime and motivate explicit separation between architectural improvements and improvements due to architecture-specific tuning. Our source code is available at: https://github.com/aisia-research-lab/PhoPunct