Class-Structure Preservation Beats Diversity: A Comprehensive Benchmark of Text Augmentation Methods for Imbalanced Text Classification

A new benchmark of 11 text augmentation methods on seven public datasets finds that all LLM-based methods are statistically equivalent or inferior to the classical embedding-space SMOTE-style retrieval (EmbSMOTE), with the gap widening as imbalance increases and reaching a macro F1 difference of about 0.063 on GoEmotions-28. The study, posted on arXiv (2608.12340v1), concludes that class-conditional structural fidelity, not surface-level diversity, drives performance and recommends retrieval-based oversampling as the default for imbalanced multi-class classification.

arXiv:2608.12340v1 Announce Type: new Abstract: With the rapid advancement of large language models LLMs , generative data augmentation has attracted considerable attention for imbalanced text classification in natural language processing. However, no empirical benchmark to date has compared LLM-based augmentation against the embedding-space SMOTE-style retrieval EmbSMOTE , a strong classical reference for imbalanced classification. In this study, a controlled benchmark of 11 augmentation methods, spanning classical perturbation, embedding-space retrieval, and LLM-based generation, is newly constructed on seven public text classification datasets covering class counts $K=2$-$28$ and imbalance ratios of 1.1 to over 500, evaluated with five random seeds per cell using macro F1, Welch's $t$-tests, five distributional metrics, and an LLM-family sensitivity analysis based on Qwen3-8B. The experimental results reveal that all LLM-based methods are statistically equivalent or inferior to EmbSMOTE, with the performance gap widening monotonically as imbalance increases and reaching $\Delta\text{F1} \text{macro}\ \approx\ 0.063$ on GoEmotions-28. Furthermore, it is observed that surface-level uniqueness has negligible correlation with downstream performance, whereas LLM-specific artifacts, such as text elongation and label-distribution uniformization, are negatively associated with classification accuracy. Compared with six LLM-based and four classical augmentation baselines, these results demonstrate that the effective variable is not surface-level diversity but class-conditional structural fidelity, namely the degree to which augmented samples preserve the class-conditioned geometry of the training distribution. Accordingly, retrieval-based oversampling should be adopted as the default for imbalanced multi-class classification, and a higher empirical bar should be required before LLM-based augmentation is deployed in practice.