{"slug": "a-comparative-study-of-transformer-based-embeddings-for-topic-coherence", "title": "A comparative study of transformer-based embeddings for topic coherence", "summary": "A new study systematically examined the effect of transformer-based language model size on topic quality, finding that model size had a negligible impact on coherence and divergence metrics. Researchers tested seven models ranging from 22 million to 13 billion parameters in a BERTopic pipeline across multiple corpora. The findings suggest that smaller models can achieve comparable topic quality to larger models, challenging assumptions about the necessity of massive language models for topic modeling tasks.", "body_md": "arXiv:2605.28832v1 Announce Type: new\nAbstract: Topic modeling is a branch of Natural Language Processing (NLP) that aims to organize large collections of texts into coherent groups according to word co-occurrence patterns, with Latent Dirichlet Allocation (LDA) remaining one of the most widely used and interpretable probabilistic approaches. Recent advances in NLP, particularly transformer-based language models, offer improved document representations. It is also known that the size of the model (in terms of number of parameters) has a significant impact in the performance of the language models on different pre-defined tasks. In this study, we systematically examine the effect of model size on topic quality by analyzing the performances of seven transformer-based language models (from small models such as MiniLM to large ones such as LLaMA-2) in a BERTopic pipeline on a variety of corpora. Topic quality is evaluated using coherence and divergence metrics following R{\\\"o}der et al. (2015). Our results indicate that model size, ranging from 22 million to 13 billion parameters, has a negligible impact on the quality of the topic, suggesting that smaller models can achieve comparable performance to larger models.", "url": "https://wpnews.pro/news/a-comparative-study-of-transformer-based-embeddings-for-topic-coherence", "canonical_source": "https://arxiv.org/abs/2605.28832", "published_at": "2026-05-29 04:00:00+00:00", "updated_at": "2026-05-29 04:24:51.445915+00:00", "lang": "en", "topics": ["natural-language-processing", "machine-learning", "large-language-models", "artificial-intelligence", "ai-research"], "entities": ["Latent Dirichlet Allocation", "MiniLM", "LLaMA-2", "BERTopic", "Röder"], "alternates": {"html": "https://wpnews.pro/news/a-comparative-study-of-transformer-based-embeddings-for-topic-coherence", "markdown": "https://wpnews.pro/news/a-comparative-study-of-transformer-based-embeddings-for-topic-coherence.md", "text": "https://wpnews.pro/news/a-comparative-study-of-transformer-based-embeddings-for-topic-coherence.txt", "jsonld": "https://wpnews.pro/news/a-comparative-study-of-transformer-based-embeddings-for-topic-coherence.jsonld"}}