# emb-diversity: A Tool for Embedding-Based Measurement of Data Diversity

> Source: <https://www.machinebrief.com/news/emb-diversity-a-tool-for-embedding-based-measurement-of-data-ib97>
> Published: 2026-07-23 04:00:00+00:00

arXiv:2607.19848v1 Announce Type: new
Abstract: There is growing evidence that data diversity is crucial for developing fair and robust NLP models. However, current approaches to measure diversity remain inconsistent and fragmented: While there exist a number of tools for measuring the lexical diversity of texts, researchers lack standardized tools for quantifying diversity based on embeddings. Embedding-based diversity measures are highly flexible: They work with any embedding model and any data that can be embedded, and are thus applicable to many notions of diversity. With emb-diversity, we provide a comprehensive embedding-based diversity measurement tool, spanning a broad range of measures. We demonstrate its potential for several use cases: measuring the stylistic, semantic, language and speaker diversity of datasets. https://github.com/nlpsoc/emb-diversity/