Clustering Unstructured Text with LLM Embeddings and HDBSCAN

A new tutorial demonstrates how to build a text clustering pipeline using large language model embeddings and the HDBSCAN algorithm to automatically discover topics in unlabeled text data. The pipeline leverages sentence-transformers for embedding generation, UMAP for dimensionality reduction, and HDBSCAN for clustering, enabling unsupervised topic discovery without prior labeling.

In this article, you will learn how to build a text clustering pipeline by combining large language model embeddings with HDBSCAN, a density-based clustering algorithm, to automatically discover topics in unlabeled text data. Topics we will cover include: - How to generate text embeddings for raw documents using a pre-trained sentence-transformers model. - How to reduce the dimensionality of those embeddings with UMAP to prepare them for clustering. - How to apply HDBSCAN to automatically discover topic clusters and visualize the results. Introduction The current era of Generative AI seems to primarily focus on chat interfaces and prompts, but the range of applications of large language models , or LLMs for short, is not limited to just that. Indeed, one of their most powerful downstream abilities consists of turning raw, messy, unstructured text into semantically rich mathematical representations called embeddings . Once that’s done, we can use these text representations for a variety of machine learning use cases, with clustering being no exception. In particular, embeddings can be combined with advanced, density-based clustering techniques like HDBSCAN , allowing as a result for the discovery of hidden topics, patterns, or categories in your collection of text documents: all without the need for prior labeling. This article shows how to construct a text-based clustering pipeline from scratch. We will use a freely available dataset containing text instances, as well as an open-source LLM that has been trained for generating embeddings — i.e. a so-called embedding model. The icing on the cake: we’ll use free and handy, modern Python libraries providing implementations of clustering algorithms like HDBSCAN. Step-by-Step Walkthrough First, let’s start by installing the key Python libraries we will need: Sentence transformers , to load a pre-trained LLM for embedding generation from Hugging Face — you’ll need a Hugging Face API key, also called an access token https://huggingface.co/docs/hub/security-tokens , to be able to load the model. Umap-learn , to apply an algorithm to reduce the dimensionality of embeddings. Likewise, if you are working on a local IDE instead of a cloud notebook environment and don’t have scikit-learn and pandas , you may need to install them too. pip install sentence-transformers umap-learn 1 pip install sentence-transformers umap-learn Now we start the coding part by getting some fresh data. The fetch 20newsgroups function, which fetches a dataset containing texts from categorized news articles, will do. Note that even though the dataset contains labels, we will omit them, as we are pretending not to know this information for the sake of clustering these data instances into groups based on similarity. Also, we sample down the dataset to 150 instances, which will be representative enough for our example. python import pandas as pd from sklearn.datasets import fetch 20newsgroups Fetching a highly targeted subset of data ~150-200 docs categories = 'sci.space', 'sci.med', 'rec.autos' newsgroups = fetch 20newsgroups subset='train', categories=categories, remove= 'headers', 'footers', 'quotes' Sampling down into a representative, illustrative subset df = pd.DataFrame {'text': newsgroups.data, 'true label': newsgroups.target} df = df df 'text' .str.strip .str.len 100 .sample 150, random state=42 .reset index drop=True print f"Loaded {len df } text documents." print "\nSample document:" print df 'text' .iloc 0 :150 + "..." 1234567891011121314 import pandas as pdfrom sklearn.datasets import fetch 20newsgroups Fetching a highly targeted subset of data ~150-200 docs categories = 'sci.space', 'sci.med', 'rec.autos' newsgroups = fetch 20newsgroups subset='train', categories=categories, remove= 'headers', 'footers', 'quotes' Sampling down into a representative, illustrative subsetdf = pd.DataFrame {'text': newsgroups.data, 'true label': newsgroups.target} df = df df 'text' .str.strip .str.len 100 .sample 150, random state=42 .reset index drop=True print f"Loaded {len df } text documents." print "\nSample document:" print df 'text' .iloc 0 :150 + "..." Output: Loaded 150 text documents. Sample document: Okay Mr. Dyer, we're properly impressed with your philosophical skills and ability to insult people. You're a wonderful speaker and an adept politic... 123456 Loaded 150 text documents. Sample document: Okay Mr. Dyer, we're properly impressed with your philosophical skills andability to insult people. You're a wonderful speaker and an adept politic... The next step is to obtain the embeddings from raw texts. To do this, we load all-MiniLM-L6-v2 from Hugging Face’s sentence-transformers library. This is a lightweight yet effective model to obtain embeddings quickly. python from sentence transformers import SentenceTransformer Loading the free, open-source model model = SentenceTransformer 'all-MiniLM-L6-v2' Encoding text documents into dense vector embeddings print "Generating embeddings..." embeddings = model.encode df 'text' .tolist , show progress bar=True print f"Embedding matrix shape: {embeddings.shape}" 12345678910 from sentence transformers import SentenceTransformer Loading the free, open-source modelmodel = SentenceTransformer 'all-MiniLM-L6-v2' Encoding text documents into dense vector embeddingsprint "Generating embeddings..." embeddings = model.encode df 'text' .tolist , show progress bar=True print f"Embedding matrix shape: {embeddings.shape}" Since the embedding dimension is originally too high for clustering purposes, we now apply a dimensionality reduction technique by using the UMAP algorithm from the namesake library installed earlier: python import umap Reducing embedding dimensions to 5, to retain enough density information for clustering reducer = umap.UMAP n neighbors=15, n components=5, min dist=0.0, random state=42 reduced embeddings = reducer.fit transform embeddings print f"Reduced matrix shape: {reduced embeddings.shape}" 1234567 import umap Reducing embedding dimensions to 5, to retain enough density information for clusteringreducer = umap.UMAP n neighbors=15, n components=5, min dist=0.0, random state=42 reduced embeddings = reducer.fit transform embeddings print f"Reduced matrix shape: {reduced embeddings.shape}" Now our numerical embedding vectors associated with news articles consist of five dimensions attributes only. Let’s see if this compact representation is meaningful enough to obtain insightful clustering by applying the HDBSCAN algorithm, which is a density-based clustering approach: python from sklearn.cluster import HDBSCAN Initializing HDBSCAN min cluster size=8: we specified that each cluster must have at least 8 documents clusterer = HDBSCAN min cluster size=8, min samples=3, store centers='centroid' df 'cluster' = clusterer.fit predict reduced embeddings Counting instances per cluster cluster counts = df 'cluster' .value counts print "\nCluster Distribution:" print cluster counts 1234567891011 from sklearn.cluster import HDBSCAN Initializing HDBSCAN min cluster size=8: we specified that each cluster must have at least 8 documentsclusterer = HDBSCAN min cluster size=8, min samples=3, store centers='centroid' df 'cluster' = clusterer.fit predict reduced embeddings Counting instances per clustercluster counts = df 'cluster' .value counts print "\nCluster Distribution:" print cluster counts Important : the clustering results are partly influenced by the hyperparameter settings we defined for HDBSCAN. I recommend you try out other configurations for the minimum cluster size and other hyperparameters to explore how this affects results. Result: Cluster Distribution: cluster 0 101 1 49 Name: count, dtype: int64 12345 Cluster Distribution:cluster0 1011 49Name: count, dtype: int64 It looks like HDBSCAN detected two clusters associated with high-density regions in the data space. Would there also be noisy points that were not allocated to either of these two clusters? Let’s check: for cluster id in sorted df 'cluster' .unique : if cluster id == -1: print "\n=== CLUSTER: NOISE / UNCLASSIFIED ===" else: print f"\n=== CLUSTER: Discovered Topic {cluster id} ===" Getting up to 3 sample texts from this cluster samples = df df 'cluster' == cluster id 'text' .head 3 .tolist for i, sample in enumerate samples, 1 : clean sample = " ".join sample.split :120 print f" {i}. {clean sample}..." 1234567891011 for cluster id in sorted df 'cluster' .unique : if cluster id == -1: print "\n=== CLUSTER: NOISE / UNCLASSIFIED ===" else: print f"\n=== CLUSTER: Discovered Topic {cluster id} ===" Getting up to 3 sample texts from this cluster samples = df df 'cluster' == cluster id 'text' .head 3 .tolist for i, sample in enumerate samples, 1 : clean sample = " ".join sample.split :120 print f" {i}. {clean sample}..." Output: === CLUSTER: Discovered Topic 0 === 1. Okay Mr. Dyer, we're properly impressed with your philosophical skills and ability to insult people. You're a wonderful ... 2. I was at an interesting seminar at work UK's R.A.L. Space Science Dept. on this subject, specifically on a small-scale... 3. This is the second post which seems to be blurring the distinction between real disease caused by Candida albicans and t... === CLUSTER: Discovered Topic 1 === 1. It's great that all these other cars can out-handle, out-corner, and out- accelerate an Integra. But, you've got to ask ... 2. l diamond star cars Talon/Eclipse/Laser put out 190 hp in the turbo models, and 195 hp in the AWD turbo models, These ... 3. Sorry for the mis-spelling, but I forgot how to spell it after my series of exams and NO-on hand reference here. Is it s... 123456789 === CLUSTER: Discovered Topic 0 === 1. Okay Mr. Dyer, we're properly impressed with your philosophical skills and ability to insult people. You're a wonderful ... 2. I was at an interesting seminar at work UK's R.A.L. Space Science Dept. on this subject, specifically on a small-scale... 3. This is the second post which seems to be blurring the distinction between real disease caused by Candida albicans and t... === CLUSTER: Discovered Topic 1 === 1. It's great that all these other cars can out-handle, out-corner, and out- accelerate an Integra. But, you've got to ask ... 2. l diamond star cars Talon/Eclipse/Laser put out 190 hp in the turbo models, and 195 hp in the AWD turbo models, These ... 3. Sorry for the mis-spelling, but I forgot how to spell it after my series of exams and NO-on hand reference here. Is it s... Seems like all data points in the sample of 150 were allocated to either one of the two clusters identified, thus hinting at the clue that the news articles might easily separable according to topic. For extra insight, we can show some cluster visualizations with the aid of the supplementary code provided below, which shows a scatterplot for every pairwise combination of the five existing components that describe each data point: python import matplotlib.pyplot as plt import seaborn as sns import itertools Creating a DataFrame for the 5 reduced embeddings and cluster labels reduced df = pd.DataFrame reduced embeddings, columns= f'UMAP D{i+1}' for i in range reduced embeddings.shape 1 reduced df 'cluster' = df 'cluster' Getting all unique pairwise combinations of the 5 dimensions dim pairs = list itertools.combinations reduced df.columns :-1 , 2 num plots = len dim pairs num cols = 3 num rows = num plots + num cols - 1 // num cols plt.figure figsize= num cols 5, num rows 4 for i, dim1, dim2 in enumerate dim pairs : plt.subplot num rows, num cols, i + 1 sns.scatterplot x=dim1, y=dim2, hue='cluster', data=reduced df, palette='viridis', s=70, alpha=0.7, legend='full' plt.title f'{dim1} vs {dim2}' plt.xlabel dim1 plt.ylabel dim2 plt.grid True, linestyle='--', alpha=0.6 plt.tight layout plt.show 123456789101112131415161718192021222324252627282930313233343536 import matplotlib.pyplot as pltimport seaborn as snsimport itertools Creating a DataFrame for the 5 reduced embeddings and cluster labelsreduced df = pd.DataFrame reduced embeddings, columns= f'UMAP D{i+1}' for i in range reduced embeddings.shape 1 reduced df 'cluster' = df 'cluster' Getting all unique pairwise combinations of the 5 dimensionsdim pairs = list itertools.combinations reduced df.columns :-1 , 2 num plots = len dim pairs num cols = 3num rows = num plots + num cols - 1 // num cols plt.figure figsize= num cols 5, num rows 4 for i, dim1, dim2 in enumerate dim pairs : plt.subplot num rows, num cols, i + 1 sns.scatterplot x=dim1, y=dim2, hue='cluster', data=reduced df, palette='viridis', s=70, alpha=0.7, legend='full' plt.title f'{dim1} vs {dim2}' plt.xlabel dim1 plt.ylabel dim2 plt.grid True, linestyle='--', alpha=0.6 plt.tight layout plt.show Result: By trying different configurations for HDBSCAN, you may come across results in which the number of identified clusters could be different from two. Just give it a try Wrapping Up Once we have gone through the process of building the text-based clustering pipeline, it is worth concluding by pointing out the key reasons why putting together LLM embeddings with HDBSCAN is worth it. These include the ability to retain and capture, to some extent, the true semantic meaning and linguistic nuances of the original text, thanks to the properties inherent to embeddings obtained through sentence-transformers. Moreover, HDBSCAN automatically determines an optimal number of clusters and is able to detect outlying points that might be noise or outliers that would distort group-level statistics.