# Tracking Vietnamese news trends with embeddings, HDBSCAN and an LLM

> Source: <https://dev.to/mikevu/tracking-vietnamese-news-trends-with-embeddings-hdbscan-and-an-llm-h>
> Published: 2026-11-05 20:47:28+00:00

A big event is usually covered by dozens of outlets at once, under different headlines. Reading article by article, it's hard to see the overall picture. The more useful question isn't "what's in the news today", but: **which stories are heating up, which are cooling off, and why?**

To answer that, I built [Tin tức hôm nay có gì?](https://news.vuminhhoa.com/) ("What's in the news today?"), a system that reads about 95 Vietnamese RSS feeds, groups articles about the same thing into *topics*, tracks each topic's lifecycle, and has an LLM summarize what is driving it. This post covers how it works and the numbers I picked along the way.

It's a single TypeScript service running on [Bun](https://bun.sh), with its data in one SQLite file (`bun:sqlite`). The pipeline runs in passes, and each pass goes through these stages:

The same image runs as `web` (dashboard and API), `pipeline`, or both. In production the two run separately, so the dashboard isn't affected while the pipeline is busy.

News doesn't arrive evenly through the day, so the pipeline doesn't run evenly either. The schedule is computed in Vietnam time:

| Time of day | Every | 
|---|---|
| 6–9 am and 5–10 pm | 15 minutes | 
| 9 am–5 pm | 30 minutes | 
| Overnight | 60 minutes | 

Passes are aligned to round times (7:00, 7:15, 7:30…), so the dashboard can always say when the next one will run.

The same article often shows up in several feeds of one outlet (front page, section, latest news). I hash the URL into a `content_hash` and skip anything already in the database. It's simple and good enough, since one article nearly always keeps one URL. Two *different* outlets covering the same event are not duplicates, though: that's exactly the signal clustering needs.

By default I use `jina-embeddings-v3` through Jina AI's API; it's multilingual and handles Vietnamese well. Without an API key the service falls back to running **LaBSE** locally (via `@xenova/transformers`). It's slower, but the pipeline doesn't depend on an outside service to keep working.

Vectors are stored with the article, and each pass only embeds new articles, up to 1,000 at a time.

With news, I don't know in advance how many topics there are, and most articles are **noise**: one-off stories that don't belong to anything bigger. K-means makes you pick the number of clusters and forces every point into one. HDBSCAN finds the number of clusters on its own and labels points that don't fit anywhere as `-1`. For this problem, that's exactly the behavior I want.

It runs in a small Python process (the `hdbscan` library); Bun pipes the vectors in on stdin and reads the labels from stdout. The current settings:

```
HDBSCAN(
    min_cluster_size=8,
    min_samples=2,
    metric="euclidean",
    cluster_selection_method="leaf",
)
```

A few of these choices are worth explaining:

`euclidean` ranks neighbors exactly as cosine would, and HDBSCAN is faster with it.`leaf` rather than `eom`.` min_cluster_size=8`.
HDBSCAN only knows about the articles in the current pass. For a topic to *live* for hours, each topic stores a **centroid** (the mean of its vectors), and every new cluster is compared against the centroids of active topics:

| Rule | Cosine threshold | 
|---|---|
| New cluster joins an existing topic | ≥ 0.75 | 
| Merge two clusters in the same pass | ≥ 0.82 | 
| Merge two topics across passes | ≥ 0.78 | 
| Split a topic that got too big | from 30 articles | 

The merge threshold is higher than the join threshold on purpose: a wrong merge is hard to undo, while leaving two clusters apart for one more pass costs almost nothing. After each merge the centroid is recomputed right away, which stops cascades (A absorbs B, then AB absorbs C only because it grew).

Each topic moves through a simple state machine driven by **velocity** (how fast new articles arrive, measured over a 2-hour window) and **acceleration** (how velocity is changing):

*Only Signal through Peak count as active, which is also what limits the LLM calls.*

Every transition is recorded with its reason (for example "Count >= 10 and velocity > 1.0"), so when a topic jumps around I can read its history and see why.

For ranking, each topic gets a score from 0 to 100, the sum of six parts normalized to 0–1:

| Part | Weight | Maxes out at | 
|---|---|---|
| Velocity (last 2 hours) | 25 | 10 articles | 
| Volume (24 hours) | 20 | 150 articles | 
| Acceleration | 20 | 5 | 
| Distinct sources | 15 | 8 sources | 
| Velocity relative to the average | 10 | 3× | 
| Topic freshness | 10 | decays to 0 over 7 days | 

Distinct sources get their own weight because 20 articles from one outlet are very different from 8 outlets covering the same story. The score is then multiplied by a decay factor based on the time since the latest article: no decay in the first hour, then 20% after 1 hour, 40% after 6, 60% after 12, 80% after 24, and zero after 48 hours.

The LLM is called through an OpenAI-compatible API (DeepSeek by default), for two kinds of requests:

`driving_force` (the main cause), `trend_rationale` (why it's heating up or calming down), `opportunity` (a concrete opportunity or risk, and for which industry), `watch_next` (what to watch for) and an `opportunity_score` from 0 to 10.
LLM calls cost money, so choosing which topics to analyze matters as much as the prompt:

Token usage is recorded per pass to keep an eye on cost. Setting `LLM_ENABLED=false` keeps the whole pipeline running, just without names and analysis.

Day to day, the service needs little attention:

There is also room for improvement:

The live version is at [news.vuminhhoa.com](https://news.vuminhhoa.com/). If you're working on something with Vietnamese news too, I'd like to hear about it: my email is on the [home page](https://vuminhhoa.com/en#contact).