{"slug": "tracking-vietnamese-news-trends-with-embeddings-hdbscan-and-an-llm", "title": "Tracking Vietnamese news trends with embeddings, HDBSCAN and an LLM", "summary": "A developer built \"Tin tức hôm nay có gì?\", a TypeScript/Bun service that ingests roughly 95 Vietnamese RSS feeds, embeds articles with jina-embeddings-v3 (falling back to local LaBSE), and clusters them with HDBSCAN to group coverage of the same event into tracked topics. Topics persist across pipeline passes via centroids with cosine thresholds of 0.75 for joining, 0.82 for same-pass merges and 0.78 for cross-pass merges, and each topic's lifecycle is tracked by velocity and acceleration over a 2-hour window, with an LLM summarizing what drives it.", "body_md": "A big event is usually covered by dozens of outlets at once, under different headlines. Reading article by article, it's hard to see the overall picture. The more useful question isn't \"what's in the news today\", but: **which stories are heating up, which are cooling off, and why?**\n\nTo answer that, I built [Tin tức hôm nay có gì?](https://news.vuminhhoa.com/) (\"What's in the news today?\"), a system that reads about 95 Vietnamese RSS feeds, groups articles about the same thing into *topics*, tracks each topic's lifecycle, and has an LLM summarize what is driving it. This post covers how it works and the numbers I picked along the way.\n\nIt's a single TypeScript service running on [Bun](https://bun.sh), with its data in one SQLite file (`bun:sqlite`). The pipeline runs in passes, and each pass goes through these stages:\n\nThe same image runs as `web` (dashboard and API), `pipeline`, or both. In production the two run separately, so the dashboard isn't affected while the pipeline is busy.\n\nNews doesn't arrive evenly through the day, so the pipeline doesn't run evenly either. The schedule is computed in Vietnam time:\n\n| Time of day | Every | \n|---|---|\n| 6–9 am and 5–10 pm | 15 minutes | \n| 9 am–5 pm | 30 minutes | \n| Overnight | 60 minutes | \n\nPasses are aligned to round times (7:00, 7:15, 7:30…), so the dashboard can always say when the next one will run.\n\nThe same article often shows up in several feeds of one outlet (front page, section, latest news). I hash the URL into a `content_hash` and skip anything already in the database. It's simple and good enough, since one article nearly always keeps one URL. Two *different* outlets covering the same event are not duplicates, though: that's exactly the signal clustering needs.\n\nBy default I use `jina-embeddings-v3` through Jina AI's API; it's multilingual and handles Vietnamese well. Without an API key the service falls back to running **LaBSE** locally (via `@xenova/transformers`). It's slower, but the pipeline doesn't depend on an outside service to keep working.\n\nVectors are stored with the article, and each pass only embeds new articles, up to 1,000 at a time.\n\nWith news, I don't know in advance how many topics there are, and most articles are **noise**: one-off stories that don't belong to anything bigger. K-means makes you pick the number of clusters and forces every point into one. HDBSCAN finds the number of clusters on its own and labels points that don't fit anywhere as `-1`. For this problem, that's exactly the behavior I want.\n\nIt runs in a small Python process (the `hdbscan` library); Bun pipes the vectors in on stdin and reads the labels from stdout. The current settings:\n\n```\nHDBSCAN(\n    min_cluster_size=8,\n    min_samples=2,\n    metric=\"euclidean\",\n    cluster_selection_method=\"leaf\",\n)\n```\n\nA few of these choices are worth explaining:\n\n`euclidean` ranks neighbors exactly as cosine would, and HDBSCAN is faster with it.`leaf` rather than `eom`.` min_cluster_size=8`.\nHDBSCAN only knows about the articles in the current pass. For a topic to *live* for hours, each topic stores a **centroid** (the mean of its vectors), and every new cluster is compared against the centroids of active topics:\n\n| Rule | Cosine threshold | \n|---|---|\n| New cluster joins an existing topic | ≥ 0.75 | \n| Merge two clusters in the same pass | ≥ 0.82 | \n| Merge two topics across passes | ≥ 0.78 | \n| Split a topic that got too big | from 30 articles | \n\nThe merge threshold is higher than the join threshold on purpose: a wrong merge is hard to undo, while leaving two clusters apart for one more pass costs almost nothing. After each merge the centroid is recomputed right away, which stops cascades (A absorbs B, then AB absorbs C only because it grew).\n\nEach topic moves through a simple state machine driven by **velocity** (how fast new articles arrive, measured over a 2-hour window) and **acceleration** (how velocity is changing):\n\n*Only Signal through Peak count as active, which is also what limits the LLM calls.*\n\nEvery transition is recorded with its reason (for example \"Count >= 10 and velocity > 1.0\"), so when a topic jumps around I can read its history and see why.\n\nFor ranking, each topic gets a score from 0 to 100, the sum of six parts normalized to 0–1:\n\n| Part | Weight | Maxes out at | \n|---|---|---|\n| Velocity (last 2 hours) | 25 | 10 articles | \n| Volume (24 hours) | 20 | 150 articles | \n| Acceleration | 20 | 5 | \n| Distinct sources | 15 | 8 sources | \n| Velocity relative to the average | 10 | 3× | \n| Topic freshness | 10 | decays to 0 over 7 days | \n\nDistinct sources get their own weight because 20 articles from one outlet are very different from 8 outlets covering the same story. The score is then multiplied by a decay factor based on the time since the latest article: no decay in the first hour, then 20% after 1 hour, 40% after 6, 60% after 12, 80% after 24, and zero after 48 hours.\n\nThe LLM is called through an OpenAI-compatible API (DeepSeek by default), for two kinds of requests:\n\n`driving_force` (the main cause), `trend_rationale` (why it's heating up or calming down), `opportunity` (a concrete opportunity or risk, and for which industry), `watch_next` (what to watch for) and an `opportunity_score` from 0 to 10.\nLLM calls cost money, so choosing which topics to analyze matters as much as the prompt:\n\nToken usage is recorded per pass to keep an eye on cost. Setting `LLM_ENABLED=false` keeps the whole pipeline running, just without names and analysis.\n\nDay to day, the service needs little attention:\n\nThere is also room for improvement:\n\nThe live version is at [news.vuminhhoa.com](https://news.vuminhhoa.com/). If you're working on something with Vietnamese news too, I'd like to hear about it: my email is on the [home page](https://vuminhhoa.com/en#contact).", "url": "https://wpnews.pro/news/tracking-vietnamese-news-trends-with-embeddings-hdbscan-and-an-llm", "canonical_source": "https://dev.to/mikevu/tracking-vietnamese-news-trends-with-embeddings-hdbscan-and-an-llm-h", "published_at": "2026-11-05 20:47:28+00:00", "updated_at": "2026-11-05 20:49:23.443380+00:00", "lang": "en", "topics": ["natural-language-processing", "machine-learning", "ai-tools", "large-language-models"], "entities": ["Tin tức hôm nay có gì?", "Bun", "SQLite", "Jina AI", "jina-embeddings-v3", "LaBSE", "HDBSCAN"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/tracking-vietnamese-news-trends-with-embeddings-hdbscan-and-an-llm", "markdown": "https://wpnews.pro/news/tracking-vietnamese-news-trends-with-embeddings-hdbscan-and-an-llm.md", "text": "https://wpnews.pro/news/tracking-vietnamese-news-trends-with-embeddings-hdbscan-and-an-llm.txt", "jsonld": "https://wpnews.pro/news/tracking-vietnamese-news-trends-with-embeddings-hdbscan-and-an-llm.jsonld"}}