{"slug": "the-llm-overruled-the-math-and-it-was-right", "title": "The LLM Overruled the Math and It Was Right", "summary": "A two-layer trend-detection system called TrendsWorks, built by Data Elixir newsletter curator Ben Lorica, overruled its own math layer on its first run and kept \"AI Systems Evaluation & Operational Cost\" flagged as Emerging for nearly four months and 17 weekly runs, after the math layer had labeled the cluster \"fading\" with a z-score of −0.18 and 0.85× growth versus its recent average. The system pairs a stateless math layer (embeddings, clustering and week-over-week counts labeling clusters new, emerging, accelerating, fading or steady) with a stateful LLM analyst that reads the numbers plus a markdown memory file and records every disagreement with a reason. The first weekly report covered 225 clusters drawn from 12,243 stories for October 30 – November 5, 2026, and the design deliberately lets the two layers contradict each other, with neither getting the final word.", "body_md": "**This week’s report:** [Trends radar: October 30 – November 5, 2026](../../radar/2026-10-30/) — 225 clusters from 12,243 stories, and 5 topics where the LLM analyst disagreed with the numbers.\n\nI curate the [Data Elixir](https://dataelixir.com/) newsletter and I read a lot about tech, especially data and AI. To help keep up, I built a system called TrendsWorks that tells me what topics are *moving* across weeks. It helps me identify topics that are new, topics that are gaining traction, and topics that are quietly fading away.\n\nIdentifying trends that span weeks or months is a different problem than identifying topics that are trending *today*. Over time, topics evolve. They split, merge, go quiet, and then come back. The same idea gets new vocabulary, and the same word starts meaning something different. Counting keywords can’t follow that.\n\nSo TrendsWorks has two brains. The first is math: embeddings, clustering, and week-over-week counts that decide what’s emerging. The second is an LLM analyst with a memory. It reads those numbers, remembers what it said last month, and tells the story. I deliberately kept them separate, and I designed them to disagree.\n\n## The Math Said “Fading”\n\nOn the very first run, the math had embedded about a thousand recent articles, grouped them into hundreds of clusters, and stamped a status on the 8 that qualified.\n\nOne of those clusters was a set of articles about evaluating AI systems. The math called it “fading” with a z-score of −0.18 and growth of 0.85× compared to its recent average. That’s a pretty clear verdict. The topic had a soft week and was on its way out.\n\nBut then the LLM analyst read the same numbers and overruled it:\n\nThe quantitative layer calls this cluster “fading” based on a z-score of –0.18, but the underlying articles span Gradient Flow and Eugene Yan and cover three distinct sub-arguments: evaluation is the real bottleneck (not model quality), AI operational bills are rising even as token prices fall, and agents need to be judged on task completion rather than capability demos. That is a coherent, durable storyline rather than a single viral piece decaying. Flagging as **Emerging** in the ledger rather than fading…\n\nThat wasn’t a bug, and it wasn’t the LLM going off the rails either. The system is *designed* so the two halves can contradict each other. Every disagreement is recorded with a reason, and the weekly report has a section just for exploring the disagreements. Neither layer gets the final word.\n\nNearly four months and 17 weekly runs later, “AI Systems Evaluation & Operational Cost” is still on the analyst’s ledger as Emerging. One soft week wasn’t decay after all.\n\nGetting here took three wrong turns, and each one taught me something about the data. There’s a prompt at the end if you want to build something like this yourself. The scheme isn’t specific to articles. It should work on any stream of text with timestamps, such as forum posts, support tickets, papers, reviews, or commit messages.\n\n## The Shape of It\n\nItems come in every day and pile up in a SQLite database. Once a week, two very different layers look at that pile:\n\n- **The math layer** is stateless. It embeds items, clusters them, counts each cluster by week, and applies a fixed rule that labels each one either*new* ,*emerging* ,*accelerating* ,*fading* , or*steady* . Everything is re-derived from scratch on every run. The only place an LLM touches this layer is to give a name to clusters that already qualified on the numbers.\n- **The analyst layer** is stateful. A single LLM call reads the math’s output along with its own memory, which is a markdown file of themes it’s been tracking. It writes the narrative, updates that file, and lists every place where it disagrees with the math.\n\nThat’s the whole system. The analyst is responsible for carrying storylines across weeks, and the math is responsible for the numbers. The analyst can argue with the numbers but can’t change them.\n\nHere’s what a disagreement looks like in the output. This one is from the September 11 run, where the analyst noticed that a cluster’s label didn’t match what was in it:\n\n```\n{\n  \"subject\": \"Apple App Store policies and developer disputes (34 items, emerging, z 3.33)\",\n  \"math_says\": \"A sizeable emerging cluster labeled around App Store policy and developer disputes.\",\n  \"analyst_says\": \"Mostly a mislabeled Apple hardware-launch spike — the exemplars are iPhone 18 Pro, AirPods 5, Mac mini M6, iPhone Duo, not policy disputes.\",\n  \"why\": \"The cluster label doesn't describe its contents; the mass is a September Apple-event product dump (a predictable calendar spike), with only one genuine App-Store-rejection item, so the 'developer disputes' framing overstates a real trend.\"\n}\n```\n\nIt took a few wrong turns to get to that point.\n\n## Failure #1: One Cluster Called “Everything AI”\n\nThe first clustering run used a cosine similarity threshold of 0.75, which is a pretty standard starting point. It produced one giant cluster of 715 items, about 70% of the week’s stories. Its first few members were “Israel’s $50M Experiment to Change U.S. Public Opinion,” “The World Cup Could Not Be Americanized,” “The Secret Origins of Amazon’s Alexa,” and “My Images Are Dithered.” Nothing in that list belongs together.\n\nThe problem was that I trusted a threshold before I’d measured the space it was operating in. So I ran a quick test on two sentences that have nothing to do with each other:\n\n``` python\nfrom shared.embeddings import embed_texts\n\na, b = embed_texts([\n    \"DuckDB adds columnar json support\",\n    \"A recipe for sourdough bread\",\n])\nprint(a @ b)   # 0.595\n```\n\nDatabases and sourdough came out at 0.595. With this embedding model (gemini-embedding-001), *unrelated* text sits around 0.6. A 0.75 threshold is barely above noise, which is why nearly anything with “AI” in it ended up in the same bucket.\n\nSo I swept the threshold over a week of stories (about 1,000 items):\n\n| threshold | clusters | largest | singleton rate | \n|---|---|---|---|\n| 0.75 | 26 | 715 | 65% | \n| 0.80 | 204 | 346 | 80% | \n| 0.83 | 459 | 200 | 83% | \n| **0.85** | **628** | **114** | 87% | \n| 0.88 | 840 | 31 | 94% | \n\nThe numbers alone don’t tell you which row is right. For that, I wrote a small script that prints the contents of each cluster at a given threshold, and then I just read them. At 0.85, the best clusters snapped into single topics. One was the launch of a new Claude model: “Claude Opus 5”, “Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard”, “Elevated errors on Claude Opus 5”. At 0.88, related articles started splitting apart, and 94% of clusters held a single story.\n\n```\n--- cluster of 23 ---\n    [hn] Claude Opus 5\n    [hn] Opus 5 is currently #1 on Artificial Analysis Intelligence Leaderboard\n    [hn] \"We removed over 80% of Claude Code's system prompt for Opus 5 and Fable 5\"\n    [hn] ARC-AGI Leaderboard\n    [hn] Claude Code has a hardcoded instruction telling Opus 5 not to use subagents\n    [hn] Elevated Errors for Opus 5\n    ... and 17 more\n```\n\nThere’s no metric here. The evaluation is me reading clusters and deciding whether they belong together. At this scale, that’s fine. Someone who knows the domain can look at 30 clusters in ten minutes and tell whether they’re coherent, and that’s a better check than any metric I could have created.\n\nThis came up again later. On their own, titles are thin, so I changed what gets embedded. Now it’s the title plus the top comments, which say a lot more about what a story is actually about. When I re-ran the calibration, the best threshold was still 0.85, but the clusters at that threshold were a lot better. The big catch-all blob split out into several coherent themes. It was the same number, but different input, and I knew it still held because I checked.\n\nWindow size matters too. Run the same 0.85 over three months (about 11,000 items) and the largest cluster grows from 114 to several hundred, because a greedy algorithm that keeps absorbing near neighbors drifts as more stories arrive. That’s one reason the analyst reads cluster contents and doesn’t trust their labels.\n\n## Failure #2: Blogs Don’t Trend\n\nWith clustering fixed, the first run’s strongest signal (z = 1.36, growth 3.14×) turned out to be… one prolific author writing about PostgreSQL. Great posts, but one person publishing on their own schedule isn’t a trend. The analyst spent a big chunk of that first report explaining why single-source clusters don’t count.\n\nThe next strongest signals were two clusters built entirely from one vendor’s product blog. The analyst threw those out too:\n\nThese are product release notes, not trend signal. A vendor announcing GreyNoise integration or SOC 2 certification does not constitute a community storyline.\n\nIt was right and the problem wasn’t the model. It was the data. At that point, the corpus was mostly RSS feeds from about 200 blogs. Blogs are great *context*, but trends need many independent people reacting to the same things, with real timestamps. A feed of 200 personal blogs is 200 sample sizes of one.\n\nSo I changed the main source to the Hacker News firehose and pulled every story that gets at least 10 points, which works out to about 150 a day. I also dropped the keyword filtering.\n\nThe original version of the HN agent searched for keywords like “pandas,” “sql,” and “machine learning,” which seems sensible, but it means the system could only ever find things I already knew to look for. If I’m building a system to judge quality, filtering for precision makes sense. For a radar, I want coverage. My filters were encoding my blind spots.\n\nAnother advantage of using Hacker News data is Algolia indexes all of it, including timestamps, going back years. I backfilled 12 months of history in minutes. The corpus went from about 4,300 mostly-blog items to about 61,000 today.\n\n## Failure #3: Time is a Schema Decision\n\nThis one is less dramatic but it’s the gotcha most people will hit.\n\nHacker News has a memory. You can ask Algolia for everything posted last March and get it back with real timestamps. You can’t do that with RSS. A feed shows you what’s in it right now, and once a post scrolls off the end, it’s gone for good. That’s why the TrendsWorks system keeps its own corpus. If you can’t go back and get a source’s history, you have to collect it as it happens.\n\nAnd once you’re collecting, you need a clock. Every item gets a discovered_at timestamp the first time the system sees it, and that value never changes. If the same story shows up the next day with more points, only the engagement numbers get updated.\n\nThat sounds like a small detail, but if re-seeing an item bumped its timestamp, every popular story would smear across several weeks and the weekly counts would be wrong. `discovered_at` is the trend clock, and the whole math layer depends on it being right.\n\nThen the backfill nearly broke it. Backfilled items need `discovered_at` set to the story’s *original* published time. If you naively stamp them with “now,” a year of history collapses into a single day. In my case it also tripped a sanity rule that was meant to catch RSS feeds that resurface ancient posts. Neither failure is obvious until you look at a histogram with one enormous bar in it.\n\nThere’s one more time rule worth mentioning: the math only analyzes *complete* weeks. The report goes out Friday morning and covers Friday through Thursday night. The week that’s in-progress is excluded, so the system never mistakes “the week isn’t over yet” for “this trend is dying.”\n\n## Why I Kept It Boring\n\nThere are two choices here that people will want to argue with.\n\n**No vector database.** Embeddings are stored as BLOBs in SQLite. The pipeline never does live nearest-neighbor lookups. It clusters one window, once a week, in one batch, and brute-force cosine similarity over the whole thing is a few milliseconds of numpy at this scale. A vector database would be more infrastructure to run for a search problem this system doesn’t have.\n\n**No HDBSCAN.** The clustering is a greedy centroid algorithm in about 40 lines. It has one knob, the threshold from Failure #1, and every assignment has a one-sentence explanation: “this item was at least 0.85 similar to that cluster’s centroid.” Here’s the whole thing:\n\n``` php\ndef cluster_items(vectors: np.ndarray, threshold: float = 0.85) -> list[Cluster]:\n    \"\"\"Assign each vector (rows L2-normalized, chronological order) to the\n    most similar existing cluster if cosine ≥ threshold, else start a new\n    one. Returns clusters ordered by first appearance.\"\"\"\n    clusters: list[Cluster] = []\n    if len(vectors) == 0:\n        return clusters\n    centroids = np.empty((0, vectors.shape[1]), dtype=np.float32)\n    sums = np.empty((0, vectors.shape[1]), dtype=np.float32)  # unnormalized member sums\n\n    for i, vec in enumerate(vectors):\n        if len(clusters) > 0:\n            sims = centroids @ vec\n            best = int(np.argmax(sims))\n            if sims[best] >= threshold:\n                clusters[best].member_indices.append(i)\n                sums[best] += vec\n                c = sums[best] / np.linalg.norm(sums[best])\n                centroids[best] = c\n                clusters[best].centroid = c\n                continue\n        # start a new cluster\n        clusters.append(Cluster(centroid=vec.copy(), member_indices=[i]))\n        centroids = np.vstack([centroids, vec[None, :]])\n        sums = np.vstack([sums, vec[None, :]])\n\n    return clusters\n```\n\nA fancier algorithm could probably produce slightly better clusters, but the whole point of this system is that the analyst (and I) can argue with the math, and you can’t argue with something you can’t explain.\n\n## Where the LLMs Actually Are\n\nIt’s easy to assume a system like this is LLM calls all the way down. It isn’t:\n\n| call | model tier | volume | cost | \n|---|---|---|---|\n| embeddings | commodity embedding model | ~150 items/day (backfill once) | pennies; ~$1–2 for the whole year | \n| cluster labels | cheapest chat tier (Haiku) | ~40 tiny calls/week | cents | \n| the analyst | frontier tier (Opus) | **1 call/week** | ~$1.50 | \n\nGetting items into the system is completely mechanical. No LLM decides what to fetch or what’s relevant. The funnel narrows *before* the expensive judgment happens, not after. By the time the analyst runs, it’s reading a compact summary of maybe a few dozen clusters, not tens of thousands of articles.\n\nEarly on, I was on the free tier for embeddings, which capped me at 1,000 requests a day. I had a backlog of more than 50,000 items, so at that rate it would have taken almost two months to catch up.\n\nThe fix was to embed newest first. The weekly analysis only looks at the last few weeks, so that part filled in within a day or two and the deep history caught up on its own. When you’re rationed, the order you spend the quota in matters as much as how much you have.\n\n## The Memory Is a Markdown File\n\nAll of the analyst’s memory across weeks lives in one file, `themes.md`. Each theme gets a status, when it was first seen, a summary, evidence, and a “watch for” line. Here’s the entry the analyst wrote for the AI evaluation theme on that first run:\n\n```\n## AI Systems Evaluation & Operational Cost\n**Status:** Emerging\n**First seen:** 2026-07-16\n**Last updated:** 2026-07-16\n**Summary:** A growing body of practitioner writing argues that evaluation quality and inference cost management — not model capability — are the real bottlenecks for AI in production. The conversation spans LLM evals, agent task-completion benchmarks, and the paradox of rising AI bills even as per-token prices fall.\n**Evidence:** Gradient Flow published at least five pieces in the 12-week window on this theme ('Generation is cheap. Evaluation is everything.', 'Your AI agent looks capable. But can it actually finish the job?', 'Why your AI bills are going up even as tokens get cheaper'). Eugene Yan's 'How to Work and Compound with AI' adds a practitioner workflow angle. Consistent weekly presence across the full window.\n**Watch for:** New tooling announcements around LLM evaluation frameworks (e.g., updates to LangSmith, RAGAS, or new entrants); cost-benchmarking posts from cloud providers or infra startups; whether the agent-reliability angle spawns its own cluster.\n```\n\nI like the “watch for” line a lot. It’s the analyst leaving a note for its future self about what would confirm or kill the theme. On the next run, when no standalone cluster showed up for it, it added: “If it stays invisible in the math for another 2–3 runs, reconsider status.” That’s the kind of judgment a z-score can’t make.\n\nThe analyst rewrites this file every week, but anything I add by hand under a theme survives the rewrite. So if I think it’s wrong about something, I can say so right there in its own memory, and it’ll read my note next Friday.\n\nI didn’t need a database for this. A markdown file is easy to read, easy to edit, and easy to put in git.\n\n## What the Argument Buys You\n\nGoing back to that first run, the analyst’s override only carries weight because the math’s verdict is printed right next to it. If I’d let the LLM quietly adjust the status, I’d have a system that sounds smart and can’t be checked. If I’d only trusted the math, I’d have thrown out a real theme after one slow week.\n\nNeither layer is the ground truth. The tension between them is the product.\n\nIt keeps working, too. In the September 11 run, the math flagged a brand-new cluster about a UN vote on world map projections: z-score 6.91, growth 14×. The analyst called it what it was, one news moment and out of scope. In the same report, it took an AI-and-mathematics cluster that the math just called “emerging,” connected it to two quiet weeks and a new research-integrity angle it had been tracking in its ledger, and upgraded the theme to Accelerating. The math couldn’t see that trajectory. The analyst couldn’t have found the cluster without the math.\n\nAnd when they agree, that’s a signal too. The disagreements section never gets dropped from the email. If there’s nothing to report, it says “None this week — the layers agree.”\n\n## Prompt\n\nIf you’re interested in playing with these ideas, here’s a prompt to get started with. Most of the lessons in this post came from questions about the data, not the model, so the prompt starts with an interview instead of a spec. It covers the four questions that shaped every design decision here.\n\nThis is also exactly how TrendsWorks started: an interview about sources, what “trend” should mean, and when the report is needed, all before any code got written.\n\n## Show the prompt\n\n```\nI want to build a trends radar over a stream of text — something that\ndetects what's *moving* over time, not just what's new today. Before\nwriting any code, interview me:\n\n1. **The stream.** What text arrives over time that I care about?\n   (forum posts, support tickets, papers, reviews, commit messages,\n   news…) For whatever I name, press me on three properties: Does it\n   carry honest timestamps? Can history be backfilled, or does the\n   corpus only start the day we turn it on? And is it many independent\n   voices reacting to the same world — which can trend — or a few\n   voices on their own schedules, which is context, not signal?\n2. **The decision.** What decision or ritual will the report feed, and\n   when does it happen? The time-bucket boundary and delivery day\n   should serve that moment, not the calendar.\n3. **What \"trend\" means to me.** Rising volume? Acceleration? Things\n   appearing from nothing? Things fading? Which of those would I\n   actually act on?\n4. **What the model will actually see.** For each item, what text\n   genuinely describes what it's *about*? Titles? Bodies? Replies?\n   Make me verify the input isn't boilerplate before we embed anything.\n\nThen propose a design that follows these principles — and tell me where\nmy situation justifies breaking them:\n\n- **Two deliberately contrasting layers.** A stateless quantitative\n  layer (embed → cluster → count per complete time bucket →\n  deterministic status rules) that re-derives everything each run, and\n  an LLM analyst with a human-editable memory file that interprets the\n  numbers, carries storylines across windows, and must list every\n  disagreement with the math. Disagreements are surfaced, never\n  auto-reconciled.\n- **A persistent corpus** (SQLite is plenty) where the first-seen\n  timestamp is immutable — that's the trend clock — and re-ingestion\n  only refreshes engagement and metadata. Analyze complete buckets\n  only, so an unfinished week never looks like a decline.\n- **Calibrate, don't default.** Pick the clustering threshold by\n  eyeballing real clusters at several values; it depends on the\n  embedding model *and* the input text, and it won't survive changing\n  either.\n- **LLM placement discipline.** Intake is mechanical and LLM-free; a\n  cheap model names clusters; the best model available makes the\n  once-per-period analyst call — that one call is the product.\n- **Prefer pieces you can explain** — greedy clustering, z-scores —\n  until the data proves you need more.\n\nStart with the interview. Then give me a build plan whose first\nmilestone is a walking skeleton running against my real data in the\nfirst session.\n```\n\n", "url": "https://wpnews.pro/news/the-llm-overruled-the-math-and-it-was-right", "canonical_source": "https://lonriesberg.com/posts/trends-radar-two-brains/", "published_at": "2026-11-10 14:45:22+00:00", "updated_at": "2026-11-10 14:54:35.486322+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-agents"], "entities": ["TrendsWorks", "Data Elixir", "Ben Lorica", "Gradient Flow", "Eugene Yan"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-llm-overruled-the-math-and-it-was-right", "markdown": "https://wpnews.pro/news/the-llm-overruled-the-math-and-it-was-right.md", "text": "https://wpnews.pro/news/the-llm-overruled-the-math-and-it-was-right.txt", "jsonld": "https://wpnews.pro/news/the-llm-overruled-the-math-and-it-was-right.jsonld"}}