{"slug": "you-don-t-need-an-llm-to-cluster-llm-traces", "title": "You don't need an LLM to cluster LLM traces", "summary": "Seldon's Trace Audit v0 clusters LLM traces into compilable programs using hard contract blocks and block-local DBSCAN on a short deterministic feature string, achieving 1.000/0.9999/1.000 pair precision/recall/purity on 12,517 traces with near-zero model cost, while GPT-4.1, GPT-5.6-luna, and Phi-4 summarizers hurt purity and should be used only for labeling after clustering.", "body_md": "[← Blog](/blog)\n\n# You don't need an LLM to cluster LLM traces\n\nIf you run an OpenAI-compatible gateway long enough, you drown in traces that look different and are the same program. Same JSON schema. Same “return the answer span.” Same “normalize this issuer name.” Different companies, amounts, and sentences — identical contracts. That is the bet behind Seldon: find those programs, then compile the repetitive ones into cheaper deterministic pipelines. This post is about the first half of that loop — Trace Audit — and what we learned when we stress-tested it at scale.\n\nCompilation only works if we can answer a precise question first:\n\nWhich traces are the **same replaceable program** — not merely the same topic?\n\nWe built that answer for Trace Audit v0, measured it on **12,517** traces, and kept the mistakes that taught us more than the models did. Audit is only half the story: once clusters exist, Seldon treats them as **specs for a restricted form of program synthesis**. We sketch that compilation step below; a dedicated follow-up will go deep on the synthesizer. For the framing we already published, see [Program synthesis, and why compiling LLM calls into ETL is one](/blog/program-synthesis-and-automated-etl).\n\n**TL;DR.** Hard **contract blocks** first, then **block-local DBSCAN** on a short deterministic feature string (`embedding_text`\n\n). We hit **1.000 / 0.9999 / 1.000** pair precision / recall / purity — with about **$0** model cost. Live summarizers (GPT-4.1, GPT-5.6-luna, Phi-4) did **not** help clustering; they hurt purity. Use LLMs for labels and dossiers *after* clustering, not as the clustering signal.\n\n## Why this matters if you ship LLM products\n\nMost teams start with a frontier model because it is the fastest way to prototype extraction, classification, and normalization. Then volume shows up, and a large share of calls turn into the same shape over and over:\n\n```\ndocument / ticket / record\n  → same prompt template\n  → same response_format / output shape\n  → different payloads\n```\n\nYou are paying reasoning prices for ETL wearing a prompt — often without lineage, without a ranked view of *which* workflows dominate spend, and without a safe path off the model.\n\nSeldon’s loop is deliberately boring:\n\n**See it**— gateway traffic → contract-compatible clusters → workflow map ranked by savings.*(this post)***Compile it**— search a typed operator library for a cheaper pipeline that reproduces the cluster.*(teased below; deeper post coming)***Trust it**— shadow validation on*your*traces; automatic LLM fallback; demote when quality slips.\n\nTrace Audit is step one. Get the buckets wrong and everything downstream is fiction.\n\n## The wrong instinct (and why it’s everywhere)\n\nThe fashionable pipeline for “understand my LLM traffic” looks like this:\n\nIt is intuitive. Summaries are readable. Embeddings are familiar. Product-analytics stacks popularized the pattern.\n\nFor a **compiler**, it is the wrong objective function.\n\nWe do not want “these traces talk about invoices.”\n\nWe want “these traces are the **same compilable extraction program**.”\n\n| Trace A | Trace B |\n|---|---|\n`response_format: json_schema` | free-form prose |\noutput: `{vendor, invoice_number, total}` | output: a paragraph |\n| → ETL / structured-extraction candidate | → leave on the LLM |\n\nTopic similarity merges them. A compiler must not.\n\nConversely, two span-extraction calls with different entities *should* cluster: same roles, same mode, same output shape — different payloads.\n\nProgram identity ≠ document similarity.\n\n## The bigger picture: audit → compile → runtime\n\nBefore we zoom into clustering, here is where Audit sits in the product.\n\nDay one you change `base_url`\n\nand get a normal gateway. Over time, repeated traces author free I/O specs. Audit finds the **same program** hiding under different payloads. The compiler tries to **synthesize a cheaper implementation** of that program. Runtime keeps the LLM as a safety net — compilation is partial on purpose.\n\nThat framing is deliberate program synthesis, not “ask an LLM to write a pipeline and hope”:\n\n| Classical synthesis idea | How Seldon uses it |\n|---|---|\n| Spec | The clustered traces (programming-by-example) |\n| Search space | Fixed, typed operator library (families A–G + adapters) |\n| Proposal prior | Frontier model at compile time only — suggests decompositions |\n| Acceptance | Empirical: match the LLM oracle above threshold on held-out traces |\n| Failure mode | Refuse to compile; leave the call on the frontier model |\n\nThree departures from textbook synthesis matter in production: acceptance is **statistical**, not a proof; the LLM oracle is **noisy** (sometimes the compiled path is *more* correct on deterministic families); and the I/O contract is **inferred** from traffic before search starts. Programs are shallow DAGs of whole NLP/ML operators — no deep control flow. If a task needs genuine reasoning or recursion, Gate 1 says **no** and we do not pretend otherwise.\n\nWe will unpack operator signatures, composition, shadow promotion, and “when we refuse to compile” in a **follow-up post on the Seldon compiler**. This post stays on the prerequisite: can we partition traffic into compiler-safe clusters at all?\n\n## How Trace Audit v0 works\n\nBefore synthesis, Audit has to partition traffic into **contract-compatible** clusters. False merges poison synthesis — you would search against a mixed spec. False splits hide savings — you would never notice the volume.\n\n### Pipeline\n\n### What we put in the contract block\n\nWe hash the things that decide whether two calls are **substitutable**:\n\n- workspace / scope\n- role sequence (\n`system`\n\n→`user`\n\n, …) - response mode (\n`text`\n\n,`json_object`\n\n,`json_schema`\n\n, tools, …) - schema / tool signature\n- output shape (\n`id`\n\n,`span`\n\n,`json_object`\n\n,`numeric`\n\n,`prose`\n\n, …) - JSON keys (when structured)\n- IO-contract bucket (classification / extraction / …)\n\nWe deliberately **do not** put raw prompt length in the block key. Length is a payload attribute, not a substitutability constraint. (We learned that the hard way — more below.)\n\n### What we cluster on\n\nNot the raw prompt. A short deterministic string we call `embedding_text`\n\n:\n\n```\ntask_signature | roles=… | mode=… | shape=… | keys=… | hints=…\n```\n\nVariable entities (names, amounts, spans) are collapsed on purpose. That is the feature, not a bug: we are clustering **programs**, not documents.\n\n### What “compilable” means in v0\n\nAudit is not trying to rewrite every chat. v0 targets a small taxonomy of atomic tasks that already had mature non-LLM pipelines:\n\n| Family | Example surface | Typical compiled path |\n|---|---|---|\nA Classification | intent / ticket label | classical classifier |\nB Span extraction | SQuAD-style answer span | sequence labeler / rules |\nC Structured extraction | invoice JSON | schema-bound ETL |\nD Entity / id resolution | name → canonical id | gazetteer / linker |\nE Similarity | pair score | embedding / metric model |\nF Normalization | legal name → match key | deterministic transforms |\nG Numeric | word problem → number | calculator / DSL |\n∅ Non-compilable | open-ended prose / judgment | stay on the LLM |\n\nIf a block is all prose, or has no support, we **skip** it. Refusing to pretend is part of the product.\n\n## The experiment\n\nWe built a realistic OpenAI-shaped suite at about **10% of each public train split**, plus small curated families:\n\n| Source | Samples |\n|---|---|\n| Banking77 (intent) | 1,000 |\n| SQuAD (span extraction) | 8,759 |\n| CoNLL-2003 (entity → id) | 1,404 |\n| STS-B (pair similarity) | 574 |\n| GSM8K (numeric answer) | 747 |\n| Curated invoice / issuer / prose | 10 each |\n| Adversarial “bridge” traces | 3 |\nTotal | 12,517 |\n\n**12,504** traces were eligible after skipping tiny / all-prose blocks.\n\nBy design, raw inputs stay diverse (**2,109** distinct) while `embedding_text`\n\ncollapses (**998** distinct). That is program identity showing up in the feature space.\n\nWe compared, under the **same** blocks, eligibility guard, and cosine cutoff (~0.82 / `eps=0.18`\n\n):\n\n(no LLM)`embedding_text`\n\n+ block-local DBSCAN- Deterministic summary + DBSCAN\n- Live LLM summary + DBSCAN for\n**GPT-4.1**,** GPT-5.6-luna**, and** Phi-4.7 → Phi-4**\n\n## Results\n\n| Method | Precision | Recall | Purity | Notes |\n|---|---|---|---|---|\n`embedding_text` + DBSCAN | 1.000 | 0.9999 | 1.000 | Winning path |\n| Deterministic summary + DBSCAN | 1.000 | 0.9999 | 1.000 | Same score, more work |\n| GPT-4.1 summary + DBSCAN | 0.966 | 0.998 | 0.920 | 8.2% degraded summaries |\n| GPT-5.6-luna summary + DBSCAN | 0.966 | 0.988 | 0.920 | 0.6% degraded — still loses |\n| Phi-4 summary + DBSCAN | 1.000 | 0.771 | 1.000 | Early-stopped subset; Foundry too slow |\n\nWhat we would tell another team building this:\n\n**The win is ordering + density, not “smarter text.”** Hold the algorithm fixed and summaries add nothing over`embedding_text`\n\n.**Live LLM summaries reduce purity.** They re-expand topical wording that contract features suppressed. Great for humans. Bad for program identity.**Reliability ≠ usefulness.** Luna was a much cleaner summarizer than GPT-4.1 — and still lost to the no-LLM arm.**Zero incompatible contract false merges** across completed arms. The block key carries precision. Do not loosen it casually.**Cost follows quality here.** Winning arm model bill ≈**$0**. Full-suite GPT summarization is the expensive path — and the worse one.\n\n## Two bugs that mattered more than the model\n\nBefore the numbers looked this clean, recall was stuck near **0.73** for every embedding variant. The culprit was not DBSCAN or the model. It was the block key:\n\n**Input-length buckets**(`short`\n\n/`medium`\n\n/`long`\n\n) split identical workflows purely on prompt length. Removed.**Numeric-as-span** typing was too aggressive (e.g. a year that “looks numeric”). We only reclassify a numeric output as an extraction span when the prompt signals extraction*and*the number appears as a whole token in the input — so computed-numeric workflows (GSM8K, margins) stay`numeric`\n\n.\n\nAfter those fixes, the recall ceiling jumped to **~0.999** with no precision loss.\n\nFor Trace Audit, **block-key design dominates embedding model choice**.\n\nWe also compared density clustering to mutual-kNN connected components at the same similarity cutoff. DBSCAN recovered far more recall; mutual-kNN under-connected tight blocks unless we added lexical/template edges — which then created the false merges that cost purity. Density inside a contract block is the right inductive bias.\n\n## What we’re shipping (and what we’re not)\n\n| Do | Don’t |\n|---|---|\n| Hard contract blocking first | Cluster by topic / raw prompt similarity alone |\nBlock-local DBSCAN on `embedding_text` | Put LLM summaries on the clustering critical path |\n| Skip singleton / prose-only blocks | Treat every trace as compilable |\n| Use LLMs for post-hoc labels & family dossiers | Assume a bigger summarizer fixes merging |\n| Keep the LLM as runtime fallback | Silently replace open-ended reasoning |\n\nThis is how Seldon stays honest as a **compiler**, not a magic black-box optimizer: refuse what is not compilable, validate what is, fall back when quality drops.\n\nNext on the Audit path (still contract-first): cluster **dossiers** → family labels A–G / non-compilable → ranked savings map → handoff to synthesis. The LLM belongs in that dossier / proposal step, not in the partition.\n\nAnd after Audit? That is the compiler post: how we turn a clean cluster into an SIR operator graph, run it in **shadow** against your history, promote only above threshold, and keep one-click rollback + frontier fallback. Same `base_url`\n\neither way.\n\n## Why developers should care\n\nIf you are building:\n\n- an\n**LLM gateway / router** with cost analytics, - an\n**eval or observability** stack that groups “similar” calls, - or an internal “auto-ETL from traces” prototype,\n\n…the default ML instinct (“embed the prompt / the summary”) optimizes for the wrong similarity. You will get beautiful clusters your compiler cannot trust.\n\nContract-first Trace Audit is how Seldon turns gateway traffic into a **ranked map of compilable savings**. The synthesis layer is how those savings become **inspectable, versioned pipelines** — the opposite of distilling into another black-box model. We already introduced that framing in [Program synthesis, and why compiling LLM calls into ETL is one](/blog/program-synthesis-and-automated-etl); a follow-up will go deeper on the Trace Audit → compile handoff.\n\nIntegrate like any OpenAI-compatible router. Keep your provider keys. Let the traces author the spec.\n\n## Caveats\n\n- Labels are fixture\n`workflow_id`\n\ns aligned with contract families — partly self-fulfilling by construction. Real customer traffic is the next bar. - Curated invoice / issuer / prose rows are template-cycled.\n- Phi-4 was measured on an early-stopped stratified subset (Foundry latency); GPT arms ran the full eligible set.\n- Public NLP benchmarks are not your production mix. Treat absolute scores as fixture-bound.\n\n## Bottom line\n\nSeldon Trace Audit clusters programs, not vibes.\n\nOn 12.5k traces, the cheapest path was also the best path: deterministic contracts → hard blocks → block-local density clustering. Adding GPT-4.1 or GPT-5.6-luna summaries made clusters *worse*. That is not an argument against LLMs in the product — it is an argument for putting them where they belong: explaining buckets and proposing decompositions **after** the compiler-safe partition is already correct.\n\nAudit is the front door. Behind it sits a restricted, example-driven synthesizer: traces as spec, typed operators as search space, your held-out history as verifier, frontier model as fallback. We will write that story next.\n\nIf you are tired of paying frontier prices for the same extraction job ten thousand times, that is the system we are building.\n\nRoute your traffic through Seldon and watch recurring work turn into cheaper pipelines.\n\n[Sign up for open beta →](/sign-up)", "url": "https://wpnews.pro/news/you-don-t-need-an-llm-to-cluster-llm-traces", "canonical_source": "https://seldon-ai.com/blog/you-dont-need-an-llm-to-cluster-llm-traces", "published_at": "2026-07-23 17:38:38+00:00", "updated_at": "2026-07-23 17:52:23.940955+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-products", "developer-tools"], "entities": ["Seldon", "Trace Audit", "GPT-4.1", "GPT-5.6-luna", "Phi-4"], "alternates": {"html": "https://wpnews.pro/news/you-don-t-need-an-llm-to-cluster-llm-traces", "markdown": "https://wpnews.pro/news/you-don-t-need-an-llm-to-cluster-llm-traces.md", "text": "https://wpnews.pro/news/you-don-t-need-an-llm-to-cluster-llm-traces.txt", "jsonld": "https://wpnews.pro/news/you-don-t-need-an-llm-to-cluster-llm-traces.jsonld"}}