# You don't need an LLM to cluster LLM traces

> Source: <https://seldon-ai.com/blog/you-dont-need-an-llm-to-cluster-llm-traces>
> Published: 2026-07-23 17:38:38+00:00

[← Blog](/blog)

# You don't need an LLM to cluster LLM traces

If you run an OpenAI-compatible gateway long enough, you drown in traces that look different and are the same program. Same JSON schema. Same “return the answer span.” Same “normalize this issuer name.” Different companies, amounts, and sentences — identical contracts. That is the bet behind Seldon: find those programs, then compile the repetitive ones into cheaper deterministic pipelines. This post is about the first half of that loop — Trace Audit — and what we learned when we stress-tested it at scale.

Compilation only works if we can answer a precise question first:

Which traces are the **same replaceable program** — not merely the same topic?

We built that answer for Trace Audit v0, measured it on **12,517** traces, and kept the mistakes that taught us more than the models did. Audit is only half the story: once clusters exist, Seldon treats them as **specs for a restricted form of program synthesis**. We sketch that compilation step below; a dedicated follow-up will go deep on the synthesizer. For the framing we already published, see [Program synthesis, and why compiling LLM calls into ETL is one](/blog/program-synthesis-and-automated-etl).

**TL;DR.** Hard **contract blocks** first, then **block-local DBSCAN** on a short deterministic feature string (`embedding_text`

). We hit **1.000 / 0.9999 / 1.000** pair precision / recall / purity — with about **$0** model cost. Live summarizers (GPT-4.1, GPT-5.6-luna, Phi-4) did **not** help clustering; they hurt purity. Use LLMs for labels and dossiers *after* clustering, not as the clustering signal.

## Why this matters if you ship LLM products

Most teams start with a frontier model because it is the fastest way to prototype extraction, classification, and normalization. Then volume shows up, and a large share of calls turn into the same shape over and over:

```
document / ticket / record
  → same prompt template
  → same response_format / output shape
  → different payloads
```

You are paying reasoning prices for ETL wearing a prompt — often without lineage, without a ranked view of *which* workflows dominate spend, and without a safe path off the model.

Seldon’s loop is deliberately boring:

**See it**— gateway traffic → contract-compatible clusters → workflow map ranked by savings.*(this post)***Compile it**— search a typed operator library for a cheaper pipeline that reproduces the cluster.*(teased below; deeper post coming)***Trust it**— shadow validation on*your*traces; automatic LLM fallback; demote when quality slips.

Trace Audit is step one. Get the buckets wrong and everything downstream is fiction.

## The wrong instinct (and why it’s everywhere)

The fashionable pipeline for “understand my LLM traffic” looks like this:

It is intuitive. Summaries are readable. Embeddings are familiar. Product-analytics stacks popularized the pattern.

For a **compiler**, it is the wrong objective function.

We do not want “these traces talk about invoices.”

We want “these traces are the **same compilable extraction program**.”

| Trace A | Trace B |
|---|---|
`response_format: json_schema` | free-form prose |
output: `{vendor, invoice_number, total}` | output: a paragraph |
| → ETL / structured-extraction candidate | → leave on the LLM |

Topic similarity merges them. A compiler must not.

Conversely, two span-extraction calls with different entities *should* cluster: same roles, same mode, same output shape — different payloads.

Program identity ≠ document similarity.

## The bigger picture: audit → compile → runtime

Before we zoom into clustering, here is where Audit sits in the product.

Day one you change `base_url`

and get a normal gateway. Over time, repeated traces author free I/O specs. Audit finds the **same program** hiding under different payloads. The compiler tries to **synthesize a cheaper implementation** of that program. Runtime keeps the LLM as a safety net — compilation is partial on purpose.

That framing is deliberate program synthesis, not “ask an LLM to write a pipeline and hope”:

| Classical synthesis idea | How Seldon uses it |
|---|---|
| Spec | The clustered traces (programming-by-example) |
| Search space | Fixed, typed operator library (families A–G + adapters) |
| Proposal prior | Frontier model at compile time only — suggests decompositions |
| Acceptance | Empirical: match the LLM oracle above threshold on held-out traces |
| Failure mode | Refuse to compile; leave the call on the frontier model |

Three departures from textbook synthesis matter in production: acceptance is **statistical**, not a proof; the LLM oracle is **noisy** (sometimes the compiled path is *more* correct on deterministic families); and the I/O contract is **inferred** from traffic before search starts. Programs are shallow DAGs of whole NLP/ML operators — no deep control flow. If a task needs genuine reasoning or recursion, Gate 1 says **no** and we do not pretend otherwise.

We will unpack operator signatures, composition, shadow promotion, and “when we refuse to compile” in a **follow-up post on the Seldon compiler**. This post stays on the prerequisite: can we partition traffic into compiler-safe clusters at all?

## How Trace Audit v0 works

Before synthesis, Audit has to partition traffic into **contract-compatible** clusters. False merges poison synthesis — you would search against a mixed spec. False splits hide savings — you would never notice the volume.

### Pipeline

### What we put in the contract block

We hash the things that decide whether two calls are **substitutable**:

- workspace / scope
- role sequence (
`system`

→`user`

, …) - response mode (
`text`

,`json_object`

,`json_schema`

, tools, …) - schema / tool signature
- output shape (
`id`

,`span`

,`json_object`

,`numeric`

,`prose`

, …) - JSON keys (when structured)
- IO-contract bucket (classification / extraction / …)

We deliberately **do not** put raw prompt length in the block key. Length is a payload attribute, not a substitutability constraint. (We learned that the hard way — more below.)

### What we cluster on

Not the raw prompt. A short deterministic string we call `embedding_text`

:

```
task_signature | roles=… | mode=… | shape=… | keys=… | hints=…
```

Variable entities (names, amounts, spans) are collapsed on purpose. That is the feature, not a bug: we are clustering **programs**, not documents.

### What “compilable” means in v0

Audit is not trying to rewrite every chat. v0 targets a small taxonomy of atomic tasks that already had mature non-LLM pipelines:

| Family | Example surface | Typical compiled path |
|---|---|---|
A Classification | intent / ticket label | classical classifier |
B Span extraction | SQuAD-style answer span | sequence labeler / rules |
C Structured extraction | invoice JSON | schema-bound ETL |
D Entity / id resolution | name → canonical id | gazetteer / linker |
E Similarity | pair score | embedding / metric model |
F Normalization | legal name → match key | deterministic transforms |
G Numeric | word problem → number | calculator / DSL |
∅ Non-compilable | open-ended prose / judgment | stay on the LLM |

If a block is all prose, or has no support, we **skip** it. Refusing to pretend is part of the product.

## The experiment

We built a realistic OpenAI-shaped suite at about **10% of each public train split**, plus small curated families:

| Source | Samples |
|---|---|
| Banking77 (intent) | 1,000 |
| SQuAD (span extraction) | 8,759 |
| CoNLL-2003 (entity → id) | 1,404 |
| STS-B (pair similarity) | 574 |
| GSM8K (numeric answer) | 747 |
| Curated invoice / issuer / prose | 10 each |
| Adversarial “bridge” traces | 3 |
Total | 12,517 |

**12,504** traces were eligible after skipping tiny / all-prose blocks.

By design, raw inputs stay diverse (**2,109** distinct) while `embedding_text`

collapses (**998** distinct). That is program identity showing up in the feature space.

We compared, under the **same** blocks, eligibility guard, and cosine cutoff (~0.82 / `eps=0.18`

):

(no LLM)`embedding_text`

+ block-local DBSCAN- Deterministic summary + DBSCAN
- Live LLM summary + DBSCAN for
**GPT-4.1**,** GPT-5.6-luna**, and** Phi-4.7 → Phi-4**

## Results

| Method | Precision | Recall | Purity | Notes |
|---|---|---|---|---|
`embedding_text` + DBSCAN | 1.000 | 0.9999 | 1.000 | Winning path |
| Deterministic summary + DBSCAN | 1.000 | 0.9999 | 1.000 | Same score, more work |
| GPT-4.1 summary + DBSCAN | 0.966 | 0.998 | 0.920 | 8.2% degraded summaries |
| GPT-5.6-luna summary + DBSCAN | 0.966 | 0.988 | 0.920 | 0.6% degraded — still loses |
| Phi-4 summary + DBSCAN | 1.000 | 0.771 | 1.000 | Early-stopped subset; Foundry too slow |

What we would tell another team building this:

**The win is ordering + density, not “smarter text.”** Hold the algorithm fixed and summaries add nothing over`embedding_text`

.**Live LLM summaries reduce purity.** They re-expand topical wording that contract features suppressed. Great for humans. Bad for program identity.**Reliability ≠ usefulness.** Luna was a much cleaner summarizer than GPT-4.1 — and still lost to the no-LLM arm.**Zero incompatible contract false merges** across completed arms. The block key carries precision. Do not loosen it casually.**Cost follows quality here.** Winning arm model bill ≈**$0**. Full-suite GPT summarization is the expensive path — and the worse one.

## Two bugs that mattered more than the model

Before the numbers looked this clean, recall was stuck near **0.73** for every embedding variant. The culprit was not DBSCAN or the model. It was the block key:

**Input-length buckets**(`short`

/`medium`

/`long`

) split identical workflows purely on prompt length. Removed.**Numeric-as-span** typing was too aggressive (e.g. a year that “looks numeric”). We only reclassify a numeric output as an extraction span when the prompt signals extraction*and*the number appears as a whole token in the input — so computed-numeric workflows (GSM8K, margins) stay`numeric`

.

After those fixes, the recall ceiling jumped to **~0.999** with no precision loss.

For Trace Audit, **block-key design dominates embedding model choice**.

We also compared density clustering to mutual-kNN connected components at the same similarity cutoff. DBSCAN recovered far more recall; mutual-kNN under-connected tight blocks unless we added lexical/template edges — which then created the false merges that cost purity. Density inside a contract block is the right inductive bias.

## What we’re shipping (and what we’re not)

| Do | Don’t |
|---|---|
| Hard contract blocking first | Cluster by topic / raw prompt similarity alone |
Block-local DBSCAN on `embedding_text` | Put LLM summaries on the clustering critical path |
| Skip singleton / prose-only blocks | Treat every trace as compilable |
| Use LLMs for post-hoc labels & family dossiers | Assume a bigger summarizer fixes merging |
| Keep the LLM as runtime fallback | Silently replace open-ended reasoning |

This is how Seldon stays honest as a **compiler**, not a magic black-box optimizer: refuse what is not compilable, validate what is, fall back when quality drops.

Next on the Audit path (still contract-first): cluster **dossiers** → family labels A–G / non-compilable → ranked savings map → handoff to synthesis. The LLM belongs in that dossier / proposal step, not in the partition.

And after Audit? That is the compiler post: how we turn a clean cluster into an SIR operator graph, run it in **shadow** against your history, promote only above threshold, and keep one-click rollback + frontier fallback. Same `base_url`

either way.

## Why developers should care

If you are building:

- an
**LLM gateway / router** with cost analytics, - an
**eval or observability** stack that groups “similar” calls, - or an internal “auto-ETL from traces” prototype,

…the default ML instinct (“embed the prompt / the summary”) optimizes for the wrong similarity. You will get beautiful clusters your compiler cannot trust.

Contract-first Trace Audit is how Seldon turns gateway traffic into a **ranked map of compilable savings**. The synthesis layer is how those savings become **inspectable, versioned pipelines** — the opposite of distilling into another black-box model. We already introduced that framing in [Program synthesis, and why compiling LLM calls into ETL is one](/blog/program-synthesis-and-automated-etl); a follow-up will go deeper on the Trace Audit → compile handoff.

Integrate like any OpenAI-compatible router. Keep your provider keys. Let the traces author the spec.

## Caveats

- Labels are fixture
`workflow_id`

s aligned with contract families — partly self-fulfilling by construction. Real customer traffic is the next bar. - Curated invoice / issuer / prose rows are template-cycled.
- Phi-4 was measured on an early-stopped stratified subset (Foundry latency); GPT arms ran the full eligible set.
- Public NLP benchmarks are not your production mix. Treat absolute scores as fixture-bound.

## Bottom line

Seldon Trace Audit clusters programs, not vibes.

On 12.5k traces, the cheapest path was also the best path: deterministic contracts → hard blocks → block-local density clustering. Adding GPT-4.1 or GPT-5.6-luna summaries made clusters *worse*. That is not an argument against LLMs in the product — it is an argument for putting them where they belong: explaining buckets and proposing decompositions **after** the compiler-safe partition is already correct.

Audit is the front door. Behind it sits a restricted, example-driven synthesizer: traces as spec, typed operators as search space, your held-out history as verifier, frontier model as fallback. We will write that story next.

If you are tired of paying frontier prices for the same extraction job ten thousand times, that is the system we are building.

Route your traffic through Seldon and watch recurring work turn into cheaper pipelines.

[Sign up for open beta →](/sign-up)