If you run an OpenAI-compatible gateway long enough, you drown in traces that look different and are the same program. Same JSON schema. Same “return the answer span.” Same “normalize this issuer name.” Different companies, amounts, and sentences — identical contracts. That is the bet behind Seldon: find those programs, then compile the repetitive ones into cheaper deterministic pipelines. This post is about the first half of that loop — Trace Audit — and what we learned when we stress-tested it at scale.
Compilation only works if we can answer a precise question first:
Which traces are the same replaceable program — not merely the same topic?
We built that answer for Trace Audit v0, measured it on 12,517 traces, and kept the mistakes that taught us more than the models did. Audit is only half the story: once clusters exist, Seldon treats them as specs for a restricted form of program synthesis. We sketch that compilation step below; a dedicated follow-up will go deep on the synthesizer. For the framing we already published, see Program synthesis, and why compiling LLM calls into ETL is one.
TL;DR. Hard contract blocks first, then block-local DBSCAN on a short deterministic feature string (embedding_text
). We hit 1.000 / 0.9999 / 1.000 pair precision / recall / purity — with about $0 model cost. Live summarizers (GPT-4.1, GPT-5.6-luna, Phi-4) did not help clustering; they hurt purity. Use LLMs for labels and dossiers after clustering, not as the clustering signal.
Why this matters if you ship LLM products #
Most teams start with a frontier model because it is the fastest way to prototype extraction, classification, and normalization. Then volume shows up, and a large share of calls turn into the same shape over and over:
document / ticket / record
→ same prompt template
→ same response_format / output shape
→ different payloads
You are paying reasoning prices for ETL wearing a prompt — often without lineage, without a ranked view of which workflows dominate spend, and without a safe path off the model.
Seldon’s loop is deliberately boring:
See it— gateway traffic → contract-compatible clusters → workflow map ranked by savings.(this post)Compile it— search a typed operator library for a cheaper pipeline that reproduces the cluster.(teased below; deeper post coming)Trust it— shadow validation onyourtraces; automatic LLM fallback; demote when quality slips.
Trace Audit is step one. Get the buckets wrong and everything downstream is fiction.
The wrong instinct (and why it’s everywhere) #
The fashionable pipeline for “understand my LLM traffic” looks like this:
It is intuitive. Summaries are readable. Embeddings are familiar. Product-analytics stacks popularized the pattern.
For a compiler, it is the wrong objective function.
We do not want “these traces talk about invoices.”
We want “these traces are the same compilable extraction program.”
| Trace A | Trace B |
|---|---|
response_format: json_schema |
free-form prose |
output: {vendor, invoice_number, total} |
output: a paragraph |
| → ETL / structured-extraction candidate | → leave on the LLM |
Topic similarity merges them. A compiler must not.
Conversely, two span-extraction calls with different entities should cluster: same roles, same mode, same output shape — different payloads.
Program identity ≠ document similarity.
The bigger picture: audit → compile → runtime #
Before we zoom into clustering, here is where Audit sits in the product.
Day one you change base_url
and get a normal gateway. Over time, repeated traces author free I/O specs. Audit finds the same program hiding under different payloads. The compiler tries to synthesize a cheaper implementation of that program. Runtime keeps the LLM as a safety net — compilation is partial on purpose.
That framing is deliberate program synthesis, not “ask an LLM to write a pipeline and hope”:
| Classical synthesis idea | How Seldon uses it |
|---|---|
| Spec | The clustered traces (programming-by-example) |
| Search space | Fixed, typed operator library (families A–G + adapters) |
| Proposal prior | Frontier model at compile time only — suggests decompositions |
| Acceptance | Empirical: match the LLM oracle above threshold on held-out traces |
| Failure mode | Refuse to compile; leave the call on the frontier model |
Three departures from textbook synthesis matter in production: acceptance is statistical, not a proof; the LLM oracle is noisy (sometimes the compiled path is more correct on deterministic families); and the I/O contract is inferred from traffic before search starts. Programs are shallow DAGs of whole NLP/ML operators — no deep control flow. If a task needs genuine reasoning or recursion, Gate 1 says no and we do not pretend otherwise.
We will unpack operator signatures, composition, shadow promotion, and “when we refuse to compile” in a follow-up post on the Seldon compiler. This post stays on the prerequisite: can we partition traffic into compiler-safe clusters at all?
How Trace Audit v0 works #
Before synthesis, Audit has to partition traffic into contract-compatible clusters. False merges poison synthesis — you would search against a mixed spec. False splits hide savings — you would never notice the volume.
Pipeline
What we put in the contract block
We hash the things that decide whether two calls are substitutable:
- workspace / scope
- role sequence (
system
→user
, …) - response mode (
text
,json_object
,json_schema
, tools, …) - schema / tool signature
- output shape (
id
,span
,json_object
,numeric
,prose
, …) - JSON keys (when structured)
- IO-contract bucket (classification / extraction / …)
We deliberately do not put raw prompt length in the block key. Length is a payload attribute, not a substitutability constraint. (We learned that the hard way — more below.)
What we cluster on
Not the raw prompt. A short deterministic string we call embedding_text
:
task_signature | roles=… | mode=… | shape=… | keys=… | hints=…
Variable entities (names, amounts, spans) are collapsed on purpose. That is the feature, not a bug: we are clustering programs, not documents.
What “compilable” means in v0
Audit is not trying to rewrite every chat. v0 targets a small taxonomy of atomic tasks that already had mature non-LLM pipelines:
| Family | Example surface | Typical compiled path |
|---|---|---|
| A Classification | intent / ticket label | classical classifier |
| B Span extraction | SQuAD-style answer span | sequence labeler / rules |
| C Structured extraction | invoice JSON | schema-bound ETL |
| D Entity / id resolution | name → canonical id | gazetteer / linker |
| E Similarity | pair score | embedding / metric model |
| F Normalization | legal name → match key | deterministic transforms |
| G Numeric | word problem → number | calculator / DSL |
| ∅ Non-compilable | open-ended prose / judgment | stay on the LLM |
If a block is all prose, or has no support, we skip it. Refusing to pretend is part of the product.
The experiment #
We built a realistic OpenAI-shaped suite at about 10% of each public train split, plus small curated families:
| Source | Samples |
|---|---|
| Banking77 (intent) | 1,000 |
| SQuAD (span extraction) | 8,759 |
| CoNLL-2003 (entity → id) | 1,404 |
| STS-B (pair similarity) | 574 |
| GSM8K (numeric answer) | 747 |
| Curated invoice / issuer / prose | 10 each |
| Adversarial “bridge” traces | 3 |
| Total | 12,517 |
12,504 traces were eligible after skipping tiny / all-prose blocks.
By design, raw inputs stay diverse (2,109 distinct) while embedding_text
collapses (998 distinct). That is program identity showing up in the feature space.
We compared, under the same blocks, eligibility guard, and cosine cutoff (~0.82 / eps=0.18
):
(no LLM)embedding_text
- block-local DBSCAN- Deterministic summary + DBSCAN
- Live LLM summary + DBSCAN for GPT-4.1,** GPT-5.6-luna**, and** Phi-4.7 → Phi-4**
Results #
| Method | Precision | Recall | Purity | Notes |
|---|---|---|---|---|
embedding_text + DBSCAN |
1.000 | 0.9999 | 1.000 | Winning path |
| Deterministic summary + DBSCAN | 1.000 | 0.9999 | 1.000 | Same score, more work |
| GPT-4.1 summary + DBSCAN | 0.966 | 0.998 | 0.920 | 8.2% degraded summaries |
| GPT-5.6-luna summary + DBSCAN | 0.966 | 0.988 | 0.920 | 0.6% degraded — still loses |
| Phi-4 summary + DBSCAN | 1.000 | 0.771 | 1.000 | Early-stopped subset; Foundry too slow |
What we would tell another team building this:
The win is ordering + density, not “smarter text.” Hold the algorithm fixed and summaries add nothing overembedding_text
.Live LLM summaries reduce purity. They re-expand topical wording that contract features suppressed. Great for humans. Bad for program identity.Reliability ≠ usefulness. Luna was a much cleaner summarizer than GPT-4.1 — and still lost to the no-LLM arm.Zero incompatible contract false merges across completed arms. The block key carries precision. Do not loosen it casually.Cost follows quality here. Winning arm model bill ≈**$0**. Full-suite GPT summarization is the expensive path — and the worse one.
Two bugs that mattered more than the model #
Before the numbers looked this clean, recall was stuck near 0.73 for every embedding variant. The culprit was not DBSCAN or the model. It was the block key:
Input-length buckets(short
/medium
/long
) split identical workflows purely on prompt length. Removed.Numeric-as-span typing was too aggressive (e.g. a year that “looks numeric”). We only reclassify a numeric output as an extraction span when the prompt signals extractionandthe number appears as a whole token in the input — so computed-numeric workflows (GSM8K, margins) staynumeric
.
After those fixes, the recall ceiling jumped to ~0.999 with no precision loss.
For Trace Audit, block-key design dominates embedding model choice.
We also compared density clustering to mutual-kNN connected components at the same similarity cutoff. DBSCAN recovered far more recall; mutual-kNN under-connected tight blocks unless we added lexical/template edges — which then created the false merges that cost purity. Density inside a contract block is the right inductive bias.
What we’re shipping (and what we’re not) #
| Do | Don’t |
|---|---|
| Hard contract blocking first | Cluster by topic / raw prompt similarity alone |
Block-local DBSCAN on embedding_text |
Put LLM summaries on the clustering critical path |
| Skip singleton / prose-only blocks | Treat every trace as compilable |
| Use LLMs for post-hoc labels & family dossiers | Assume a bigger summarizer fixes merging |
| Keep the LLM as runtime fallback | Silently replace open-ended reasoning |
This is how Seldon stays honest as a compiler, not a magic black-box optimizer: refuse what is not compilable, validate what is, fall back when quality drops.
Next on the Audit path (still contract-first): cluster dossiers → family labels A–G / non-compilable → ranked savings map → handoff to synthesis. The LLM belongs in that dossier / proposal step, not in the partition.
And after Audit? That is the compiler post: how we turn a clean cluster into an SIR operator graph, run it in shadow against your history, promote only above threshold, and keep one-click rollback + frontier fallback. Same base_url
either way.
Why developers should care #
If you are building:
- an LLM gateway / router with cost analytics, - an eval or observability stack that groups “similar” calls, - or an internal “auto-ETL from traces” prototype,
…the default ML instinct (“embed the prompt / the summary”) optimizes for the wrong similarity. You will get beautiful clusters your compiler cannot trust.
Contract-first Trace Audit is how Seldon turns gateway traffic into a ranked map of compilable savings. The synthesis layer is how those savings become inspectable, versioned pipelines — the opposite of distilling into another black-box model. We already introduced that framing in Program synthesis, and why compiling LLM calls into ETL is one; a follow-up will go deeper on the Trace Audit → compile handoff.
Integrate like any OpenAI-compatible router. Keep your provider keys. Let the traces author the spec.
Caveats #
- Labels are fixture
workflow_id
s aligned with contract families — partly self-fulfilling by construction. Real customer traffic is the next bar. - Curated invoice / issuer / prose rows are template-cycled.
- Phi-4 was measured on an early-stopped stratified subset (Foundry latency); GPT arms ran the full eligible set.
- Public NLP benchmarks are not your production mix. Treat absolute scores as fixture-bound.
Bottom line #
Seldon Trace Audit clusters programs, not vibes.
On 12.5k traces, the cheapest path was also the best path: deterministic contracts → hard blocks → block-local density clustering. Adding GPT-4.1 or GPT-5.6-luna summaries made clusters worse. That is not an argument against LLMs in the product — it is an argument for putting them where they belong: explaining buckets and proposing decompositions after the compiler-safe partition is already correct.
Audit is the front door. Behind it sits a restricted, example-driven synthesizer: traces as spec, typed operators as search space, your held-out history as verifier, frontier model as fallback. We will write that story next.
If you are tired of paying frontier prices for the same extraction job ten thousand times, that is the system we are building.
Route your traffic through Seldon and watch recurring work turn into cheaper pipelines.