# We fired the frontier model from two AI pipelines. Here are the receipts

> Source: <https://seldon-ai.com/blog/we-fired-the-frontier-model>
> Published: 2026-10-15 21:40:18+00:00

[← Blog](https://seldon-ai.com/blog)

# We fired the frontier model from two AI pipelines. Here are the receipts.

Seldon Functions compiles the LLM call in the middle of your data pipeline into a faster, cheaper program, and proves it against the model you trust. Two production experiments: 5× cheaper and more accurate on agent traces. ~8,000× cheaper and up to 10× faster on PII detection.

## TLDR;

**Routers pick a model. Seldon builds a program.** We pointed it at two hard, high-volume LLM workloads, and here's what came back.

- **Agent trace analytics: 5× cheaper, more accurate and faster.** Seldon replaced a frontier-model call with a compiled program. The program passed 95.0% of held-out checks where the frontier model passed 92.5%, cost 80% less per request and ran 21% faster. Its output also clustered better downstream, at 0.56 ARI against 0.33.
- **PII detection: ~8,000× cheaper, up to 10× faster, every document processed.** Seldon compiled the LLM out*entirely* : a specialized span model plus deterministic code. It handled all 66 test documents for**$0.00014 in total** , including 8 that the frontier-model baseline failed on. It isn't perfect. It missed some bare first names, and the receipts below show exactly where.
- **Zero fallbacks, a one-line switch, and about $15 to compile.** Both compiled programs served every fresh input on their own primary path. Each pays for itself in roughly 1,000 to 2,300 calls.

Switching takes one line. Seldon gives you a new model ID, and the rest of your code doesn't change:

```
- model="gpt-5.6-sol"
+ model="seldon/function/<FUNCTION_ID>@<REVISION_ID>"
```

### Why it works

Most LLM calls in production aren't chat. They're **functions**: text goes in, JSON with a fixed schema comes out, millions of times. You pay frontier-model prices on every call because a frontier model is what worked when you wrote the prompt. The usual alternatives are shortcuts: sample less, truncate more, or swap in the cheapest model and hope. Almost nobody measures what those shortcuts cost in quality.

Seldon turns that guess into a measurement:

1. **Your model sets the bar.** You bring the prompt, the schema and a reference model you trust. Every alternative is scored against that reference, on your contract.
2. **It searches whole programs, not just models.** Smaller LLMs, specialized models (GLiNER for spans, Jev for decisions) and deterministic kernels, combined into multi-step pipelines. That's how a PII detector ends up with no LLM on its main path.
3. **You see the trade before you make it.** A comparison report puts up to three alternatives next to your reference, with quality, critical failures, cost, p50/p95 and fallback rate for each. You choose, and nothing activates on its own.
4. **It ships with a safety net.** Outputs are validated at runtime, and your original call stays wired in as a fallback. The endpoint is OpenAI-compatible, so the OpenAI SDK, LiteLLM, LangChain and the Vercel AI SDK work unchanged.

### How we tested it

We built both experiments as standalone scripts and ran them against production `api.seldon-ai.com`, with real models and real wallet charges. Each compiled Function went head to head with the simple baseline (the original prompt on GPT-5.6 Sol) on the same held-out inputs. We also ran the tools you'd otherwise reach for: regex, Presidio + GLiNER, hand-written features and LLM summaries. We published the failures too.

**Where these experiments come from.** PostHog's public [write-up on LLM trace clustering](https://posthog.com/blog/llm-analytics-clustering-how-it-works) inspired the trace experiment. The anonymization needs that companies licensing enterprise data, like [Mercor](https://www.mercor.com/data/), describe publicly inspired the PII experiment. We designed both setups, built the data and ran everything ourselves. We don't represent either company, neither was involved or endorsed this, and nothing here measures, reproduces or challenges their systems.

## The receipts: Seldon against the simple baseline

The simple baseline is what most teams ship first: send the original prompt to a frontier model (GPT-5.6 Sol) for every record. Each Seldon pipeline below sits side by side with that baseline, on the same inputs. **Bold** marks the better result in each row.

### Trace signals (experiment inspired by PostHog's clustering write-up)

|  | Baseline: one Sol call | Seldon Function | Change | 
|---|---|---|---|
| What runs per request | `gpt-5.6-sol` , original prompt | `gpt-5.6-luna` , original prompt, with Sol only as fallback | Smaller model | 
| ***Seldon's held-out report (80 synthetic examples)*** |  |  |  | 
| Passed | 74/80 (92.5%) | **76/80 (95.0%)** | +2.5 pts | 
| Critical failures | 0 | 0 | Same | 
| Cost per request | $0.0080 | **$0.0016** | **−80% (5× cheaper)** | 
| p50 / p95, serial | 3.3 s / 4.8 s | **2.6 s / 3.8 s** | −21% / −21% | 
| ***External test: 40 held-out Snorkel traces*** |  |  |  | 
| Traces completed | 20 of 40 | 20 of 40 | Same (the strict quote check rejects the same long traces) | 
| Clustering quality, task-ID ARI (15 paired traces) | 0.33 | **0.56** | +0.23 | 
| Cost per call | $0.0188 | **$0.0041** | −78% | 
| p50 / p95 per call, through the gateway | 5.0 s / 6.8 s | **4.4 s / 5.7 s** | −12% / −16% | 
| Fallback used | n/a | 0 of 124 calls |  | 
| One-time compile cost | n/a | $14.58, 110 min |  | 

### PII identification (experiment inspired by enterprise data-anonymization needs)

|  | Baseline: one Sol call | Seldon Function | Change | 
|---|---|---|---|
| What runs per request | `gpt-5.6-sol` , original prompt | `gliner2.5-multi-v1` spans, then 2 deterministic steps, with Sol only as fallback | No LLM on the primary path | 
| ***Seldon's held-out report (80 synthetic examples)*** |  |  |  | 
| Passed | **67/80 (83.8%)** | 64/80 (80.0%), flagged unreliable by Seldon | −3.8 pts | 
| Critical failures | 2 | **1** | −1 | 
| Cost per request | $0.018 | **$0.000002** | **~9,000× cheaper** | 
| p50 / p95, serial | 6.8 s / 11.1 s | **0.67 s / 1.0 s** | ~10× faster | 
| ***External test: 66 fictional documents, 264 gold entities*** |  |  |  | 
| Documents completed | 53 of 66 | **66 of 66** | +13 documents | 
| Entities fully covered, on the 53 both completed | **100%** | 93.2% | −6.8 pts | 
| Documents with a leak, on those 53 | **0** | 10 | +10 leaky documents | 
| Entities fully covered, all 66 | n/a (13 documents unprocessed) | 92.4% |  | 
| Text over-redacted | **0.2%** | 1.2% | +1.0 pt | 
| Total cost, 66 documents | ≈ $1.14 | **$0.00014** | ~8,000× cheaper | 
| p50 / p95 per document, through the gateway | 6.7 s / 11.9 s | **1.7 s / 2.0 s** | ~4× / ~6× faster | 
| Fallback used | n/a | 0 of 66 calls |  | 
| One-time compile cost | n/a | $17.12, 95 min |  | 

*Held-out reports use Seldon's own synthetic examples, an independent judge and serial timing. External tests call the production gateway, so their latencies include network and gateway overhead. Seldon costs are settled receipts. Sol costs in the external tests are token-price estimates at $5 input / $30 output per million tokens. Both baselines are our own Sol calls. None of these numbers describe PostHog's or Mercor's systems.*

Put plainly, Seldon beat the baseline on every trace row that differs. On PII it's far cheaper, several times faster, and finished every document. But it leaked names the baseline caught, and a local Presidio + GLiNER setup beat it on recall (see below). One result is a clear win. The other is a tradeoff you'd want to see before making it. Seldon shows you both.

## How a Function gets compiled

The lifecycle has four explicit steps, and nothing is activated unless you activate it.

1. 
**Create.** Upload the call's prompt, input/output JSON schemas and reference model. The trace demo's contract is`{"text": "..."} → {intent, actions[], outcome, failure_signals[], evidence[]}` .
2. 
**Preview** (paid, capped). Seldon generates synthetic examples for the task, runs your reference on them, infers an evaluator and returns a quote: forecast cost, forecast minutes and which components are eligible. The server enforces your cap, currently up to $100.
3. 
**Optimize.** Bounded search over complete programs: up to 64 candidates, 16 steps and 3 learned nodes. Building blocks:
  - smaller LLMs (e.g. GPT-5.6 Luna)
  - specialized models: `fastino/gliner2.5-multi-v1` for spans and records,`typesafe/jev` for boolean/enum/pair decisions
  - deterministic kernels: exact span alignment, JSON-pointer record projection, lookup, arithmetic and constraint checks
  - the reference itself, as an optional fallback
 Small search spaces are screened exhaustively. Larger ones use SMAC random-forest Bayesian optimization over whole-program evaluations.
4. 
**Compare and activate.** The run publishes a**comparison report** with up to three executable alternatives beside the reference. Each one comes with quality, critical failures, full-path cost, p50/p95, fallback rate and coverage. You choose. An alternative can't be activated without proof that its primary path executes; a candidate that only ever succeeds through fallback is never callable.

The compiler only ever sees your prompt, your schemas, one training example and the synthetic data it generates itself. In both demos, the held-out documents and traces, and every gold label, stayed on the demo side. None of them shaped the search.

Then you call it like any other model:

``` python
from openai import OpenAI
import json

client = OpenAI(base_url="https://api.seldon-ai.com/v1", api_key=SELDON_API_KEY)

resp = client.chat.completions.create(
    model=f"seldon/function/{FUNCTION_ID}@{REVISION_ID}",
    messages=[{"role": "user", "content": json.dumps({"text": trace_fragment})}],
)
signals = json.loads(resp.choices[0].message.content)
```

It's the same OpenAI-compatible endpoint, so the OpenAI SDK, LiteLLM, LangChain and the Vercel AI SDK work unchanged. Every call writes an execution receipt with its real charge.

### Where Seldon stops and your code starts

In both demos, Seldon replaces exactly one step: the LLM call your prompt used to make. Everything around it is ordinary Python in a standalone `demo.py` on your machine. That covers loading data, chunking, checking outputs, redaction, embeddings, clustering and scoring. The demo doesn't import the Seldon package, and it calls the Function over plain HTTP.

The diagrams below use one color code throughout:

- **Purple:** hosted by Seldon, inside the compiled Function.
- **Grey:** the demo script, i.e. your application code.
- **Green:** local models running on your own CPU.
- **Orange:** a direct call to the reference model (Sol) as a baseline, outside any Function.
- **Dashed:** the reference fallback. It's wired in, but it never fired on the fresh inputs in these runs.

Every baseline goes through the *same* script. Only the one step that the Function replaces gets swapped out, so each comparison measures that step and nothing else.

## Demo 1: Agent trace analytics

### The inspiration

PostHog has published a detailed [walkthrough of how it built LLM trace clustering](https://posthog.com/blog/llm-analytics-clustering-how-it-works), with links to its code. It's one of the clearest public explanations of this problem, and it shaped how we set up our experiment. As they describe it:

- each trace is rendered as line-numbered text
- huge traces are uniformly downsampled to fit context
- a sample of traces is taken every hour
- each sampled trace gets a structured summary from GPT-4.1 nano
- the summaries are embedded with `text-embedding-3-large`
- the embeddings go through UMAP and HDBSCAN
- a GPT-5.2 agent names each cluster

It's a thoughtful design, and the post is candid about tradeoffs that anyone building trace analytics runs into:

- **Cost forces sampling.** Summarizing every trace would be expensive. So they sample a subset each hour, use a small, fast model, and plan to move to the Batch API.
- **Downsampling loses information.** Dropping lines from huge traces keeps them inside context limits, at what they call "the cost of some accuracy/quality".
- **One schema for everyone.** A single general-purpose summary schema can't fit every kind of trace. Letting users bring their own prompts and structured outputs is listed as a possible future improvement.
- **Structure beats prose.** They summarize into a structured schema, with line references, specifically because it embeds better than free text.

As described, one step in that design is a function-shaped LLM call:

*An illustration drawn from PostHog's published description, not from their code. Purple, thick border: the kind of step Seldon Functions is designed for. Dotted grey: an agent loop, which Functions doesn't compile.*

In a design like this, the summary step is a fixed prompt plus a fixed schema, run on every sampled trace. That's the kind of call a compiler can work on, and it's the step our experiment focuses on. The labeling agent is a multi-step loop that uses tools, and Functions doesn't touch those. Embeddings, UMAP, HDBSCAN and orchestration stay in your code.

### Our experiment

We borrowed the overall shape and changed the parts we wanted to test. This is our own setup, not a reproduction of PostHog's system, and we didn't run anything against it. The left column is only there to show where our choices differ from the design that inspired them.

|  | PostHog's write-up (for context) | Our experiment | 
|---|---|---|
| Data | Customer traces | Public [Snorkel τ²-Bench airline traces](https://huggingface.co/datasets/snorkelai/Tau2-Bench-Verified-Airline-With-Code-Agents/tree/23e3afcdd9492e9e6cc240c124be07665a8d211b) , pinned revision | 
| Huge traces | Uniform line downsampling | Overlapping 4,000-char chunks; every character processed | 
| Coverage | Hourly sample | Every held-out trace in a seeded sample of 40 | 
| Per-trace call | Structured summary with line references | Structured signals with exact-quote evidence, compiled by Seldon | 
| Embedding | `text-embedding-3-large` , 3,072-d | Pinned MiniLM, run locally | 
| Clustering | UMAP to 100-d, HDBSCAN | UMAP to 5-d, HDBSCAN, same settings for every arm | 
| Cluster naming | GPT-5.2 agent | Not included | 

Our extraction contract asks for the user's intent, the ordered actions, the observed outcome (`completed` / `failed` / `unclear`) and any failure signals. Each signal must be backed by evidence quoted *exactly* from the trace, and the script checks every quote in code.

Before the call, the script drops rewards, task IDs and simulator state from every trace. Tasks are hash-partitioned, so repeated attempts at one task can't leak between train and test. After the call, the whole trace is rejected if any chunk's evidence isn't an exact quote. Task IDs are only used at the very end, for scoring.

The fallback has narrow triggers. A step's output has to fail schema or grounding validation, or serving admission has to refuse the request before anything is sent. Then Seldon runs your original Sol call inside the same 55-second deadline. If a provider call's outcome is uncertain, the fallback doesn't fire. A request that may already have been processed is never sent a second time.

### What Seldon built

Search returned two executable alternatives. Both are shown below exactly as the compiler recorded them.

- **Alternative 1: Luna alone, with Sol as fallback.** 76/80 on Seldon's held-out set against Sol's 74/80, with zero critical failures for either. It costs 20% as much per request and runs about 21% faster at both p50 and p95.
- **Alternative 2: a Jev classifier feeding Luna.** Jev makes the one enum decision,`outcome` , and Luna fills in the free-text fields using that decision as context. Deterministic code then assembles the final object, setting each field exactly once. This alternative matched Sol on pass rate but had two critical failures.

We wrote the selection rule down before any results arrived: highest quality first, then lowest cost, then lowest p95. Luna came out on top, and we activated it.

So a smaller model beat the frontier model on this task. That's less surprising than it sounds. It's a narrow extraction task with a strict, quote-grounded output, and a bigger model brings more ways to paraphrase when the rubric wants an exact quote. Is the win "just a cheaper model"? Here, yes. Seldon's job was to find it, prove it against your reference on your own contract, and serve it with a safety net.

### Does it cluster better?

We ran 40 seeded held-out traces through four representations. Each arm swaps out only the representation step. The chunking, embeddings, UMAP and HDBSCAN settings and the scoring are identical:

The two middle columns are the head-to-head: the same structured-signals contract, run once as a plain Sol call and once through the compiled Function. The outer columns give context.

|  | Hand-written features | Free-text summary | **Baseline: Sol structured signals** | **Seldon structured signals** | 
|---|---|---|---|---|
| What runs | `demo.py` , no model | Sol, summary prompt | Sol, original prompt | Compiled Function (Luna) | 
| Traces completed (of 40) | 40 | 40 | 20 | 20 | 
| Model calls | 0 | 235 | 128 | 124 | 
| Total cost | $0 | $2.92 | $2.40 | **$0.51** | 
| Cost per call | $0 | $0.0124 | $0.0188 | **$0.0041** | 
| p50 / p95 per call | n/a | 3.9 s / 5.0 s | 5.0 s / 6.8 s | **4.4 s / 5.7 s** | 
| Task-ID ARI (15 paired traces) | 0.36 | 0.05 | 0.33 | **0.56** | 

Against its baseline, Seldon's pipeline clustered better (0.56 against 0.33) at 22% of the per-call cost and lower latency. It also beat the hand-written features, which cost nothing but sit well below it. The free-text summary clustered barely better than random against task identity. That backs PostHog's own reasoning for structured summaries. Our summary arm is a single free-text field, though, not their schema.

The caveats matter here. Both extraction arms completed only 20 of the 40 traces, and 15 of their 20 failures were the *same* traces. The demo rejects a whole trace if any chunk's evidence isn't an exact quote, and long traces (median 6 chunks against 3) trip that more often. That's our harness being strict, not Seldon regressing against Sol. The next version scores per chunk. ARI is computed on the 15 traces both arms completed, using task IDs as a stand-in for real cluster labels. Read it as "Seldon's signals cluster at least as well as Sol's," not as proof of cluster quality.

### What this suggests for trace-analytics pipelines

PostHog's post names tradeoffs that any team building trace analytics will recognize. This isn't advice to PostHog. It's how a compiler relates to each of those tradeoffs in general, with our experiment as one data point:

- **Cost pushes teams toward sampling.** The common answer is a small model plus sampling, and the quality cost of that choice is rarely measured. Seldon flips the order. You set the quality bar with a model you trust, and it finds the cheapest program that holds that bar on your schema. Then you decide how much sampling you still need. Be careful with the comparison: we compiled against Sol, not against GPT-4.1 nano. A nano-class model may well be cheaper per call than our Luna program. What a comparison report adds is a measured number for how much quality any particular choice gives up on your schema.
- **Downsampling trades accuracy for context.** Our experiment chunks traces instead, and processes every character. That means more calls per long trace, so full coverage is only affordable if each call is cheap. That's the lever compilation pulls. You can also measure the trade directly: run the summary arm with`--summary-max-chars 24000` to compare uniform downsampling against full coverage.
- **Custom schemas multiply the work.** This is the strongest fit. Custom summaries mean thousands of distinct prompt-and-schema contracts, and nobody can hand-tune and cost-manage each one. With Functions, each schema is its own Function, compiled and measured against its own reference, and swapped in by model ID. The caveat is compile cost: about $15 and two hours per contract here. That pays off for schemas with real volume (break-even around 1,000 to 2,300 calls in this demo), not for one-off filters.
- **Summaries need to stay grounded.** PostHog's design uses line references for this. Our experiment used a different mechanism: it checked every quote against the source in code. That check rejected the same long traces for Sol and for Seldon. A grounding rule enforced in code finds failures that a prompt instruction only hopes to prevent, and those failures are what the next compilation should be scored on.

## Demo 2: PII identification

### The inspiration

Companies that license enterprise data to AI labs have to anonymize it first. Mercor is one public example. Its [data page](https://www.mercor.com/data/) describes licensing companies' everyday workflow data, such as contracts, case files, clinical notes, claims, underwriting files, client decks, invoices and shipping manifests. Its [homepage](https://www.mercor.com/) says automated anonymization masks more than 60 categories of sensitive identifiers.

We don't know how Mercor's anonymization is built, and we don't represent Mercor. Nothing here describes, tests or competes with its system. What we took from those public descriptions is the shape of a problem that any anonymization pipeline at this scale faces, and it's genuinely hard:

- **Recall is the only number that matters.** One unmasked name in a licensed corpus breaks the promise made to whoever is named. Over-redaction is a cost. A leak is a failure.
- **Volume multiplies cost.** At one frontier-model call per document, anonymization becomes a major line item, and the latency piles up across a corpus.
- **Documents contain text that looks like instructions.** Real files include emails, notes and pasted content addressed to someone. An LLM detector can refuse them, or worse, follow them.
- **LLMs are bad at counting characters.** A detector that names the right entity at the wrong offset still leaks.
- **There are 60+ categories and dozens of document types.** That's many detection contracts, not one.

### Our experiment

We designed this setup ourselves. We built a fictional, recruiting-style corpus: 66 documents with 264 gold entities, generated by `make_corpus.py --seed 20261012`. It covers screening notes, résumés, interview excerpts, reference requests, contractor invoices, documents with no PII, and cover notes with *embedded instructions*. We used five labels (PERSON, ORGANIZATION, EMAIL, PHONE and INVOICE), which is a small slice of a 60-category problem. Scoring is strict: a partially redacted entity counts as a leak. No data from Mercor or any other company was used.

The Function only identifies spans with exact Unicode offsets. The redaction and pseudonymization are deterministic and run in your code.

### What Seldon built

There's no LLM on the primary path. The activated program has three steps, and only the first one calls a model:

GLiNER returns spans with offsets the runtime has validated against the source. The two deterministic kernels reshape those spans into your output schema without inferring anything new. If any step fails validation, the original Sol call runs instead. On the fresh documents it never had to.

Everything that decides what happens to a span is script code, and so is the scoring:

All four arms pass through the same validation, redaction and scoring code. A partially covered entity counts as a leak, no matter which arm produced the span.

Again, the two middle columns are the head-to-head. Because Sol didn't finish every document, the table also shows each arm on the 53 documents that all four completed. That's the only fair like-for-like comparison.

|  | Regex | **Baseline: Sol prompt** | **Seldon Function** | Presidio + GLiNER | 
|---|---|---|---|---|
| Where it runs | `demo.py` | Sol, direct call | Hosted, GLiNER2.5 | Your CPU | 
| Documents completed (of 66) | 66 | 53 | **66** | **66** | 
| *On the 53 documents all arms completed:* |  |  |  |  | 
| Entities fully covered | n/a | **100%** | 93.2% | 95.7% | 
| Documents with a leak | n/a | **0** | 10 | 5 | 
| *On every document each arm completed:* |  |  |  |  | 
| Entities fully covered | 25.8% | 100% of 53 | 92.4% of 66 | **96.2% of 66** | 
| Documents with a leak | 53 | **0** of 53 | 14 of 66 | 6 of 66 | 
| Text over-redacted | 0% | **0.2%** | 1.2% | 1.4% | 
| Total provider cost | $0 | ≈ $1.14 | **$0.00014** | $0 + your hardware | 
| p50 / p95 per document | under 1 ms | 6.7 s / 11.9 s | 1.7 s / 2.0 s | 0.22 s / 0.84 s | 

There are three things worth noticing.

**1. Against the baseline, cost and latency collapse.** The Seldon pipeline handled all 66 documents for $0.00014 in total, against about $1.14 for the Sol arm. Through the gateway, it was about 4× faster at p50 and 6× faster at p95. In Seldon's serial timing it was 10× faster. Presidio is faster still, because it runs on your own machine with no network hop.

**2. The baseline didn't finish the job.** All 8 cover notes with embedded instructions came back as HTTP 400 `invalid_request`. We haven't yet diagnosed whether the gateway or the provider rejected them. Another 5 outputs had offsets that didn't point at the text they claimed. Seldon processed all 66, including those 8. A span model doesn't read embedded instructions as instructions, because it doesn't follow instructions at all. It labels tokens. In an anonymization pipeline, a rejected document is a gap: it either gets dropped or it goes out unmasked.

**3. It isn't the most accurate arm, and we won't pretend otherwise.** Seldon leaked in 14 documents, where Sol leaked in none on the 53 it completed. Every miss was a first name standing on its own, like an interview speaker label ("Wen: …") or an email greeting ("Hi Kalani,"). That's 20 of 94 PERSON entities. Seldon found every full name, organization, email, phone number and invoice ID. It also over-redacts in a pattern: it tagged "1–4" in "a 1–4 scale" as an INVOICE in every no-PII document. Local Presidio + GLiNER covered more, at 96.2%. Most of its misses came from our demo's own invoice regex, and a one-line fix would likely push it near 99%. That's the real bar to beat.

Seldon's own compile-time report scored this pipeline at 80.0% against the reference's 83.8%, and it flagged its own quality measurement as unreliable for this task. We activated the pipeline anyway, as a deliberate choice under the pre-registered selection rule, to measure it end to end. That's the point of the comparison report. Seldon doesn't make the call for you, and it doesn't hide the number you'd need to make it.

This is pseudonymization tooling on fictional data. It is not an anonymization certification.

### What this suggests for anonymization pipelines

These points are general. They're drawn from our experiment, not from any company's system.

- **Cost and coverage at corpus scale.** A span-model program with validated offsets processed every document, including the ones the frontier-model baseline failed on, at a cost that rounds to zero and about 2 seconds or less per document through the gateway. For anonymization that runs on every document, that changes what you can afford to re-run, re-check and audit.
- **Offsets you can trust.** On our synthetic tasks, Sol often didn't match itself on character offsets (more below). Native span models return offsets that are checked against the source before anything leaves the Function.
- **Many contracts, each measured.** Sixty categories across contracts, clinical notes and invoices aren't one detection problem. Each document family or category group can be its own Function, compiled and measured against its own reference. The comparison report tells you the leak rate of every alternative before you activate anything.
- **What we have to fix first.** Bare first names are exactly the leak a privacy team would test for first, so recall comes before any claim here. The error analysis points to fixes that keep per-document cost near zero.*These are proposals, not shipped features:* derive label descriptions from your prompt instead of bare enum names, make the extraction threshold a searchable parameter, and add a deterministic name-propagation kernel, so that finding "Wen Iyer" also masks a later bare "Wen". After that comes the hybrid a privacy team would probably want: the span model handles the bulk, and the frontier model only sees documents where the span model's confidence is low.

## The part where it didn't work

These results came after several production runs that produced no callable revision. Those failures shaped the product more than the successes did.

**The frontier model couldn't agree with itself.** Seldon checks that your reference is stable before searching. On the PII task, Sol was asked the same question twice and matched itself only 13–16 times out of 20 across runs. In every mismatch the entity text and labels were *identical*, and only the character offsets differed. Separately, 11 of 156 answers contained offsets that didn't select the text they named. Asking an LLM to count characters is a weak contract, and the search space now has two answers to it. `align_spans` is a deterministic kernel that finds an LLM's quoted text in the source and computes the offsets itself, rejecting missing or ambiguous matches. Native span models like GLiNER return validated offsets directly. The winning pipeline uses the second.

**A pass/fail gate hid useful answers.** Earlier runs required a candidate to clear 95% quality, ≤70% of the reference's cost and ≤110% of its p95, or nothing shipped. On one trace run, Luna scored 97.45% against Sol's 96.82% on answered cases, at about a fifth of the cost, and still produced no callable revision. Reference-refused inputs fell out of the denominator and blocked qualification. So we replaced the gate with a comparison report. Targets steer the search, the measurements are published, and you choose with the evidence in front of you.

**Early quotes didn't fit.** The first previews forecast $57.83 and $72.24, against a $25 cap at the time. Optimize refused them, activation without qualification returned 409, and no fake revision was substituted. The forecasts and the operating envelope had to be reconciled first.

**Serving failed twice under load.** During the comparison round, two Seldon calls failed fast with HTTP 502. The trace revision reserves about 135k tokens per call, and the PII compilation running in the same workspace had used up that workspace's tokens-per-minute budget. Serving admission therefore failed before any provider call was made. Both receipts showed $0. We root-caused the bug and fixed it in PR #584.

We left all of this in because a benchmark that only shows wins isn't one you can rely on.

## The economics

Compilation is a one-time cost. Here's a back-of-envelope break-even on these measurements:

- **Trace signals:** $14.58 to compile, saving $0.0064–$0.0147 per call depending on trace length. It pays for itself somewhere between ~1,000 and ~2,300 calls.
- **PII:** $17.12 to compile, saving about $0.018 per document against Sol. It pays for itself after roughly 950 documents.

Both compilations still had held amounts that hadn't settled: $3.87 for traces and $8.48 for PII. If those settle in full, break-even moves to about 2,900 calls and 1,400 documents. These figures exclude engineering time on your side and any self-hosting cost for the local baselines. They also exclude the cost of the misses you accept, which is the number only you can price.

The full comparison round cost $40.35 in wallet charges, against an approved $100. The temporary API key was revoked afterward and verified to return 401.

## What we're not claiming

- **Inspired by, not representing or competing.** PostHog's public write-up inspired the trace experiment. The anonymization needs that companies like Mercor describe publicly inspired the PII experiment. We designed both setups ourselves. Neither company was involved or has endorsed this, and we don't speak for either one. Nothing here is a benchmark against, or a critique of, their systems. We used none of their code or data.
- **Not a reproduction of PostHog's system.** Our trace experiment borrows the overall shape of PostHog's published design. It uses different embeddings, no sampling and no labeling agent. Our summary arm is a free-text summary from Sol, not their structured schema on GPT-4.1 nano. We did not benchmark against nano.
- **Nothing about Mercor's internals.** We don't know how Mercor's anonymization works. Our PII demo covers 5 categories on templated fictional documents, which are simpler than real ones. 92.4% coverage on them is not a privacy guarantee on yours.
- **Synthetic held-out sets.** Seldon's held-out reports use synthetic examples it generated itself. The external comparisons use a public benchmark and a fictional corpus. Neither one is your production traffic.
- **Proxy cluster quality.** Trace ARI is a task-ID proxy on 15 paired traces. Cluster coherence still needs human review.
- **No production SLOs.** Latency figures are serial, per-request measurements from one client location.

## Code release

Codebase used for the benchmark will be open sourced soon.

**[Get started at seldon-ai.com →]**

Route your traffic through Seldon and investigate recurring work that could benefit from a specialized program.

[Sign up for open beta →](https://seldon-ai.com/sign-up)