How do you turn traces into a training dataset?

Overmind Lab details a four-stage pipeline for converting production agent traces into fine-tuning datasets, emphasizing standardized capture via OpenTelemetry GenAI conventions and strategies like failure-weighted sampling. The process includes trace collection, sampling, labeling, and weekly rebuilds, using a refund agent as an example.

If you're running agents in production, you already have a training dataset sitting in your traces and most teams never look. Here's the four-stage pipeline for turning that raw telemetry into something you can actually fine-tune on. Originally published at overmindlab.ai. Every agentic system can create its own training data. Every agentic AI system you run is quietly writing its own training set, and most teams never use it. When an agent runs, it emits a stream of inputs and outputs: the prompt, the tool calls, the reasoning and the final answer. That stream is a trace, and once you standardise how you capture it the OpenTelemetry GenAI semantic conventions https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/ is the default schema here you have raw telemetry. A training dataset is narrower: a curated set of runs, scored against a spec, designed for fine-tuning jobs. Building a dataset like this is a four-stage pipeline. Four stages, then a weekly rebuild. Here's a worked example, using a refund agent, to show what each stage actually involves. If your agent is instrumented at all, you already have this part. Every run generates a trace: the goal it was given, every model call, every tool invocation, every observation from the environment, and the eventual outcome. That trace is the raw material everything downstream depends on. The industry is converging on a shared vocabulary for this. OpenTelemetry's GenAI working group has been building standard span attributes for LLM calls https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/ since 2024, covering model name, token counts, latency, and tool execution as first-class fields rather than something each team invents from scratch. That matters more than it sounds like it should, because a trace schema you have to redesign every time you switch observability vendors is a trace schema nobody trusts enough to build a dataset on top of. This is also where a tool like Overmind https://www.overmindlab.ai/?utm source=devto&utm medium=syndication&utm campaign=research-repost tends to sit. Its SDK wraps the model call interface directly, so a single overmind.init call captures every LLM invocation across OpenAI, Anthropic, Google Gemini and Agno, logging inputs, outputs, latency, token counts and errors without extra plumbing on your side. The point isn't the SDK itself. It's that capture has to be automatic and total, or the sampling and labelling stages downstream never get the raw material they need. You should not try to label every run. Most of what an agent does in a given week is unremarkable, and reviewing all of it teaches a labelling team nothing it didn't already know. The job at this stage is picking which runs are worth a human's attention. Three strategies cover most cases: | Sampling strategy | What it gives you | When to use it | |---|---|---| | Random | An honest baseline of what the agent does on an ordinary day | Every cycle, as the control slice you compare everything else against | | Stratified | Deliberate coverage of cases you already care about, such as a specific refund reason, a customer tier, or a tool that keeps timing out | When a known segment matters more than the average run | | Failure-weighted | The most signal per run, because the runs that went wrong carry the most information | When you have an error flag or a satisfaction score to sort on | Out of the refund agent's 50,000 weekly runs, a reasonable pull is around 500: a random slice plus every run that hit an error or a low satisfaction score. Whatever strategy you pick, run it on a cadence, weekly to start, rather than treating it as a one-off export you remember to do after something breaks in production. This is the step almost everyone skips, and it's the one that decides whether the dataset is worth anything. The instinct is to label by gut: skim a run, decide it feels fine, move to the next one. That works at a hundred runs and falls apart completely at ten thousand, mostly because "feels fine" means something slightly different to every reviewer and drifts over time even for the same reviewer. The alternative is writing down, explicitly, what good looks like, and scoring every sampled run against that spec. You're not starting from nothing here. Your agent's own codebase already encodes most of what it's supposed to do: the outputs it should produce, the tools it's allowed to call, the checks it runs before acting, the paths it should and shouldn't take. Read the code first and most of the labelling criteria fall directly out of it. For the refund agent, that read produces a checklist along these lines: | Criterion | What it requires | Example violation | |---|---|---| | Refund within terms | Amount within the order value and the 30-day window | Refunded a 90-day-old order | | No invented terms | Only cites the published refund terms | Quoted a returns rule that does not exist | | Escalate disputes | Hands chargebacks off to a human | Auto-refunded a disputed charge | Automating this scoring step is where LLM-as-judge techniques have become the practical default, since manually reviewing thousands of runs against a rubric doesn't scale. The approach has real limits worth knowing before you lean on it: research comparing LLM judges against human-labelled relevance data found strong rank correlation but only fair agreement on exact labels https://eugeneyan.com/writing/llm-evaluators/ , and accuracy drops sharply on the more nuanced categories rather than the easy pass/fail calls. A survey of LLM-as-judge methods https://arxiv.org/html/2412.05579v2 notes it emerged specifically because manually assessing helpfulness in training data got too expensive to do at scale by hand, which is exactly the tradeoff a labelling pipeline is making. Overmind runs this scoring as evaluators against a rubric you write. Six evaluator kinds cover it, from a deterministic check to an LLM judge, and every run is scored against a baseline before the result counts. The runs that fail a criterion, whatever the customer clicked afterward, are the highest-value training data you have. The labels encode your judgment about what good looks like, and that judgment is the one thing no generic tool can supply for you. Do the first pass yourself for the first month. It's the fastest way to find out what your criteria actually are, as opposed to what you assumed they were when you wrote the checklist. Labelled runs aren't training examples yet. The last step shapes them for whatever method you're about to run, and the method decides the shape. | Supervised fine-tuning SFT | Reinforcement learning RL | | |---|---|---| | Which runs you keep | Runs that passed every criterion | Sampled runs with their criterion scores attached | | What a row holds | An input-output pair of the behaviour you want repeated | The run plus the scores, used as a reward signal | | What the model learns | To imitate a fixed set of good examples | To produce runs that score higher | | What labelling has to produce | A pass/fail verdict per criterion | A usable score per criterion, not just pass/fail | Either way, the format needs to be consistent and machine-readable, with the goal and outcome attached to every run. This is also the stage where the case for smaller, specialised models gets concrete. Overmind's own research argues that every model invocation inside an agentic workflow is a natural source of high-quality training data https://www.overmindlab.ai/research/think-smaller?utm source=devto&utm medium=syndication&utm campaign=research-repost , precisely because the prompts are narrow and well-defined and the pass/fail signal is clean, unlike open-ended chat data. A team that instruments its model calls, clusters the resulting patterns, and fine-tunes a specialist model on them ends up with a system that improves with every production run instead of one that's frozen at whatever a general-purpose model happened to learn at pretraining time. Which method you feed, supervised fine-tuning or RL, is its own separate decision https://www.overmindlab.ai/research/types-of-fine-tuning?utm source=devto&utm medium=syndication&utm campaign=research-repost . The dataset from steps 1 through 3 is what feeds either one. None of these four steps is exotic on its own. What makes the whole thing difficult is that it never stops. Production keeps producing new traces, your criteria keep getting sharper as you find edge cases you didn't anticipate, and the dataset has to be rebuilt against what users actually did this week, not what they did last quarter. The dotted feedback line on that diagram at the top is the entire job. It's also exactly the gap most observability tooling leaves open. Datadog's own writeup on GenAI tracing gets at this directly: teams are encouraged to promote interesting production traces into curated, version-controlled "golden" datasets https://www.datadoghq.com/blog/llm-otel-semantic-convention/ and layer evaluation metadata on top, which is essentially this same capture-to-label pipeline described from the observability side. An observability platform hands you the traces and stops there. Stitching production traces to behavioural training data to a deployed, improved model is work that mostly happens in spreadsheets and one-off scripts today, and it's the specific gap platforms like Overmind are built to close, running the optimise-evaluate-accept loop end to end instead of leaving it as a manual export. Owning that labelled dataset matters because differentiation in agentic AI increasingly lives in proprietary behavioural data, not in which foundation model you call. Anyone can capture traces. The labelled dataset built from them, tuned to your own definition of correct, is the thing actually worth owning. Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own. Get started.