How Adaptive Tail Sampling Works in the OpenTelemetry Collector

A technical deep dive into Honeycomb's adaptive tail sampling processor for the OpenTelemetry Collector: how decisions get made, the samplers available, how thresholds compose with the rest of a sampling pipeline, performance benchmarks, deployment limitations, and how it compares to Refinery.

By: Mike Goldsmith

The Engineer’s Guide to Sampling in the Age of AI

Read Now

You're producing more trace data than you want to pay to store, so you sample. A fixed 1-in-100 rate cuts your bill, but it's blind. It keeps 1% of your errors, 1% of the requests to that rarely-hit route, and 1% of the health checks, all at the same rate. The noisy traffic you care about least dominates what you keep while the traces you need during an incident are the ones most likely to be gone.

We recently announced the adaptive_tail_sampling processor, Honeycomb's contribution of the adaptive sampling algorithms behind Refinery and dynsampler-go to the OpenTelemetry Collector. I built the processor and maintain it upstream with my colleague Yingrong Zhao. The announcement covers why we did it and how to try it. In this post, we'll go over the decision mechanics, the samplers, and how it composes with the rest of a sampling pipeline.

One thing worth saying up front: this is not Refinery ported into the Collector. It's the same algorithm family re-expressed in OpenTelemetry-native terms. Rules are OTTL expressions validated at startup. Sampling decisions are encoded as a threshold in W3C TraceState (ot=th) following the OpenTelemetry consistent probability sampling specification, not a vendor field. It composes with SDK head samplers and other Collector samplers. Anything that understands the specification can reweight the sampled data, whichever backend you send it to.

Three ways to sample traces #

The announcement blog introduced the three approaches available in the Collector today. It's worth being precise about their semantics, because the differences are what motivated a new processor.

Probabilistic (head) sampling decides at trace start. It's the cheapest option, no buffering at all, but it can't see what it's throwing away. Errors and rare traffic are kept at the same rate as everything else. The numbers get stark for rare events. Say a payment failure occurs once in every 1,000 requests and you head sample at 1%. Each failure's trace survives with the same 1% probability as everything else, so on average, you keep one failure trace per 100,000 requests. A service handling 50,000 requests a day hits that failure around 50 times a day, but keeps evidence of it once every two days. The sampler can't favor the failure because nothing has failed yet when the decision is made.

Tail sampling buffers the whole trace and decides with full knowledge, using the tail_sampling processor. Its policies compose as an OR. Decisions are plain keep/drop with no sampling probability attached, so kept data generally can't be reweighted downstream and counts computed from sampled data no longer reflect real traffic. The OR composition also shuts out adaptive sampling. A policy that doesn't keep a trace returns no decision rather than a drop, so the trace falls through to the next policy in the list and any later policy can still keep it. An adaptive sampler under those semantics has no control over the probability it's supposed to be enforcing.

Adaptive tail sampling, this processor, combines tail-based buffering with adaptive samplers that always know the probability they're applying. Rules evaluate first match rather than OR-composed. Every decision carries an explicit sampling threshold encoded as ot=th. You keep the interesting tail without hand-tuning, and your counts stay statistically honest after sampling.

How it works #

Spans buffer in memory, grouped by trace ID. A trace becomes ready for a decision when a root span arrives, when trace_timeout (default 30s) expires or when the trace accumulates span_limit spans (default 10,000), whichever comes first. The span limit bounds how much memory a single giant trace can hold and decides immediately, since waiting would let the trace keep growing. For the other two triggers, the processor waits with decision_delay (default 2s) for straggler spans before evaluating. What counts as a root span is itself configurable with an OTTL expression (root_span_condition), which matters when only the server side of a cross-process trace reaches your Collector, or when a producer tags a message consumer span as the effective root.

Rules evaluate in order and the first match wins. Each rule names a sampler and a rule with no conditions acts as the catch-all. A typical config keeps every error and lets an adaptive sampler settle the rest:

processors:
  adaptive_tail_sampling:
    rules:
      - name: keep-errors
        conditions:
          - span.status.code == STATUS_CODE_ERROR
        sampler:
          type: always_sample

      - name: default
        sampler:
          type: adaptive_percentage
          goal_percentage: 10
          fingerprint_attributes:
            - resource.attributes["service.name"]
            - span.attributes["http.route"]

A quick word on terminology, because OpenTelemetry and Honeycomb describe the same decision differently. In OpenTelemetry terms, every trace carries 56 bits of randomness, either explicit in TraceState as ot=rv or taken from the least-significant 7 bytes of the trace ID, giving a value between 0 and 2^56 - 1. A sampler picks a sampling threshold in that same range and keeps any trace whose randomness is at or above it. Keep everything above the halfway point and you've kept 50%. The threshold travels with the kept spans as ot=th, so any later stage can recover the probability the trace survived with. Honeycomb expresses the same decision as a sample rate: keep 1-in-N. The dynsampler-go algorithms produce rates in that form. The processor translates between the two, converting each rate into the equivalent threshold.

Sampled traces are forwarded with the threshold in TraceState plus span attributes naming the matched rule (otelcol.processor.adaptive_tail_sampling.rule) and the event that triggered the decision ( otelcol.processor.adaptive_tail_sampling.trigger), so you can see exactly which rule kept each trace on your dashboards. A decision cache remembers recent outcomes, so late-arriving spans are stamped or dropped consistently with the rest of their trace.

The processor is also honest under pressure. When the buffer fills (num_traces, default 50k), the oldest trace is evicted with a real sampling decision rather than merely discarded. The default eviction policy runs your rules on the spans seen so far, so keep-errors keeps working under duress. A constant-time probabilistic policy is available for deployments where eviction means genuine overload. Shutdown drains the buffer rather than discarding it, so a clean restart doesn't lose data.

Choosing a sampler #

The adaptive samplers group traffic by fingerprint_attributes, scoped selectors such as resource.attributes["service.name"] and span.attributes["http.route"]. Pick attributes that classify traffic, like route, method or status code, rather than ones that identify individual requests, which would give every trace its own fingerprint and defeat the adaptation. Per fingerprint, the sampler computes a sample rate. The processor converts that rate to a threshold and compares it against the trace's randomness. Two adaptive types cover the common goals:

• adaptive_percentage targets a goal percentage of span volume (goal_percentage ). It tracks per-fingerprint traffic with an exponential moving average, so rare fingerprints are kept at high rates while chatty ones are aggressively sampled and the total converges on the goal. In our validation runs, it held 9.4-10.3% against a 10% goal at every load level we tested.
• adaptive_throughput adapts toward a volume budget instead (goal_throughput , in spans per second, enforced per collector instance). Use it when the constraint is a fixed downstream budget. The defaultema algorithm smooths traffic with the same moving average, whilealgorithm: windowed recalculates over a sliding window, reacting faster to traffic shifts at the cost of being more sensitive to short spikes.

There are also always_sample and probabilistic (a fixed fraction, the inline equivalent of the probabilistic_sampler processor) for rules that don't need to adapt. One important detail is that samplers only ever produce a rate. The rate-to-threshold comparison against trace randomness is the decision mechanism for every rule, which is what makes decisions reproducible and every kept span correctly weighted downstream.

Decisions the rest of the pipeline can trust #

Because the decision is a spec-encoded threshold rather than a bare keep/drop, the processor composes with upstream and downstream sampling stages. If an SDK head sampler or a probabilistic_sampler processor already wrote ot=th, the stricter stage wins and the surviving threshold always reflects the effective end-to-end probability.

We validated this with a probabilistic_sampler in equalizing mode at 50% upstream and a 10% adaptive_percentage goal in this processor. Roughly 10% of the original traffic survived (not 10% of the upstream's 50%). Every kept span carried the 10% threshold. With an always-keep rule downstream instead, the upstream 50% threshold was preserved untouched. In both directions, estimators reconstructed the original send volume within 1.5%.

The same property helps beyond traces. The spanmetrics connector reads ot=th to produce correctly weighted R.E.D metrics from sampled data, so your request rates and error rates stay accurate even though most spans were dropped.

A fair question is why this isn't a set of new policies inside tail_sampling. The short answer is that tail_sampling's policy contract has no way to carry a sampling probability or threshold. Its OR-composition model also breaks the accounting that adaptive samplers depend on. The README walks through the mechanics in detail. Both processors remain useful and tail_sampling users lose nothing by this being separate.

Performance #

We've benchmarked and soak-tested the processor throughout development. Yingrong and I ran a full load test of the standalone processor in a production-like two-tier deployment, with a sampling tier of two replicas at 7 vCPU and 13GiB each. A replica sustained around 100,000 spans per second on roughly 0.3 cores with typical span sizes, CPU grew in proportion to span payload size and the cardinality tests pushed past 170,000 spans per second.

Under a sustained 70x overload against a deliberately undersized buffer, eviction kept memory bounded and the process was never killed for exceeding its memory limit. The span_limit cap earned its place, cutting memory around 36% for about 34% more CPU at identical throughput on giant traces. It's performant, broadly in line with Refinery on comparable workloads. We'll continue improving it over time.

Deployment and current limitations #

Here are some things to know that will impact how you deploy the processor:

• Decisions are per-instance, so all spans of a trace must reach the same Collector. Scale out with the standard two-tierloadbalancing exporter pattern, routing by trace ID. Routing byot=rv isn't supported by theloadbalancing exporter yet, but there's anopen PR to add it as a routing key.
• Memory is bounded by trace count (num_traces) and per-trace span count (span_limit ), not bytes. It grows with span rate times the buffer window.
• Rules are fixed at startup, so changing them means a restart. Shutdown drains the buffer, so a restart doesn't lose data.
• Traces only for now. Rule conditions evaluate span context.

Getting started #

The processor is available today in beta in the Honeycomb OpenTelemetry Collector distribution, ahead of the upstream stability milestones. The announcement post has step-by-step setup for Kubernetes, Docker, and standalone configs. Upstream, the component lives in the collector-contrib repository and is working toward alpha stability, at which point it becomes part of the collector-contrib distribution.

The README has the full configuration reference, examples for common deployment patterns, and the telemetry contract for building dashboards on the processor's own metrics. Feedback and issues are very welcome on #49311.

If you've been running fixed-rate sampling and living with blind spots, or wanting tail sampling but needing accurate counts afterwards, this is built for you. I'd love to hear how it behaves on your traffic.

Why Refinery is still the leading sampling tool #

The processor brings Honeycomb's sampling algorithms to the OpenTelemetry Collector, but Refinery remains the leading sampling tool. The reasons are operational rather than algorithmic. Sampling at scale is a system you operate, not just a component you configure. Refinery has years of production hardening behind it.

• Scale without a routing tier. Refinery clusters share trace ownership across peers, so you add nodes to add capacity. The collector processor needs a two-tier topology and each instance learns rates from only its own slice of traffic.
• Live rule changes. Refinery reloads sampling rules without a restart, which matters most mid-incident when you need to tighten or loosen sampling right now.
• Overload protection. Stress relief detects saturation and switches to deterministic sampling until the cluster recovers, so it degrades predictably instead of falling over.
• Operational depth. Environment-aware multi-tenancy, rich runtime telemetry and a managed option with Refinery as a Service.

If you want adaptive sampling native to an OpenTelemetry Collector pipeline, start with the processor. If you're already running Refinery, the README includes a direct mapping from Refinery's sampler types to the processor's. When sampling becomes critical infrastructure with its own scaling, tenancy and incident-response demands, Refinery is built for exactly that.