{"slug": "i-built-an-otel-processor-that-can-save-you-12k-month-on-llm-observability", "title": "I Built an OTel Processor That Can Save You $12k/Month on LLM Observability", "summary": "A developer built TraceShrink, an OpenTelemetry Collector processor that replaces random tail sampling with cost-aware rules to cut LLM observability costs. The processor keeps traces based on cost thresholds, errors, and duration, retaining 70% of cost visibility while reducing storage by 98%. In simulations, it processed 1.7 million spans per second with near-zero overhead, potentially saving thousands of dollars monthly for large AI deployments.", "body_md": "If you're running LLM-powered services in production, you know the problem: your observability bill is exploding.\n\nEvery API call to GPT-4o, Claude, or Gemini generates traces. A typical AI gateway handles millions of spans per day. At $0.30/GB in Datadog or $0.50/million spans in Grafana Cloud, that's real money — and most of those traces are worthless $0.001 cache hits.\n\nThe standard approach is tail sampling at 1%. Your OTel Collector randomly keeps 1 in 100 traces. Problem solved?\n\nNo. Here's what random sampling loses:\n\n| Trace | Cost | Random 1% | What happens | \n|---|---|---|---|\n| GPT-4o chain-of-thought, 50K tokens | $2.50 | 99% chance dropped | You never see the expensive call | \n| Checkout flow 500 error | $0.15 | 99% chance dropped | PagerDuty fires, no trace to debug | \n| 15-second timeout on embedding | $0.08 | 99% chance dropped | User complained, you can't reproduce | \n| Health check ping | $0.001 | 99% chance dropped | Nobody cares, but you kept one | \n\nRandom sampling is **cost-blind**. It doesn't know that a $2.50 trace is 2,500x more valuable than a $0.001 health check.\n\nI built [TraceShrink](https://github.com/TimurRakhmatullin86/traceshrink) — an OTel Collector processor that replaces random sampling with cost-aware rules:\n\n```\nprocessors:\n  traceshrink:\n    cost_threshold: 0.10      # keep traces costing >= $0.10\n    keep_errors: true          # always keep error traces\n    duration_threshold: 5s     # keep slow traces\n```\n\nThat's it. One YAML block in your existing collector config.\n\n```\nOTLP Receiver → costprocessor → traceshrink → Tempo/Jaeger\n                      │                │\n                 Calculate $        Keep if:\n                 from tokens        - cost >= $0.10\n                 (2,700+ models)    - has error\n                                    - duration > 5s\n                                    Drop everything else\n```\n\n**Step 1: costprocessor** reads `gen_ai.request.model` and token counts from your spans, looks up pricing in a built-in database (LiteLLM's 2,700+ models), and writes `gen_ai.usage.cost_usd`.\n\n**Step 2: traceshrink** buffers spans by trace ID, accumulates total cost per trace, and makes a keep/drop decision once the trace is complete.\n\nI ran a simulation with 1 million spans using a realistic AI workload distribution (0.8% expensive calls, 0.2% errors, 1% slow requests):\n\n| Metric | Value | \n|---|---|\n| Input | 1,000,000 spans | \n| Output | 20,125 spans ( **2% retained** ) | \n| Storage reduction | **98%** | \n| Cost captured | **70%** of total $ cost | \n| Throughput | **1.7M spans/sec** | \n| Decision latency | **2 nanoseconds** per span | \n| Memory allocations | **0** per decision | \n\n98% fewer spans to store. 70% of the dollar cost still visible. Every error and every slow request preserved.\n\nAssume you're running an AI gateway doing 5M spans/day:\n\n|  | Before | After TraceShrink | \n|---|---|---|\n| Spans/day | 5,000,000 | 100,000 | \n| At $0.50/M spans (Grafana) | $2.50/day | $0.05/day | \n| Monthly | $75/month | $1.50/month | \n| At Datadog pricing (~$0.30/GB) | ~$400/month | ~$8/month | \n\nAt scale (50M spans/day), that's $750-4,000/month → $15-80/month. That's where the \"$12k/month\" comes from for large deployments.\n\nOTel's tail sampling processor supports OTTL (OpenTelemetry Transformation Language) conditions. You could write:\n\n```\npolicies:\n  - name: expensive\n    type: ottl_condition\n    ottl_condition:\n      span: ['attributes[\"gen_ai.usage.cost_usd\"] > 0.10']\n```\n\nTwo problems:\n\n**No aggregation.** OTTL evaluates per-span, not per-trace. You can't say \"keep this trace if the *total* cost across all its spans exceeds $X.\" A trace with 5 spans at $0.03 each ($0.15 total) gets dropped because no individual span exceeds $0.10.\n\n**Complexity.** To combine cost + error + duration, you need multiple policies, composite evaluators, and a decision strategy. TraceShrink does it in one config block.\n\nBuild a custom collector with TraceShrink:\n\n```\n# builder-config.yaml\ndist:\n  name: my-collector\n  output_path: ./dist\n\nprocessors:\n  - gomod: github.com/timurrakhmatullin86/traceshrink v0.1.0\nocb --config builder-config.yaml\n```\n\nThen add the processors to your pipeline config and deploy.\n\nThe code is Apache-2.0: [github.com/TimurRakhmatullin86/traceshrink](https://github.com/TimurRakhmatullin86/traceshrink)\n\nIf you're drowning in LLM traces, give it a try and let me know what breaks.", "url": "https://wpnews.pro/news/i-built-an-otel-processor-that-can-save-you-12k-month-on-llm-observability", "canonical_source": "https://dev.to/tim860/i-built-an-otel-processor-that-can-save-you-12kmonth-on-llm-observability-2536", "published_at": "2026-09-07 22:01:46+00:00", "updated_at": "2026-09-07 22:31:22.227034+00:00", "lang": "en", "topics": ["developer-tools", "mlops", "ai-infrastructure"], "entities": ["TraceShrink", "OpenTelemetry", "LiteLLM", "Datadog", "Grafana Cloud"], "alternates": {"html": "https://wpnews.pro/news/i-built-an-otel-processor-that-can-save-you-12k-month-on-llm-observability", "markdown": "https://wpnews.pro/news/i-built-an-otel-processor-that-can-save-you-12k-month-on-llm-observability.md", "text": "https://wpnews.pro/news/i-built-an-otel-processor-that-can-save-you-12k-month-on-llm-observability.txt", "jsonld": "https://wpnews.pro/news/i-built-an-otel-processor-that-can-save-you-12k-month-on-llm-observability.jsonld"}}