A single-binary distributed trace generator that produces realistic, topology-rich OTLP traces and correlated logs. No Docker Compose, no microservices to deploy, no infrastructure - just one executable (or a 5.7 MB container) that simulates a full e-commerce platform with up to 28 services, dozens of pods that scale with the topology, and 40 scenario flows - including AI agentic scenarios with full OTel GenAI semantic conventions. Three complexity levels (light/normal/heavy) let you scale from a clean 10-service demo to the full topology with AI.
Built for testing observability platforms, load testing trace pipelines, and showcasing distributed system visualizations - for both traditional APM and LLM observability.
Every existing trace generator falls into one of two categories:
Flat span generators(telemetrygen, tracepusher) - produce uniform, identical spans with no service topology** Full demo apps**(OTel Astronomy Shop, Jaeger HotROD) - require Docker Compose with 15+ containers and ~6GB RAM
And none of them generate AI agentic traces. The LLM observability market has no standalone tool that combines traditional APM with LLM observability. Every specialized LLM tool (Langfuse, LangSmith, Helicone, Arize, Traceloop, Portkey, Galileo) tracks token usage, model costs, and agent tool calls - but none of them provide traditional distributed tracing.
This tool generates topology-rich, failure-injectable traces from a single binary - covering both traditional microservice flows AND AI agentic patterns with OTel GenAI semantic conventions. One binary proves that a platform can visualize both.
go install github.com/ImmersiveFusion/if-opentelemetry-tracegen/cmd/tracegen@latest
tracegen -insecure
tracegen -endpoint your-otlp-endpoint:443 -headers "api-key=YOUR_KEY"
export OTEL_EXPORTER_OTLP_HEADERS="api-key=YOUR_KEY"
tracegen -endpoint your-otlp-endpoint:443
See it in 3D- Send traces to[IAPM](tracegen -endpoint otlp.iapm.app:443 -headers "api-key=YOUR_KEY"
) to explore them as a 3D force-directed graph, drill into conventional trace waterfalls for detailed analysis, and get AI-assisted insights from[Tessa]. For a ready-made example without any setup, try the[OpenTelemetry Chaos Simulator]at[demo.iapm.app]- a fully interactive sandbox with visual failure injection.
Seven always-on demo grids stream live OpenTelemetry traces into IAPM's 3D player right now — a clean baseline, an AI-native app, a blended environment, phantom-service detection, an AI-outage, and a full incident. Each grid is this container, deployed declaratively via GitOps (Argo CD) in the Immersive Fusion cloud — multi-arch and distroless, one matrix row per grid, shipping to otlp.iapm.app:443
.
See them in 3D: the full experience is the IAPM 3D client — install it and open a grid to walk the live traces. On mobile or can't install right now? IAPM Web runs the same grids in your browser at portal.iapm.app.
** Where else does TraceGen run? →** — a community board of deployments. Add yours.
| Service | Pods | Role |
|---|---|---|
| web-frontend | 2 | Browser client, SPA |
| api-gateway | 3 | HTTP routing, auth |
| order-service | 3 | Order lifecycle |
| payment-service | 2 | Stripe integration |
| inventory-service | 2 | Stock management |
| notification-service | 2 | Event-driven notifications |
| user-service | 2 | Auth, profiles |
| cache-service | 3 | Redis cluster |
| search-service | 2 | Elasticsearch queries |
| scheduler-service | 1 | Cron jobs (singleton) |
| auth-service | 3 | JWT, webhook verification |
| recommendation-service | 2 | ML-based recommendations |
| cart-service | 2 | Shopping cart |
| product-service | 3 | Product catalog |
| shipping-service | 2 | Rates, labels, tracking |
| fraud-service | 2 | ML fraud scoring |
| email-service | 2 | SMTP relay (SendGrid) |
| tax-service | 1 | Tax calculation |
| analytics-service | 3 | Event tracking (Kafka) |
| config-service | 1 | Feature flags |
| Service | Pods | Role |
|---|---|---|
| llm-gateway | 3 | OpenAI API routing, token tracking |
| embedding-service | 2 | Text-to-vector operations |
| vector-db-service | 2 | Qdrant similarity search |
| ai-agent-service | 2 | Agent orchestration (plan/act/reflect) |
| content-moderation-service | 2 | Safety classifiers, PII detection |
| model-registry-service | 1 | Model versioning (singleton) |
| feature-store-service | 2 | ML feature serving |
| data-pipeline-service | 2 | Batch embedding, retraining |
All 59 pods are distributed across 5 AKS VMSS nodes (2 node pools) with realistic service.instance.id
and host.name
resource attributes.
| Scenario | Graph Shape | Key Pattern |
|---|---|---|
| Create Order | ||
| Long chain (8 services, 14+ spans) | Producer/consumer with queue delays | |
| Search & Browse | ||
| Linear with cache | Elasticsearch + Redis | |
| User Login | ||
| Branching (success/failure) | Auth with session creation | |
| Failed Payment | ||
| Error chain | Stripe 402 + error propagation | |
| Bulk Notifications | ||
| Fan-out (3-5 parallel) | Batch email processing | |
| Health Check | ||
| Star topology (6 parallel) | Concurrent health pings | |
| Inventory Sync | ||
| Fan-out + reindex | Parallel cache warming | |
| Scheduled Report | ||
| Headless chain (no UI) | Cron job entry point | |
| Stripe Webhook | ||
| Headless chain (no gateway) | External callback entry | |
| Recommendations | ||
| Scatter-gather / bowtie | Fan-out to 3, gather, cache | |
| Add to Cart | ||
| Cross-service with feature flags | Config service + analytics | |
| Full Checkout | ||
| Monster chain (15 services) | Tax+shipping parallel, fraud ML | |
| Shipping Update | ||
| Carrier webhook (headless) | External webhook + email relay | |
| Saga Compensation | ||
| Forward chain + 4-way compensation fan-out | Payment retries + rollback | |
| Timeout Cascade | ||
| Branching with circuit breaker | Stale cache fallback | |
| User Registration | ||
| Linear with async branch | Email verification token, duplicate detection | |
| Product Review | ||
| Write + async moderation | Optimistic write + background processing | |
| Return/Refund | ||
| Parallel reverse flow (16-18 spans) | Parallel refund + restock, reverse money flow | |
| Wishlist + Price Alert | ||
| Write-through with async | Write-through cache, async price monitoring | |
| Coupon Application | ||
| Validation chain | Cart recalculation, validation branch | |
| Gift Card Purchase | ||
| Payment splitting | Balance check, payment splitting | |
| Subscription Management | ||
| Webhook-driven lifecycle | Stripe subscription, renewal webhook | |
| A/B Test Exposure | ||
| Feature flag branch | Variant assignment, sticky session | |
| Rate Limiting | ||
| Early termination (4-6 spans) | Redis sliding window, 429 response | |
| Admin Product CRUD | ||
| Write-amplification fan-out | Cache + search reindex on write | |
| Order History | ||
| Paginated read | Keyset pagination, cursor-based | |
| Support Ticket | ||
| Cross-domain trace | SLA assignment, team routing | |
| Multi-Currency Checkout | ||
| External API chain | FX rate API, cache hit ratio |
| Scenario | Graph Shape | Key Pattern |
|---|---|---|
| Semantic Search (RAG) | ||
| Linear with 2 LLM calls (14-16 spans) | Embedding + vector search + LLM reranking | |
| AI Chatbot with Tool Use | ||
| Double bowtie (18-22 spans) | Plan -> fan-out tool calls -> synthesize | |
| AI Content Moderation | ||
| Parallel classifiers + 3-way branch (12-16 spans) | Safety/spam scoring, guardrail decisions | |
| Multi-Step Agent | ||
| Iterative loop (28-40 spans) | Plan -> act -> reflect cycle (3-5 iterations) | |
| AI Customer Support | ||
| Branching with escalation (16-20 spans) | Sentiment classification, intent detection | |
| AI Content Generation | ||
| Linear with safety filter (12-15 spans) | Temperature-controlled generation, content safety | |
| Embedding Pipeline | ||
| High fan-out batch (25-40 spans) | Batch chunking, parallel embedding, vector upsert | |
| Dynamic Pricing Agent | ||
| Headless agent (14-18 spans) | Feature store lookup, autonomous price updates | |
| Fraud with Explainability | ||
| Linear with LLM explanation (10-12 spans) | SHAP-style feature attribution via LLM | |
| Inventory Reorder Agent | ||
| Autonomous agent (16-20 spans) | Demand forecast, autonomous purchase orders | |
| Model Retraining Pipeline | ||
| Batch pipeline (14-18 spans) | ML training spans, model registry, quality gate | |
| Conversational Commerce | ||
| Multi-turn session (10-14 spans/turn) | Growing context tokens, session continuity |
Note:Failed Payment, Saga Compensation, Timeout Cascade, lost messages, and retry storms only activate when-errors > 0
. AI error scenarios (rate limits, hallucinated tool calls, token budget exceeded, content filter blocks) also require-errors > 0
.
Every service emits OTel log records via OTLP alongside traces. Logs are automatically correlated with the active span context (trace_id, span_id), so your APM platform can link logs to the exact span that produced them.
ERROR logs are emitted alongside every exception event (cache failures, DB errors, payment declines, LLM rate limits, agent failures)WARN logs fire on auth failures, content moderation flags, payment retries, and LLM fallbacksINFO logs cover request entry points, payment processing, fraud analysis, agent invocations, and iteration progress
Disable with -no-logs
to emit traces only.
All AI scenarios emit spans following OTel GenAI Semantic Conventions and matching the exact span shapes produced by Microsoft Semantic Kernel and Microsoft Agent Framework.
Span types:
| Span Name Pattern | SpanKind | Example |
|---|---|---|
chat {model} |
||
| CLIENT | chat gpt-4o |
|
embedding {model} |
||
| CLIENT | embedding text-embedding-3-small |
|
invoke_agent {name} |
||
| CLIENT | invoke_agent CustomerSupportAgent |
|
execute_tool {name} |
||
| INTERNAL | execute_tool get_order_status |
|
{operation} {collection} |
||
| CLIENT | query product-embeddings |
Attributes on every LLM span:
gen_ai.system
- LLM provider (e.g.,
openai
)gen_ai.request.model
/gen_ai.response.model
- model requested and used
gen_ai.usage.input_tokens
/gen_ai.usage.output_tokens
-
token consumption
gen_ai.response.finish_reasons -
completion reason (
stop
,tool_calls
,length
,content_filter
)gen_ai.response.id
- unique response identifier
gen_ai.request.temperature
,gen_ai.request.max_tokens
- request parameters
Agent-specific attributes:
gen_ai.agent.id
/gen_ai.agent.name
/gen_ai.agent.description
-
agent identity
gen_ai.conversation.id -
session linking for multi-turn interactions
gen_ai.tool.name
/gen_ai.tool.type
/gen_ai.tool.call.id
-
tool call tracking
gen_ai.data_source.id -
RAG data source identifier
gen_ai.request.embedding.dimensions -
embedding dimensions
These attributes match what every LLM observability tool on the market tracks - enabling direct comparison of visualization capabilities.
| Feature | Description |
|---|---|
| Lost messages | |
| 5% chance per queue hop that the consumer never fires - trace ends abruptly | |
| Dead consumer mode | |
-no-consumers flag: producers fire, consumers never pick up |
|
| Retry storms | |
| Payment retries 3x with exponential backoff before saga compensation | |
| Timeout cascades | |
| Search service times out, gateway returns 504, circuit breaker serves stale cache | |
| Saga compensation | |
| Payment fails after order+inventory committed - triggers 4-way parallel rollback | |
| LLM rate limits | |
| OpenAI 429 with token budget details, fallback to text search | |
| Hallucinated tool calls | |
| Agent requests non-existent tool, triggers error handling | |
| Token budget exceeded | |
| Agent exceeds iteration token limit, graceful degradation | |
| Content filter blocks | |
| Safety classifier blocks content, alternate flow triggered | |
| Tunable error rate | |
-errors 0 (none) to -errors 10 (chaos) with realistic .NET stack traces |
The generated traces simulate a .NET-based e-commerce platform with AI capabilities. Stack traces and library names reflect the .NET ecosystem by design.
Stack traces: Npgsql, StackExchange.Redis, Stripe SDK, Elasticsearch.Net, System.Net.Http, OpenAI SDK, Qdrant client** Database operations**: PostgreSQL INSERT/SELECT/UPDATE with semantic conventions** Cache operations**: Redis GET/SET/HSET/MSET/DEL with TTL and key attributes** Messaging**: RabbitMQ and Kafka with producer/consumer span kinds and queue delays** External APIs**: Stripe charges, SendGrid email, UPS shipping, OpenAI chat/embeddings** LLM operations**: Chat completions, embeddings, agent tool calls with token tracking** Vector search**: Qdrant similarity search with cosine distance, dimension validation** Agent orchestration**: Plan/act/reflect loops, tool dispatch, session management** Content moderation**: Safety classifiers, PII detection, guardrail enforcement** ML inference**: Fraud detection model scoring with feature counts** Feature flags**: Config service checks that gate behavior
tracegen [flags]
Flags:
-endpoint string OTLP gRPC endpoint host:port (default "localhost:4317")
-headers string OTLP headers as key=value pairs, comma-separated (or set OTEL_EXPORTER_OTLP_HEADERS)
-complexity string Topology complexity: light, normal, heavy (default "normal")
-level int Aggressiveness 1-10 (default 1)
-errors int Error rate 0-10 (default 0)
-no-consumers Disable all async consumers
-no-ai-backends Disable LLM/AI backends (AI spans emit errors)
-ai-only Only run AI agentic scenarios
-no-logs Disable OTel log record emission (traces only)
-insecure Use plaintext gRPC (no TLS) for local backends
-log-level string Console verbosity: silent, error, info, debug (default "info")
-quiet Errors only (alias for -log-level=error); silences the periodic "traces sent" heartbeat
Console verbosity.silent
/error
suppress the periodic heartbeat. The startup banner ("what it's doing") and genuine errors/fatal exits always print to stderr at any level. You can also set it with theTRACEGEN_LOG_LEVEL
env var (handy for containers). Precedence:-quiet
-log-level
TRACEGEN_LOG_LEVEL
default (
info
). The published container image defaults toTRACEGEN_LOG_LEVEL=error
so it doesn't spam logs; the bare CLI staysinfo
. Override with-e TRACEGEN_LOG_LEVEL=info
(or-log-level
/-quiet
).
| Complexity | Services | Pods | Scenarios | Best for |
|---|---|---|---|---|
| light | ||||
| 10 core | ~20 (min replicas) | 6 | Clean demos, small graphs | |
| normal | ||||
| 20 traditional | ~40 | 16 | General testing, full e-commerce | |
| heavy | ||||
| 28 (+ AI) | 59 | 20 (of 40 defined) | Full topology with AI agentic flows |
Light includes only the e-commerce backbone: web-frontend, api-gateway, order-service, payment-service, inventory-service, user-service, cache-service, auth-service, product-service, and cart-service. Scenarios are limited to the core flows (Create Order, Search & Browse, User Login, Add to Cart, Full Checkout, Health Check).
Normal (default) adds all remaining traditional services and scenarios including chaos/failure modes.
Heavy adds all 8 AI services and 4 of the 12 AI agentic scenarios (RAG Search, AI Chatbot, Content Moderation, Multi-Step Agent).
| Level | Label | Rate |
|---|---|---|
| 1 | whisper | ~2 traces/s |
| 2 | gentle | ~3 traces/s |
| 3 | calm | ~3 traces/s |
| 4 | moderate | ~5 traces/s |
| 5 | steady | ~7 traces/s |
| 6 | brisk | ~15 traces/s |
| 7 | aggressive | ~21 traces/s |
| 8 | intense | ~40 traces/s |
| 9 | firehose | ~83 traces/s |
| 10 | SCREAM | ~350 traces/s |
tracegen -insecure
tracegen -complexity light -level 1 -insecure
tracegen -level 1 -insecure
tracegen -complexity heavy -level 3 -insecure
tracegen -level 5 -errors 5 -insecure
tracegen -level 3 -no-consumers -insecure
tracegen -level 3 -ai-only -insecure
tracegen -level 5 -no-ai-backends -errors 5 -insecure
tracegen -level 10 -errors 10 -insecure
tracegen -endpoint otlp.example.com:443 -headers "api-key=YOUR_KEY"
export OTEL_EXPORTER_OTLP_HEADERS="api-key=SECRET,x-team=platform"
tracegen -endpoint otlp.example.com:443
tracegen -endpoint otlp.iapm.app:443 -headers "API-Key=YOUR_IAPM_KEY"
| Capability | tracegen | OTel telemetrygen | OTel Astronomy Shop | Jaeger HotROD | k6 + xk6-tracing |
|---|---|---|---|---|---|
| Single binary, zero infra | Yes | ||||
| 1 binary | 15+ containers, ~6GB | 4 containers | k6 + extension | ||
| Services | 28 | ||||
| 1 | ~22 | 4 | User-defined | ||
| Pod instances | 59 | ||||
| 0 | 1/svc | 0 | 0 | ||
| Scenario flows | 40 | ||||
| 0 | ~10 | 1 | User-defined | ||
| AI agentic scenarios | 12 | ||||
| No | No | No | No | ||
| OTel GenAI conventions | Yes | ||||
| No | No | No | No | ||
| Agent tool call traces | Yes | ||||
| No | No | No | No | ||
| RAG pipeline traces | Yes | ||||
| No | No | No | No | ||
| Diamond dependencies | Yes | ||||
| No | Implicit | No | No | ||
| Scatter-gather | Yes | ||||
| No | No | No | No | ||
| Lost messages | Yes | ||||
| No | No | No | No | ||
| Dead consumer mode | Yes | ||||
| No | No | No | No | ||
| Saga compensation | Yes | ||||
| No | No | No | No | ||
| Retry storms | Yes | ||||
| No | No | No | No | ||
| Timeout cascade | Yes | ||||
| No | No | No | No | ||
| LLM failure injection | Yes | ||||
| No | No | No | No | ||
| Tunable error rate | 0-10 | ||||
| No | Fixed | No | No | ||
| Tunable throughput | 2-350/s | ||||
| Rate flag | Locust | Fixed | k6 VUs | ||
| Headless flows (webhook/cron) | 3 | ||||
| No | No | No | No | ||
| Startup time | <1s | ||||
| <1s | 3-5 min | 30s | <5s |
Works with any OTLP gRPC-compatible backend:
IAPM(3D visualization)- Jaeger
- Grafana Tempo
- Honeycomb
- New Relic
- Datadog (with OTLP endpoint)
- Splunk Observability
- Elastic APM
- Any OpenTelemetry Collector
The AI agentic traces are also compatible with LLM-specialized observability tools that accept OTel input:
-
Langfuse (OTel-native since SDK v3)
-
Arize Phoenix (OTel instrumentation)
-
Traceloop / OpenLLMetry (built on OTel)
-
Interactive chaos engineering sandbox with visual failure injection. Complements tracegen: generate topology-rich traces here, inject chaos there,OpenTelemetry Chaos Simulatorvisualize both in 3D.
git clone https://github.com/ImmersiveFusion/if-opentelemetry-tracegen.git
cd if-opentelemetry-tracegen
go build -o tracegen ./cmd/tracegen
GOOS=linux GOARCH=amd64 go build -o tracegen ./cmd/tracegen
GOOS=darwin GOARCH=arm64 go build -o tracegen ./cmd/tracegen
GOOS=windows GOARCH=amd64 go build -o tracegen.exe ./cmd/tracegen
The LLM observability market is growing rapidly, but every specialized tool focuses exclusively on LLM workloads. No standalone LLM observability tool provides traditional APM capabilities. The only platforms addressing both are legacy APM giants (Datadog, New Relic, Dynatrace) adding LLM features to existing products.
This trace generator produces both traditional distributed traces AND AI agentic traces from the same binary - proving that a single platform can visualize both. The AI scenarios emit the exact same telemetry signals that Langfuse, LangSmith, Helicone, Arize, Traceloop, Portkey, and Galileo track.
Microsoft's Semantic Kernel and Agent Framework are the most widely adopted .NET AI frameworks. Their OTel instrumentation emits exactly three span types: invoke_agent {name}
, chat {model}
, and execute_tool {function}
. Our AI scenarios produce traces structurally identical to what a real Semantic Kernel / Agent Framework application would emit - so observability platforms can be tested against realistic .NET AI workloads.
The OTel GenAI Semantic Conventions are being adopted across the ecosystem. Langfuse SDK v3 is OTel-native, LangSmith added OTel support, Arize Phoenix uses OTel instrumentation, and Traceloop's OpenLLMetry conventions were adopted into the official OTel spec. Building on these conventions ensures the generated traces are compatible with every tool that adopts the standard.
| Source | Decision Informed |
|---|---|
OTel GenAI Agent SpansOTel GenAI Attribute Registrygen_ai.*
attribute list with typesMS Semantic Kernel Observabilitygen_ai
attribute usageMS Agent Framework Observabilityinvoke_agent
, chat
, execute_tool
Langfuse OTel IntegrationTraceloop OpenLLMetryEmail | LinkedIn | Discord | GitHub | Bluesky | Twitter/X | YouTube | Twitch
Try IAPM for your own projects.
Apache License 2.0 - see LICENSE for details.
Copyright 2026 ImmersiveFusion