{"slug": "daytona-sandbox-monitoring-tracing-with-opentelemetry", "title": "Daytona Sandbox Monitoring & Tracing with OpenTelemetry", "summary": "Daytona, the AI code execution platform, has published a guide for monitoring AI-generated code running in its sandboxes using OpenTelemetry and SigNoz, detailing four non-overlapping telemetry paths: in-sandbox app instrumentation, Daytona's built-in per-sandbox CPU/memory/disk metrics, organization quota metrics, and SDK tracing for lifecycle calls. The guide instructs users to configure a domain allow list and environment variables to route telemetry to SigNoz, with prerequisites including Python 3.10+, a Daytona API key with write:sandboxes permission, and a plan that allows per-sandbox network rules.", "body_md": "## Overview\n\n[Daytona](https://www.daytona.io/) runs AI-generated code in isolated sandboxes. Your agent creates a sandbox, executes code in it, reads files back, and tears it down. Two things go wrong in that loop, and each needs its own telemetry: the work inside the sandbox fails or hangs, and the sandbox lifecycle itself gets slow or errors out.\n\n## Prerequisites\n\n- An instance of SigNoz (either [Cloud](https://signoz.io/teams/) or[Self-Hosted](https://signoz.io/docs/install/self-host/) )\n- A Daytona account and an API key with `write:sandboxes` permission\n- Python 3.10 or later, with the `daytona` SDK installed. The SDK requires 3.10.\n- A Daytona plan that allows per-sandbox network rules. Tier 1 and Tier 2 accounts cannot override the default egress policy, which blocks SigNoz. See [Network Limits](https://www.daytona.io/docs/en/network-limits/) .\n\n## How it works\n\nDaytona exposes four OpenTelemetry paths. They do not overlap, so pick the ones that answer your question.\n\n| Path | What you get | What it costs you | \n|---|---|---|\n| App instrumentation inside the sandbox | Spans, logs, and custom metrics from the code your agent runs | Code you write and ship into the sandbox | \n| Daytona's built-in telemetry | Per-sandbox CPU, memory, and disk, plus toolbox API spans | One dashboard setting | \n| Organization quota metrics | CPU, memory, storage, and GPU used against your org quota | The same dashboard setting | \n| SDK tracing | Latency and failures for `create` ,`start` ,`stop` ,`delete` , file, and process calls | One flag on the client | \n\nStart with the first path. It shows what your agent did inside the sandbox, which nothing else reconstructs. The rest are collapsed below; expand them when you need them. All four use [OpenTelemetry](https://opentelemetry.io/) and export to the same SigNoz endpoint.\n\n## Monitor Work Inside a Daytona Sandbox\n\nThis path shows what your agent did: which steps ran, how long each took, what it logged, and where it failed.\n\n### Step 1: Allow SigNoz through the sandbox firewall\n\nSandbox egress is deny-by-default. Package registries stay reachable and Daytona blocks the rest, so an OTLP exporter fails with a TLS reset until you allow the ingestion host. Pass `domain_allow_list` when you create the sandbox:\n\n```\nrun_agent.py\npython\nimport asyncio\n \nfrom daytona import AsyncDaytona, DaytonaConfig, CreateSandboxFromSnapshotParams\n \n \nasync def main():\n    async with AsyncDaytona(DaytonaConfig(api_key=\"<your-daytona-api-key>\")) as daytona:\n        sandbox = await daytona.create(\n            CreateSandboxFromSnapshotParams(\n                domain_allow_list=\"ingest.<region>.signoz.cloud,pypi.org,files.pythonhosted.org,*.daytona.io\",\n                env_vars={\n                    \"OTEL_EXPORTER_OTLP_ENDPOINT\": \"https://ingest.<region>.signoz.cloud:443\",\n                    \"OTEL_EXPORTER_OTLP_HEADERS\": \"signoz-ingestion-key=<your-ingestion-key>\",\n                    \"OTEL_SERVICE_NAME\": \"<your-service-name>\",\n                },\n            )\n        )\n        print(\"created\", sandbox.id)\n \n \nasyncio.run(main())\n```\n\n**Verify these values:**\n\n- `<region>` : Your[SigNoz Cloud region](https://signoz.io/docs/ingestion/signoz-cloud/overview/#endpoint) .\n- `<your-ingestion-key>` : Your SigNoz[ingestion key](https://signoz.io/docs/ingestion/signoz-cloud/keys/) .\n- `<your-daytona-api-key>` : Created under Daytona Dashboard, Keys.\n- `<your-service-name>` : What your sandbox telemetry appears under in SigNoz, and what you filter on in[Validate](#validate) .\n\nThe allow list takes at most 20 comma-separated domains and supports wildcards such as `*.example.com`. It is mutually exclusive with `networkBlockAll` and `networkAllowList`, so set only one of the three per sandbox. Setting a list also drops the default allowances, so include the registries your code installs from.\n\nThe `env_vars` values reach every process in the sandbox, so the OpenTelemetry SDK picks up the endpoint and headers with no further configuration.\n\n### Step 2: Instrument the code that runs inside the sandbox\n\nThis is ordinary OpenTelemetry setup apart from the resource attributes, which read environment variables Daytona injects into every sandbox. Use them to scope telemetry to one sandbox later. The service name needs no line of its own, because `Resource.create()` reads `OTEL_SERVICE_NAME` from the environment you set in Step 1.\n\n```\nsandbox_app.py\npython\nimport logging\nimport os\n \nfrom opentelemetry import metrics, trace\nfrom opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter\nfrom opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter\nfrom opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter\nfrom opentelemetry.sdk._logs import LoggerProvider, LoggingHandler\nfrom opentelemetry.sdk._logs.export import BatchLogRecordProcessor\nfrom opentelemetry.sdk.metrics import MeterProvider\nfrom opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader\nfrom opentelemetry.sdk.resources import Resource\nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor\n \n# service.name is not set here: Resource.create() runs OpenTelemetry's\n# environment detector, which picks up the OTEL_SERVICE_NAME set in Step 1.\nresource = Resource.create(\n    {\n        \"daytona.sandbox.id\": os.environ.get(\"DAYTONA_SANDBOX_ID\", \"\"),\n        \"daytona.organization.id\": os.environ.get(\"DAYTONA_ORGANIZATION_ID\", \"\"),\n        \"daytona.region.id\": os.environ.get(\"DAYTONA_REGION_ID\", \"\"),\n        \"daytona.snapshot\": os.environ.get(\"DAYTONA_SANDBOX_SNAPSHOT\", \"\"),\n    }\n)\n \ntracer_provider = TracerProvider(resource=resource)\ntracer_provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))\ntrace.set_tracer_provider(tracer_provider)\n \nlogger_provider = LoggerProvider(resource=resource)\nlogger_provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter()))\nlogging.getLogger().addHandler(LoggingHandler(logger_provider=logger_provider))\nlogging.getLogger().setLevel(logging.INFO)\n \nreader = PeriodicExportingMetricReader(OTLPMetricExporter())\nmeter_provider = MeterProvider(resource=resource, metric_readers=[reader])\nmetrics.set_meter_provider(meter_provider)\n \ntracer = trace.get_tracer(\"agent-work\")\nmeter = metrics.get_meter(\"agent-work\")\ntasks = meter.create_counter(\"sandbox.tasks.completed\")\nlog = logging.getLogger(\"sandbox-app\")\n \nwith tracer.start_as_current_span(\"agent.task\") as span:\n    span.set_attribute(\"task.kind\", \"code-execution\")\n    log.info(\"starting work\")\n    with tracer.start_as_current_span(\"agent.step.compute\"):\n        total = sum(i * i for i in range(200000))\n    log.info(\"compute finished with total %s\", total)\n    tasks.add(1, {\"task.kind\": \"code-execution\"})\n \ntracer_provider.force_flush()\nlogger_provider.force_flush()\nmeter_provider.force_flush()\n```\n\nFlush all three providers before the process exits. Sandbox processes are short-lived, and an unflushed batch dies with the process.\n\n### Step 3: Upload the script, install the SDK, and run it\n\n```\nrun_agent.py\npython\nimport asyncio\n \nfrom daytona import AsyncDaytona, DaytonaConfig, CreateSandboxFromSnapshotParams\n \n \nasync def main():\n    async with AsyncDaytona(DaytonaConfig(api_key=\"<your-daytona-api-key>\")) as daytona:\n        sandbox = await daytona.create(\n            CreateSandboxFromSnapshotParams(\n                domain_allow_list=\"ingest.<region>.signoz.cloud,pypi.org,files.pythonhosted.org,*.daytona.io\",\n                env_vars={\n                    \"OTEL_EXPORTER_OTLP_ENDPOINT\": \"https://ingest.<region>.signoz.cloud:443\",\n                    \"OTEL_EXPORTER_OTLP_HEADERS\": \"signoz-ingestion-key=<your-ingestion-key>\",\n                    \"OTEL_SERVICE_NAME\": \"<your-service-name>\",\n                },\n            )\n        )\n \n        with open(\"sandbox_app.py\", \"rb\") as f:\n            await sandbox.fs.upload_file(f.read(), \"/home/daytona/sandbox_app.py\")\n \n        await sandbox.process.exec(\n            \"pip install --quiet opentelemetry-sdk opentelemetry-exporter-otlp-proto-http\"\n        )\n \n        result = await sandbox.process.exec(\"python3 /home/daytona/sandbox_app.py\")\n        print(result.exit_code, result.result)\n \n \nasyncio.run(main())\n```\n\nTo skip the install on every run, bake the OpenTelemetry packages into a [snapshot](https://www.daytona.io/docs/en/snapshots/) and create sandboxes from it.\n\n## ## Trace Daytona SDK Operations\n\nSDK tracing covers the other half: how long Daytona itself takes to create, start, stop, and delete sandboxes, and which of those calls fail. It runs in your application process, not in the sandbox, so the firewall rules above do not apply.\n\nSet `otel_enabled` on the client and point the standard OpenTelemetry variables at SigNoz:\n\n```\nexport DAYTONA_API_KEY=\"<your-daytona-api-key>\"\nexport OTEL_EXPORTER_OTLP_ENDPOINT=\"https://ingest.<region>.signoz.cloud:443\"\nexport OTEL_EXPORTER_OTLP_HEADERS=\"signoz-ingestion-key=<your-ingestion-key>\"\nsdk_tracing.py\npython\nimport asyncio\n \nfrom daytona import AsyncDaytona, DaytonaConfig\n \n \nasync def main():\n    # The API key is read from DAYTONA_API_KEY, exported above.\n    async with AsyncDaytona(DaytonaConfig(otel_enabled=True)) as daytona:\n        sandbox = await daytona.create()\n        await sandbox.process.code_run('print(\"hello\")')\n        await daytona.delete(sandbox)\n    # Traces flush when the context manager exits\n \n \nasyncio.run(main())\n```\n\nSet `DAYTONA_OTEL_ENABLED=true` instead of the constructor argument if you would rather keep it in the environment. Daytona also ships TypeScript, Ruby, Go, and Java SDKs with the same flag; see [OpenTelemetry Collection](https://www.daytona.io/docs/en/observability/otel-collection/).\n\nThe Python SDK bundles only the OTLP HTTP exporter, so `http/protobuf` is the only protocol. Do not set `OTEL_EXPORTER_OTLP_PROTOCOL=grpc`.\n\nSpans flush when you close the client. The async client exposes `close()` and works as a context manager. The synchronous `Daytona` client has neither, so its spans flush only at interpreter shutdown.\n\n## ## Collect Daytona's Built-in Telemetry\n\nOne dashboard setting turns on everything Daytona instruments itself: per-sandbox resource metrics, toolbox API spans, and organization quota metrics. You write no code for any of it.\n\n1. Open the [Daytona Dashboard](https://app.daytona.io/dashboard/settings) and go to the**OpenTelemetry** section. Only organization owners see it.\n2. Set **OTLP Endpoint** to`https://ingest.<region>.signoz.cloud/` . Leave the port off, and keep the trailing slash. Daytona appends the signal path itself.\n3. Add a header with key `signoz-ingestion-key` and your ingestion key as the value.\n4. Save. Daytona takes up to five minutes to apply the change.\n\nSandboxes must also be able to reach `*.daytona.io`, as described in [Step 1](#step-1-allow-signoz-through-the-sandbox-firewall). Without it the per-sandbox signals below never leave the sandbox.\n\n### Per-sandbox metrics\n\nThe daemon reports every sandbox under `service.name = sandbox-<sandbox-id>`, with the sandbox id repeated in `service.instance.id` and the daemon build in `service.version`. Ten gauges arrive:\n\n| Metric | Unit | Description | \n|---|---|---|\n| `daytona.sandbox.cpu.utilization` | percent | CPU used as a share of the limit | \n| `daytona.sandbox.cpu.limit` | cores | CPU cores the sandbox may use | \n| `daytona.sandbox.memory.utilization` | percent | Memory used as a share of the limit | \n| `daytona.sandbox.memory.usage` | bytes | Memory in use | \n| `daytona.sandbox.memory.limit` | bytes | Memory ceiling | \n| `daytona.sandbox.memory.cache` | bytes | Page cache | \n| `daytona.sandbox.filesystem.utilization` | percent | Disk used as a share of the total | \n| `daytona.sandbox.filesystem.usage` | bytes | Disk in use | \n| `daytona.sandbox.filesystem.available` | bytes | Disk free | \n| `daytona.sandbox.filesystem.total` | bytes | Disk size | \n\nEach carries `daytona_organization_id`, `daytona_region_id`, and `daytona_snapshot` as resource attributes. Note the underscores: Daytona's own telemetry uses snake_case, while attributes you set in your own code follow whatever convention you choose.\n\n### Toolbox API spans\n\nThe daemon traces the API your SDK calls into, as server spans named for the route: `POST /process/execute` when you run a command, `POST /files/bulk-upload` when you upload a file. They carry `http.route`, `url.path`, and `client.address`, and give you the sandbox-side duration of work your SDK spans only see from the outside.\n\n### Organization quota metrics\n\nPushed every 60 seconds, covering consumption against your plan limits across every sandbox. The [Daytona dashboard template](https://signoz.io/docs/dashboards/dashboard-templates/daytona-dashboard/) charts all of these already:\n\n| Metric | Unit | Description | \n|---|---|---|\n| `daytona.sandbox.used_cpu` | `{cpu}` | CPU cores consumed by active sandboxes | \n| `daytona.sandbox.total_cpu` | `{cpu}` | CPU quota for the organization | \n| `daytona.sandbox.used_ram` | `GiBy` | Memory consumed by active sandboxes | \n| `daytona.sandbox.total_ram` | `GiBy` | Memory quota for the organization | \n| `daytona.sandbox.used_storage` | `GiBy` | Disk consumed by sandboxes | \n| `daytona.sandbox.total_storage` | `GiBy` | Disk quota for the organization | \n| `daytona.sandbox.used_gpu` | `{gpu}` | GPU consumed by active sandboxes | \n| `daytona.sandbox.total_gpu` | `{gpu}` | GPU quota for the organization | \n\nEach carries `organization.id` as a resource attribute, plus `region.id` and `sandbox.class` as data point attributes. `sandbox.class` separates `container` from `windows` sandboxes, which Daytona meters against the same quota, so a headline usage number without it hides the split.\n\n## Validate\n\nWait a minute after your first run, then check each signal.\n\n**Traces:** Open the [Traces explorer](https://signoz.io/docs/userguide/traces/) and filter on `service.name = '<your-service-name>'`.\n\n**Logs:** Open the [Logs explorer](https://signoz.io/docs/userguide/logs_query_builder/) and filter on the same service name.\n\n**Metrics:** Open the [Metrics explorer](https://signoz.io/docs/metrics-management/metrics-explorer/) and search for `sandbox.tasks.completed`. Search `daytona.sandbox` for the eighteen gauges Daytona itself sends, which appear once you finish [Collect Daytona's Built-in Telemetry](#collect-daytonas-built-in-telemetry).\n\n**Daytona's own signals:** Filter either explorer on `service.name` starting with `sandbox-` to confirm the daemon is reporting. Metrics and toolbox API spans arrive within about a minute of the sandbox starting.\n\n## ## Attribute and Span Reference\n\n- **SDK spans carry no `daytona.*` attributes.** They carry no sandbox id, organization id, or region. The sandbox id appears only inside`http_url` , for example`https://proxy.app-eu.daytona.io/toolbox/<sandbox-id>/process/code-run` , so grouping SDK spans by sandbox means parsing that URL. Per-sandbox attribution comes from the telemetry you emit inside the sandbox.\n- **`OTEL_SERVICE_NAME` does not apply to SDK tracing.** The SDK sets`service.name` to`daytona-python-sdk` and`service.version` to the SDK version, no matter what you set. Every application using the SDK reports under that one name, so you cannot split SDK traffic per app through the service name.\n- **Span names include the client class.** The async client emits`AsyncDaytona.create` and`AsyncSandbox.stop` . The synchronous client drops the`Async` prefix from every span name. A filter written for one client returns nothing for the other.\n- **`daytona.snapshot` is a registry digest.** It resolves to a value such as`cr.app.daytona.io/sbox/daytona-<sha>:daytona` , not the friendly`daytonaio/sandbox:0.8.0` name you passed at creation.\n\nThe async client emits internal spans for `AsyncDaytona.create`, `get`, `list`, `list.fetch_page`, `start`, `stop`, and `delete`; `AsyncSandbox.start`, `stop`, `delete`, `refresh_data`, `wait_for_sandbox_start`, and `wait_for_sandbox_stop`; `AsyncProcess.code_run`; and `AsyncFileSystem.upload_file` and `upload_files`. Outbound HTTP calls appear as client spans named for the method alone: `GET`, `POST`, `DELETE`.\n\nSandboxes expose `DAYTONA_SANDBOX_ID`, `DAYTONA_ORGANIZATION_ID`, `DAYTONA_REGION_ID`, and `DAYTONA_SANDBOX_SNAPSHOT` to every process. Use them for resource attributes, as the sandbox script above does.\n\n## ## Troubleshooting\n\n### Connection reset when exporting from inside a sandbox\n\nSymptom: the exporter retries and gives up, with `Connection reset by peer` or `OpenSSL SSL_connect: Connection reset by peer`.\n\nLikely cause: sandbox egress is deny-by-default and the SigNoz ingestion host is not allowed. Package registries work, which makes the block look selective.\n\nFix: create the sandbox with `domain_allow_list` including `ingest.<region>.signoz.cloud`, as in Step 1. Tier 1 and Tier 2 accounts cannot set this, so upgrade the plan or export through a host that is already reachable.\n\nVerify: from inside the sandbox, `curl -s -o /dev/null -w '%{http_code}' https://ingest.<region>.signoz.cloud` returns `404`. That means TLS completed and egress is open. `000` means the connection is still blocked.\n\n### No SDK spans arrive\n\nSymptom: sandbox operations succeed, but no `daytona-python-sdk` service appears in SigNoz.\n\nLikely cause: tracing is off, or the process exited before spans flushed.\n\nFix: confirm `otel_enabled=True` or `DAYTONA_OTEL_ENABLED=true`, and close the client. Use `async with AsyncDaytona(...)` or call `await daytona.close()`.\n\nVerify: `service.name = 'daytona-python-sdk'` returns spans in the Traces explorer.\n\n### Sandbox CPU, memory, and filesystem metrics never arrive\n\nSymptom: organization gauges arrive and your own spans arrive, but nothing under `service.name = 'sandbox-<sandbox-id>'`. Nothing reports an error.\n\nLikely cause: the sandbox cannot reach `otel-collector.app.daytona.io`. A `domain_allow_list` that omits `*.daytona.io` blocks the daemon's exporter while leaving your own exporter working, so the failure is invisible from inside your code.\n\nFix: add `*.daytona.io` to `domain_allow_list` and create a new sandbox. Existing sandboxes keep the rules they were created with.\n\nVerify: from inside the sandbox, `curl -s -o /dev/null -w '%{http_code}' https://otel-collector.app.daytona.io` returns `404`. `000` means the daemon is still firewalled.\n\n### Organization metrics are missing\n\nSymptom: no `daytona.sandbox.used_cpu` after saving the configuration.\n\nLikely cause: the push interval has not elapsed, or the header is wrong.\n\nFix: Daytona takes up to five minutes to apply an endpoint change, then pushes on a 60-second interval. Wait out both with at least one sandbox running, and confirm the header key is exactly `signoz-ingestion-key`.\n\nVerify: search for `daytona.sandbox` in the Metrics explorer.\n\n## ## Limitations\n\n- **Sandbox stdout and stderr are not exported.** Daytona documents application logs as part of sandbox telemetry, but the only log records that arrive are the daemon's own startup diagnostics, four per sandbox. Anything your process prints stays in the sandbox, so route it through the OpenTelemetry logs SDK as[Step 2](#step-2-instrument-the-code-that-runs-inside-the-sandbox) does.\n- **`otelEndpointOverride` on sandbox creation has no effect.** The API accepts the field and returns`200` , then reports the sandbox with`otelEndpointOverride: null` .\n- **The documented `otel-config` API call does not work with an API key.**`PUT /api/organizations/<org>/otel-config` returns`403 Invalid authentication context` for a`dtn_` key regardless of its permissions. Use the Dashboard.\n- **A long-lived WebSocket span distorts SDK latency.** The SDK opens a connection to`wss://app.daytona.io/api/socket.io/` that lives as long as the client. It appears as a root span lasting the whole session and dominates p99 for the`daytona-python-sdk` service. Exclude it when measuring operation latency.\n- **SDK operations are separate traces.** Each call roots its own trace rather than nesting under a session, so there is no single trace covering create through delete. Correlate with your own parent span if you need that view.\n\n## Next Steps\n\n- [Import the Daytona dashboard](https://signoz.io/docs/dashboards/dashboard-templates/daytona-dashboard/) for per-sandbox resources, quota headroom, and SDK latency, or[build your own](https://signoz.io/docs/userguide/manage-dashboards/) over`sandbox.tasks.completed` .\n- [Set a metrics-based alert](https://signoz.io/docs/alerts-management/metrics-based-alerts/) on`daytona.sandbox.used_cpu` against`daytona.sandbox.total_cpu` to catch quota exhaustion before sandbox creation starts failing.\n- [Add a log-based alert](https://signoz.io/docs/alerts-management/log-based-alerts/) on errors your sandbox code logs, scoped by`daytona.sandbox.id` .\n- [Explore the traces](https://signoz.io/docs/userguide/traces/) to find which agent step is slowest across sandboxes.\n- Instrument the model calls your sandbox code makes, so agent steps and LLM spans land in one trace. Browse [all LLM observability integrations](https://signoz.io/docs/llm-observability/) .\n\n## Get Help\n\nIf you need help with the steps in this topic, please reach out to us on [SigNoz Community Slack](https://signoz.io/slack/). If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at [cloud-support@signoz.io](mailto:cloud-support@signoz.io).", "url": "https://wpnews.pro/news/daytona-sandbox-monitoring-tracing-with-opentelemetry", "canonical_source": "https://signoz.io/docs/daytona-monitoring", "published_at": "2026-09-05 00:00:00+00:00", "updated_at": "2026-09-09 07:59:30.897920+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure"], "entities": ["Daytona", "SigNoz", "OpenTelemetry"], "alternates": {"html": "https://wpnews.pro/news/daytona-sandbox-monitoring-tracing-with-opentelemetry", "markdown": "https://wpnews.pro/news/daytona-sandbox-monitoring-tracing-with-opentelemetry.md", "text": "https://wpnews.pro/news/daytona-sandbox-monitoring-tracing-with-opentelemetry.txt", "jsonld": "https://wpnews.pro/news/daytona-sandbox-monitoring-tracing-with-opentelemetry.jsonld"}}