Daytona Sandbox Monitoring & Tracing with OpenTelemetry

Daytona, the AI code execution platform, has published a guide for monitoring AI-generated code running in its sandboxes using OpenTelemetry and SigNoz, detailing four non-overlapping telemetry paths: in-sandbox app instrumentation, Daytona's built-in per-sandbox CPU/memory/disk metrics, organization quota metrics, and SDK tracing for lifecycle calls. The guide instructs users to configure a domain allow list and environment variables to route telemetry to SigNoz, with prerequisites including Python 3.10+, a Daytona API key with write:sandboxes permission, and a plan that allows per-sandbox network rules.

Overview Daytona https://www.daytona.io/ runs AI-generated code in isolated sandboxes. Your agent creates a sandbox, executes code in it, reads files back, and tears it down. Two things go wrong in that loop, and each needs its own telemetry: the work inside the sandbox fails or hangs, and the sandbox lifecycle itself gets slow or errors out. Prerequisites - An instance of SigNoz either Cloud https://signoz.io/teams/ or Self-Hosted https://signoz.io/docs/install/self-host/ - A Daytona account and an API key with write:sandboxes permission - Python 3.10 or later, with the daytona SDK installed. The SDK requires 3.10. - A Daytona plan that allows per-sandbox network rules. Tier 1 and Tier 2 accounts cannot override the default egress policy, which blocks SigNoz. See Network Limits https://www.daytona.io/docs/en/network-limits/ . How it works Daytona exposes four OpenTelemetry paths. They do not overlap, so pick the ones that answer your question. | Path | What you get | What it costs you | |---|---|---| | App instrumentation inside the sandbox | Spans, logs, and custom metrics from the code your agent runs | Code you write and ship into the sandbox | | Daytona's built-in telemetry | Per-sandbox CPU, memory, and disk, plus toolbox API spans | One dashboard setting | | Organization quota metrics | CPU, memory, storage, and GPU used against your org quota | The same dashboard setting | | SDK tracing | Latency and failures for create , start , stop , delete , file, and process calls | One flag on the client | Start with the first path. It shows what your agent did inside the sandbox, which nothing else reconstructs. The rest are collapsed below; expand them when you need them. All four use OpenTelemetry https://opentelemetry.io/ and export to the same SigNoz endpoint. Monitor Work Inside a Daytona Sandbox This path shows what your agent did: which steps ran, how long each took, what it logged, and where it failed. Step 1: Allow SigNoz through the sandbox firewall Sandbox egress is deny-by-default. Package registries stay reachable and Daytona blocks the rest, so an OTLP exporter fails with a TLS reset until you allow the ingestion host. Pass domain allow list when you create the sandbox: run agent.py python import asyncio from daytona import AsyncDaytona, DaytonaConfig, CreateSandboxFromSnapshotParams async def main : async with AsyncDaytona DaytonaConfig api key="<your-daytona-api-key " as daytona: sandbox = await daytona.create CreateSandboxFromSnapshotParams domain allow list="ingest.<region .signoz.cloud,pypi.org,files.pythonhosted.org, .daytona.io", env vars={ "OTEL EXPORTER OTLP ENDPOINT": "https://ingest.<region .signoz.cloud:443", "OTEL EXPORTER OTLP HEADERS": "signoz-ingestion-key=<your-ingestion-key ", "OTEL SERVICE NAME": "<your-service-name ", }, print "created", sandbox.id asyncio.run main Verify these values: - <region : Your SigNoz Cloud region https://signoz.io/docs/ingestion/signoz-cloud/overview/ endpoint . - <your-ingestion-key : Your SigNoz ingestion key https://signoz.io/docs/ingestion/signoz-cloud/keys/ . - <your-daytona-api-key : Created under Daytona Dashboard, Keys. - <your-service-name : What your sandbox telemetry appears under in SigNoz, and what you filter on in Validate validate . The allow list takes at most 20 comma-separated domains and supports wildcards such as .example.com . It is mutually exclusive with networkBlockAll and networkAllowList , so set only one of the three per sandbox. Setting a list also drops the default allowances, so include the registries your code installs from. The env vars values reach every process in the sandbox, so the OpenTelemetry SDK picks up the endpoint and headers with no further configuration. Step 2: Instrument the code that runs inside the sandbox This is ordinary OpenTelemetry setup apart from the resource attributes, which read environment variables Daytona injects into every sandbox. Use them to scope telemetry to one sandbox later. The service name needs no line of its own, because Resource.create reads OTEL SERVICE NAME from the environment you set in Step 1. sandbox app.py python import logging import os from opentelemetry import metrics, trace from opentelemetry.exporter.otlp.proto.http. log exporter import OTLPLogExporter from opentelemetry.exporter.otlp.proto.http.metric exporter import OTLPMetricExporter from opentelemetry.exporter.otlp.proto.http.trace exporter import OTLPSpanExporter from opentelemetry.sdk. logs import LoggerProvider, LoggingHandler from opentelemetry.sdk. logs.export import BatchLogRecordProcessor from opentelemetry.sdk.metrics import MeterProvider from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor service.name is not set here: Resource.create runs OpenTelemetry's environment detector, which picks up the OTEL SERVICE NAME set in Step 1. resource = Resource.create { "daytona.sandbox.id": os.environ.get "DAYTONA SANDBOX ID", "" , "daytona.organization.id": os.environ.get "DAYTONA ORGANIZATION ID", "" , "daytona.region.id": os.environ.get "DAYTONA REGION ID", "" , "daytona.snapshot": os.environ.get "DAYTONA SANDBOX SNAPSHOT", "" , } tracer provider = TracerProvider resource=resource tracer provider.add span processor BatchSpanProcessor OTLPSpanExporter trace.set tracer provider tracer provider logger provider = LoggerProvider resource=resource logger provider.add log record processor BatchLogRecordProcessor OTLPLogExporter logging.getLogger .addHandler LoggingHandler logger provider=logger provider logging.getLogger .setLevel logging.INFO reader = PeriodicExportingMetricReader OTLPMetricExporter meter provider = MeterProvider resource=resource, metric readers= reader metrics.set meter provider meter provider tracer = trace.get tracer "agent-work" meter = metrics.get meter "agent-work" tasks = meter.create counter "sandbox.tasks.completed" log = logging.getLogger "sandbox-app" with tracer.start as current span "agent.task" as span: span.set attribute "task.kind", "code-execution" log.info "starting work" with tracer.start as current span "agent.step.compute" : total = sum i i for i in range 200000 log.info "compute finished with total %s", total tasks.add 1, {"task.kind": "code-execution"} tracer provider.force flush logger provider.force flush meter provider.force flush Flush all three providers before the process exits. Sandbox processes are short-lived, and an unflushed batch dies with the process. Step 3: Upload the script, install the SDK, and run it run agent.py python import asyncio from daytona import AsyncDaytona, DaytonaConfig, CreateSandboxFromSnapshotParams async def main : async with AsyncDaytona DaytonaConfig api key="<your-daytona-api-key " as daytona: sandbox = await daytona.create CreateSandboxFromSnapshotParams domain allow list="ingest.<region .signoz.cloud,pypi.org,files.pythonhosted.org, .daytona.io", env vars={ "OTEL EXPORTER OTLP ENDPOINT": "https://ingest.<region .signoz.cloud:443", "OTEL EXPORTER OTLP HEADERS": "signoz-ingestion-key=<your-ingestion-key ", "OTEL SERVICE NAME": "<your-service-name ", }, with open "sandbox app.py", "rb" as f: await sandbox.fs.upload file f.read , "/home/daytona/sandbox app.py" await sandbox.process.exec "pip install --quiet opentelemetry-sdk opentelemetry-exporter-otlp-proto-http" result = await sandbox.process.exec "python3 /home/daytona/sandbox app.py" print result.exit code, result.result asyncio.run main To skip the install on every run, bake the OpenTelemetry packages into a snapshot https://www.daytona.io/docs/en/snapshots/ and create sandboxes from it. Trace Daytona SDK Operations SDK tracing covers the other half: how long Daytona itself takes to create, start, stop, and delete sandboxes, and which of those calls fail. It runs in your application process, not in the sandbox, so the firewall rules above do not apply. Set otel enabled on the client and point the standard OpenTelemetry variables at SigNoz: export DAYTONA API KEY="<your-daytona-api-key " export OTEL EXPORTER OTLP ENDPOINT="https://ingest.<region .signoz.cloud:443" export OTEL EXPORTER OTLP HEADERS="signoz-ingestion-key=<your-ingestion-key " sdk tracing.py python import asyncio from daytona import AsyncDaytona, DaytonaConfig async def main : The API key is read from DAYTONA API KEY, exported above. async with AsyncDaytona DaytonaConfig otel enabled=True as daytona: sandbox = await daytona.create await sandbox.process.code run 'print "hello" ' await daytona.delete sandbox Traces flush when the context manager exits asyncio.run main Set DAYTONA OTEL ENABLED=true instead of the constructor argument if you would rather keep it in the environment. Daytona also ships TypeScript, Ruby, Go, and Java SDKs with the same flag; see OpenTelemetry Collection https://www.daytona.io/docs/en/observability/otel-collection/ . The Python SDK bundles only the OTLP HTTP exporter, so http/protobuf is the only protocol. Do not set OTEL EXPORTER OTLP PROTOCOL=grpc . Spans flush when you close the client. The async client exposes close and works as a context manager. The synchronous Daytona client has neither, so its spans flush only at interpreter shutdown. Collect Daytona's Built-in Telemetry One dashboard setting turns on everything Daytona instruments itself: per-sandbox resource metrics, toolbox API spans, and organization quota metrics. You write no code for any of it. 1. Open the Daytona Dashboard https://app.daytona.io/dashboard/settings and go to the OpenTelemetry section. Only organization owners see it. 2. Set OTLP Endpoint to https://ingest.<region .signoz.cloud/ . Leave the port off, and keep the trailing slash. Daytona appends the signal path itself. 3. Add a header with key signoz-ingestion-key and your ingestion key as the value. 4. Save. Daytona takes up to five minutes to apply the change. Sandboxes must also be able to reach .daytona.io , as described in Step 1 step-1-allow-signoz-through-the-sandbox-firewall . Without it the per-sandbox signals below never leave the sandbox. Per-sandbox metrics The daemon reports every sandbox under service.name = sandbox-<sandbox-id , with the sandbox id repeated in service.instance.id and the daemon build in service.version . Ten gauges arrive: | Metric | Unit | Description | |---|---|---| | daytona.sandbox.cpu.utilization | percent | CPU used as a share of the limit | | daytona.sandbox.cpu.limit | cores | CPU cores the sandbox may use | | daytona.sandbox.memory.utilization | percent | Memory used as a share of the limit | | daytona.sandbox.memory.usage | bytes | Memory in use | | daytona.sandbox.memory.limit | bytes | Memory ceiling | | daytona.sandbox.memory.cache | bytes | Page cache | | daytona.sandbox.filesystem.utilization | percent | Disk used as a share of the total | | daytona.sandbox.filesystem.usage | bytes | Disk in use | | daytona.sandbox.filesystem.available | bytes | Disk free | | daytona.sandbox.filesystem.total | bytes | Disk size | Each carries daytona organization id , daytona region id , and daytona snapshot as resource attributes. Note the underscores: Daytona's own telemetry uses snake case, while attributes you set in your own code follow whatever convention you choose. Toolbox API spans The daemon traces the API your SDK calls into, as server spans named for the route: POST /process/execute when you run a command, POST /files/bulk-upload when you upload a file. They carry http.route , url.path , and client.address , and give you the sandbox-side duration of work your SDK spans only see from the outside. Organization quota metrics Pushed every 60 seconds, covering consumption against your plan limits across every sandbox. The Daytona dashboard template https://signoz.io/docs/dashboards/dashboard-templates/daytona-dashboard/ charts all of these already: | Metric | Unit | Description | |---|---|---| | daytona.sandbox.used cpu | {cpu} | CPU cores consumed by active sandboxes | | daytona.sandbox.total cpu | {cpu} | CPU quota for the organization | | daytona.sandbox.used ram | GiBy | Memory consumed by active sandboxes | | daytona.sandbox.total ram | GiBy | Memory quota for the organization | | daytona.sandbox.used storage | GiBy | Disk consumed by sandboxes | | daytona.sandbox.total storage | GiBy | Disk quota for the organization | | daytona.sandbox.used gpu | {gpu} | GPU consumed by active sandboxes | | daytona.sandbox.total gpu | {gpu} | GPU quota for the organization | Each carries organization.id as a resource attribute, plus region.id and sandbox.class as data point attributes. sandbox.class separates container from windows sandboxes, which Daytona meters against the same quota, so a headline usage number without it hides the split. Validate Wait a minute after your first run, then check each signal. Traces: Open the Traces explorer https://signoz.io/docs/userguide/traces/ and filter on service.name = '<your-service-name ' . Logs: Open the Logs explorer https://signoz.io/docs/userguide/logs query builder/ and filter on the same service name. Metrics: Open the Metrics explorer https://signoz.io/docs/metrics-management/metrics-explorer/ and search for sandbox.tasks.completed . Search daytona.sandbox for the eighteen gauges Daytona itself sends, which appear once you finish Collect Daytona's Built-in Telemetry collect-daytonas-built-in-telemetry . Daytona's own signals: Filter either explorer on service.name starting with sandbox- to confirm the daemon is reporting. Metrics and toolbox API spans arrive within about a minute of the sandbox starting. Attribute and Span Reference - SDK spans carry no daytona. attributes. They carry no sandbox id, organization id, or region. The sandbox id appears only inside http url , for example https://proxy.app-eu.daytona.io/toolbox/<sandbox-id /process/code-run , so grouping SDK spans by sandbox means parsing that URL. Per-sandbox attribution comes from the telemetry you emit inside the sandbox. - OTEL SERVICE NAME does not apply to SDK tracing. The SDK sets service.name to daytona-python-sdk and service.version to the SDK version, no matter what you set. Every application using the SDK reports under that one name, so you cannot split SDK traffic per app through the service name. - Span names include the client class. The async client emits AsyncDaytona.create and AsyncSandbox.stop . The synchronous client drops the Async prefix from every span name. A filter written for one client returns nothing for the other. - daytona.snapshot is a registry digest. It resolves to a value such as cr.app.daytona.io/sbox/daytona-<sha :daytona , not the friendly daytonaio/sandbox:0.8.0 name you passed at creation. The async client emits internal spans for AsyncDaytona.create , get , list , list.fetch page , start , stop , and delete ; AsyncSandbox.start , stop , delete , refresh data , wait for sandbox start , and wait for sandbox stop ; AsyncProcess.code run ; and AsyncFileSystem.upload file and upload files . Outbound HTTP calls appear as client spans named for the method alone: GET , POST , DELETE . Sandboxes expose DAYTONA SANDBOX ID , DAYTONA ORGANIZATION ID , DAYTONA REGION ID , and DAYTONA SANDBOX SNAPSHOT to every process. Use them for resource attributes, as the sandbox script above does. Troubleshooting Connection reset when exporting from inside a sandbox Symptom: the exporter retries and gives up, with Connection reset by peer or OpenSSL SSL connect: Connection reset by peer . Likely cause: sandbox egress is deny-by-default and the SigNoz ingestion host is not allowed. Package registries work, which makes the block look selective. Fix: create the sandbox with domain allow list including ingest.<region .signoz.cloud , as in Step 1. Tier 1 and Tier 2 accounts cannot set this, so upgrade the plan or export through a host that is already reachable. Verify: from inside the sandbox, curl -s -o /dev/null -w '%{http code}' https://ingest.<region .signoz.cloud returns 404 . That means TLS completed and egress is open. 000 means the connection is still blocked. No SDK spans arrive Symptom: sandbox operations succeed, but no daytona-python-sdk service appears in SigNoz. Likely cause: tracing is off, or the process exited before spans flushed. Fix: confirm otel enabled=True or DAYTONA OTEL ENABLED=true , and close the client. Use async with AsyncDaytona ... or call await daytona.close . Verify: service.name = 'daytona-python-sdk' returns spans in the Traces explorer. Sandbox CPU, memory, and filesystem metrics never arrive Symptom: organization gauges arrive and your own spans arrive, but nothing under service.name = 'sandbox-<sandbox-id ' . Nothing reports an error. Likely cause: the sandbox cannot reach otel-collector.app.daytona.io . A domain allow list that omits .daytona.io blocks the daemon's exporter while leaving your own exporter working, so the failure is invisible from inside your code. Fix: add .daytona.io to domain allow list and create a new sandbox. Existing sandboxes keep the rules they were created with. Verify: from inside the sandbox, curl -s -o /dev/null -w '%{http code}' https://otel-collector.app.daytona.io returns 404 . 000 means the daemon is still firewalled. Organization metrics are missing Symptom: no daytona.sandbox.used cpu after saving the configuration. Likely cause: the push interval has not elapsed, or the header is wrong. Fix: Daytona takes up to five minutes to apply an endpoint change, then pushes on a 60-second interval. Wait out both with at least one sandbox running, and confirm the header key is exactly signoz-ingestion-key . Verify: search for daytona.sandbox in the Metrics explorer. Limitations - Sandbox stdout and stderr are not exported. Daytona documents application logs as part of sandbox telemetry, but the only log records that arrive are the daemon's own startup diagnostics, four per sandbox. Anything your process prints stays in the sandbox, so route it through the OpenTelemetry logs SDK as Step 2 step-2-instrument-the-code-that-runs-inside-the-sandbox does. - otelEndpointOverride on sandbox creation has no effect. The API accepts the field and returns 200 , then reports the sandbox with otelEndpointOverride: null . - The documented otel-config API call does not work with an API key. PUT /api/organizations/<org /otel-config returns 403 Invalid authentication context for a dtn key regardless of its permissions. Use the Dashboard. - A long-lived WebSocket span distorts SDK latency. The SDK opens a connection to wss://app.daytona.io/api/socket.io/ that lives as long as the client. It appears as a root span lasting the whole session and dominates p99 for the daytona-python-sdk service. Exclude it when measuring operation latency. - SDK operations are separate traces. Each call roots its own trace rather than nesting under a session, so there is no single trace covering create through delete. Correlate with your own parent span if you need that view. Next Steps - Import the Daytona dashboard https://signoz.io/docs/dashboards/dashboard-templates/daytona-dashboard/ for per-sandbox resources, quota headroom, and SDK latency, or build your own https://signoz.io/docs/userguide/manage-dashboards/ over sandbox.tasks.completed . - Set a metrics-based alert https://signoz.io/docs/alerts-management/metrics-based-alerts/ on daytona.sandbox.used cpu against daytona.sandbox.total cpu to catch quota exhaustion before sandbox creation starts failing. - Add a log-based alert https://signoz.io/docs/alerts-management/log-based-alerts/ on errors your sandbox code logs, scoped by daytona.sandbox.id . - Explore the traces https://signoz.io/docs/userguide/traces/ to find which agent step is slowest across sandboxes. - Instrument the model calls your sandbox code makes, so agent steps and LLM spans land in one trace. Browse all LLM observability integrations https://signoz.io/docs/llm-observability/ . Get Help If you need help with the steps in this topic, please reach out to us on SigNoz Community Slack https://signoz.io/slack/ . If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at cloud-support@signoz.io mailto:cloud-support@signoz.io .