{"slug": "monitor-modal-functions-and-sandboxes-with-opentelemetry", "title": "Monitor Modal Functions and Sandboxes with OpenTelemetry", "summary": "SigNoz has published a guide for monitoring Modal functions and Sandboxes with OpenTelemetry, detailing three non-overlapping telemetry paths: the Modal OpenTelemetry integration for workspace-wide logs and metrics, tracing in the caller for per-command latency and failures, and tracing inside the Sandbox for code-level spans. The integration requires creating a Modal Secret with an authorization header containing the SigNoz ingestion key, setting the OpenTelemetry push URL to the SigNoz endpoint, and testing the connection with a test log that appears in SigNoz as 'Hello from Modal!' with service.name 'modal.test_logs'.", "body_md": "## Overview\n\n[Modal](https://modal.com) runs Python functions and Sandboxes on demand. A Sandbox is a container that Modal creates at runtime to execute untrusted or AI-generated code. Your agent creates a Sandbox, runs commands in it, reads the output, and terminates it.\n\nTwo things go wrong in that loop, and each needs its own telemetry. The code inside the Sandbox fails or hangs, and the Sandbox itself starts slowly or runs out of memory.\n\n## Prerequisites\n\n- An instance of SigNoz (either [Cloud](https://signoz.io/teams/) or[Self-Hosted](https://signoz.io/docs/install/self-host/) )\n- A SigNoz [ingestion key](https://signoz.io/docs/ingestion/signoz-cloud/keys/)\n- A [Modal account](https://modal.com/signup) , with permission to edit workspace settings\n- The `modal` CLI, installed with`pip install modal` and authenticated with`modal setup`\n\n## How it works\n\nModal offers three OpenTelemetry paths. They do not overlap, so pick the ones that answer your question.\n\n| Path | What you get | What it costs you | \n|---|---|---|\n| Modal OpenTelemetry integration | Function logs, Sandbox logs, and container metrics for the whole workspace | One workspace setting | \n| Tracing in the caller | Latency and failures for every `Sandbox.create` and every command you run | Code in the application that drives the Sandboxes | \n| Tracing inside the Sandbox | Spans and logs from the code your agent runs | Packages and a Secret in the Sandbox image | \n\nStart with the integration. It needs no code changes, and it covers every Function and every Sandbox in the workspace. It sends no traces, so add the second path when you need to know which command an agent ran, how long it took, and whether it failed.\n\n## Monitoring Modal\n\nThe Modal OpenTelemetry integration sends Function logs, Sandbox logs, and container metrics to any backend that accepts OTLP over HTTP. You give Modal two things: the base URL of your SigNoz endpoint, and a Modal Secret that holds the authentication header.\n\n### Step 1: Create the Modal Secret\n\nModal builds the request headers from a Secret. Each key starts with `OTEL_HEADER_`, and the rest of the key is the header name. Use the `authorization` header, which SigNoz accepts with the ingestion key as its value:\n\n```\nmodal secret create signoz-otel \\\n  OTEL_HEADER_authorization=<your-ingestion-key>\n```\n\nYou can create the same Secret from the [Modal Secrets page](https://modal.com/secrets) with the **OpenTelemetry** template.\n\n**Verify these values:**\n\n- `<your-ingestion-key>` : Your SigNoz[ingestion key](https://signoz.io/docs/ingestion/signoz-cloud/keys/) .\n\n### Step 2: Point Modal at your SigNoz endpoint\n\n1. Go to the [Modal metrics settings page](https://modal.com/settings/metrics) .\n2. Set the OpenTelemetry push URL to `https://ingest.<region>.signoz.cloud` .\n3. Select the `signoz-otel` Secret that you created in Step 1.\n4. Save the changes.\n\n**Verify these values:**\n\n- `<region>` : Your[SigNoz Cloud region](https://signoz.io/docs/ingestion/signoz-cloud/overview/#endpoint) .\n\n### Step 3: Test the connection\n\nClick **Send Test** on the metrics settings page. Modal sends one log line to your endpoint and reports the result.\n\nOpen **Logs** in SigNoz and search for `Hello from Modal!`. The test log arrives with `service.name = modal.test_logs`. If you see it, the push URL and the ingestion key both work.\n\nModal starts the export when you save the integration. Modal does not backfill older logs.\n\nThe integration tells you that a Sandbox ran. It does not tell you which command an agent executed, how long each step took, or which run failed. Add the OpenTelemetry SDK to the code that drives the Sandboxes to get that.\n\nThis path instruments the caller, which is the script or service that calls `Sandbox.create`. The Sandbox itself needs no changes.\n\n### Step 1: Install the packages\n\n```\npip install modal opentelemetry-sdk opentelemetry-exporter-otlp-proto-http\n```\n\n### Step 2: Configure the exporter\n\nThe exporter reads its endpoint and headers from the environment:\n\n```\nexport OTEL_EXPORTER_OTLP_ENDPOINT=\"https://ingest.<region>.signoz.cloud:443\"\nexport OTEL_EXPORTER_OTLP_HEADERS=\"signoz-ingestion-key=<your-ingestion-key>\"\nexport OTEL_SERVICE_NAME=\"<your-service-name>\"\n```\n\n**Verify these values:**\n\n- `<region>` : Your[SigNoz Cloud region](https://signoz.io/docs/ingestion/signoz-cloud/overview/#endpoint) .\n- `<your-ingestion-key>` : Your SigNoz[ingestion key](https://signoz.io/docs/ingestion/signoz-cloud/keys/) .\n- `<your-service-name>` : The name this code appears under in SigNoz, such as`modal-sandbox-runner` .\n\nThe Python SDK reads all three, so the code below passes no endpoint, header, or service name of its own.\n\n### Step 3: Wrap the Sandbox calls in spans\n\nCreate one span for the Sandbox and one span for each command it runs. Record the exit code, and mark the span as an error when the command fails:\n\n``` python\nimport modal\nfrom opentelemetry import trace\nfrom opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter\nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor\nfrom opentelemetry.trace import Status, StatusCode\n \n# The endpoint, headers, and service name all come from the environment\nprovider = TracerProvider()\nprovider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))\ntrace.set_tracer_provider(provider)\ntracer = trace.get_tracer(\"modal.sandbox\")\n \n \ndef run_in_sandbox(app, command):\n    with tracer.start_as_current_span(\"sandbox.create\") as span:\n        sandbox = modal.Sandbox.create(app=app, timeout=120)\n        span.set_attribute(\"modal.sandbox.id\", sandbox.object_id)\n \n    try:\n        with tracer.start_as_current_span(\"sandbox.exec\") as span:\n            span.set_attribute(\"modal.sandbox.id\", sandbox.object_id)\n            span.set_attribute(\"modal.sandbox.command\", \" \".join(command))\n            process = sandbox.exec(*command)\n            output = process.stdout.read()\n            process.wait()\n            span.set_attribute(\"modal.sandbox.exit_code\", process.returncode)\n            if process.returncode != 0:\n                span.set_status(\n                    Status(StatusCode.ERROR, f\"exit code {process.returncode}\")\n                )\n            return output\n    finally:\n        sandbox.terminate()\n \n \nif __name__ == \"__main__\":\n    app = modal.App.lookup(\"agent-sandboxes\", create_if_missing=True)\n    with tracer.start_as_current_span(\"agent.run\"):\n        print(run_in_sandbox(app, [\"python\", \"-c\", \"print('hello')\"]))\n        print(run_in_sandbox(app, [\"python\", \"-c\", \"raise SystemExit(3)\"]))\n    provider.shutdown()\n```\n\n`sandbox.object_id` is the Sandbox ID, such as `sb-QHSNZOpAvp2kkGxd9gVaG0`. The logs carry the same ID in their `sandbox_id` attribute, so this one attribute joins a trace to the output of that Sandbox.\n\nRun the script:\n\n```\npython agent.py\n```\n\n## Validate\n\nRun a Modal Function or start a Sandbox first, so that Modal has telemetry to send.\n\nFor the integration:\n\n1. Open **Logs** in SigNoz and filter on`service.name = modal.function-logs` . Your Function and Sandbox output arrives within a minute.\n2. Open **Metrics** in SigNoz and search for`modal.` . Query`modal.cpu.utilization` and group by`object_type` to see Functions and Sandboxes side by side.\n\nFor the SDK:\n\n1. Open **Traces** in SigNoz and filter on the`service.name` you set in`OTEL_SERVICE_NAME` . The screenshots below use`modal-sandbox-runner` .\n2. Open the `agent.run` trace. It holds a`sandbox.create` span and a`sandbox.exec` span for each run.\n3. The second `sandbox.exec` span is red, with the status message`exit code 3` .\n4. Copy a `modal.sandbox.id` value and search**Logs** for`sandbox_id = '<that value>'` to read what the Sandbox printed.\n\n## ## Run the Caller as a Modal Function\n\nWhen the code that creates Sandboxes runs on Modal itself, pass the exporter settings through a Secret:\n\n```\nmodal secret create signoz-otlp \\\n  OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.<region>.signoz.cloud:443 \\\n  \"OTEL_EXPORTER_OTLP_HEADERS=signoz-ingestion-key=<your-ingestion-key>\" \\\n  OTEL_SERVICE_NAME=<your-service-name>\n```\n\nThe Secret key restriction applies only to key names. All three names here are plain, and the hyphens in `signoz-ingestion-key` sit inside the value, which Modal accepts.\n\nAttach the Secret and install the packages in the image:\n\n```\nimage = modal.Image.debian_slim().pip_install(\n    \"opentelemetry-sdk\",\n    \"opentelemetry-exporter-otlp-proto-http\",\n)\n \n@app.function(image=image, secrets=[modal.Secret.from_name(\"signoz-otlp\")])\ndef run_agent():\n    ...\n```\n\n## ## Trace the Code Inside a Sandbox\n\nThe spans in the caller time the Sandbox from outside. To see what the code inside it did, install the packages in the Sandbox image and pass the same Secret to `Sandbox.create`:\n\n```\nimage = modal.Image.debian_slim().pip_install(\n    \"opentelemetry-sdk\",\n    \"opentelemetry-exporter-otlp-proto-http\",\n)\n \nsandbox = modal.Sandbox.create(\n    app=app,\n    image=image,\n    secrets=[modal.Secret.from_name(\"signoz-otlp\")],\n    timeout=180,\n)\n```\n\nThe code inside then configures a `TracerProvider` the same way the caller does, and calls `provider.force_flush()` before it finishes.\n\n## ## Attribute and Metric Reference\n\n### Logs\n\nFunction output and Sandbox output both arrive under `service.name = modal.function-logs`. The test log is the only exception, and it uses `service.name = modal.test_logs`.\n\nEvery log record carries `app_id`, `app_name`, `container_id`, `environment`, `file_descriptor`, `level`, `workspace`, and `workspace_id`. The rest depend on what produced the log:\n\n| Attribute | Present on | \n|---|---|\n| `function_name` ,`function_id` ,`function_call_id` ,`input_id` | Function logs | \n| `sandbox_id` | Sandbox logs | \n\nTo read Sandbox output only, filter on `sandbox_id EXISTS`. To read one Sandbox, filter on the ID that `Sandbox.create` returned, for example `sandbox_id = 'sb-QHSNZOpAvp2kkGxd9gVaG0'`.\n\n`file_descriptor` is `1` for stdout and `2` for stderr. Modal sends both with `severity_text = INFO`, so use `file_descriptor = '2'` to find error output.\n\n### Metrics\n\n| Metric | What it measures | \n|---|---|\n| `modal.cpu.utilization` | CPU use of the container, where 1 means one full core | \n| `modal.memory.usage` | Memory use of the container, in bytes | \n| `modal.container.running` | 1 while the container runs | \n| `modal.container.terminations` | Count of container terminations | \n| `modal.gpu.compute.utilization` | GPU compute use, as a fraction between 0 and 1 | \n| `modal.gpu.memory.usage` | GPU memory use, in bytes | \n| `modal.gpu.clock` | GPU SM clock frequency, in MHz | \n| `modal.gpu.power.usage` | GPU power draw, in watts | \n| `modal.gpu.temperature` | GPU temperature, in degrees Celsius | \n| `modal.input_events.elapsed_time_us` | Time to handle one input, in microseconds | \n| `modal.input_events.input_queue_time_us` | Time an input waited in the queue, in microseconds | \n| `modal.input_events.coldstart_time_us` | Cold start time, in microseconds | \n| `modal.input_events.successes` | Count of inputs that succeeded | \n| `modal.input_events.total_inputs` | Count of inputs received | \n| `modal.function.pending_inputs` | Inputs that wait for a container | \n| `modal.function.running_inputs` | Inputs that a container handles now | \n\nDivide `modal.input_events.successes` by `modal.input_events.total_inputs` to get a success rate for a Function.\n\nMost metrics carry `app_id`, `app_name`, `container_id`, `environment_id`, `environment_name`, `object_type`, `workspace_id`, and `workspace_name`.\n\n`modal.container.terminations` is the exception. It drops `function_name`, `function_id`, and `environment_id`, and it adds `object_id` and `reason`. `reason` is `finished` when the container ended on its own and `USER_CANCELLED` when you stopped it, so group on it to separate normal exits from ones you caused. A filter on `function_name` returns nothing for this metric.\n\n`object_type` tells you what the container was doing. It takes three values:\n\n| Value | Container | \n|---|---|\n| `function` | Runs one of your Functions | \n| `sandbox` | Runs a Sandbox | \n| `image` | Builds an image, before any of your code runs | \n\nFunctions and Sandboxes both report the container metrics. Only Functions report the `modal.input_events.*` and `modal.function.*` metrics, because Sandboxes take no inputs.\n\nEvery container reports the `modal.gpu.*` metrics, and the containers without a GPU report zero. Filter on a Function you know uses a GPU before you average these, or the idle containers drag the number down.\n\n`modal.cpu.utilization` counts cores rather than a share of the container. A value of 1 means one full core, so a container with several cores goes above 1, and a chart in percent shows more than 100.\n\n## ## Troubleshooting\n\n### Modal reports 401 Unauthorized\n\nModal shows an error like this one:\n\n```\nBad Request: Failed to send logs to OTEL: HTTPStatusError(\"Client error\n'401 Unauthorized' for url 'https://ingest.<region>.signoz.cloud/v1/logs'\")\n```\n\n- Likely cause: the Secret key is not `OTEL_HEADER_authorization` , or the ingestion key is wrong.\n- Fix: create the Secret again with the key `OTEL_HEADER_authorization` . Modal sends the header exactly as you name it, and SigNoz rejects any other header name.\n- Verify: click **Send Test** again, then search SigNoz Logs for`modal.test_logs` .\n\n### Modal rejects the Secret key name\n\nModal shows this error when you save the Secret:\n\n```\nSecret key name 'OTEL_HEADER_signoz-ingestion-key' is invalid for environment\nvariables. Only letters, numbers, and underscores are allowed.\n```\n\n- Likely cause: the header name contains a hyphen.\n- Fix: use `OTEL_HEADER_authorization` . See Step 1.\n- Verify: run `modal secret list` and confirm that`signoz-otel` appears.\n\n### The test log arrives but Function logs do not\n\n- Likely cause: nothing ran in Modal after you saved the integration.\n- Fix: run a Function or start a Sandbox. Modal exports only what runs after you turn the integration on.\n- Verify: filter SigNoz Logs on `service.name = modal.function-logs` .\n\n### A metric name returns no data\n\n- Likely cause: the metric is a histogram, or the object type does not report it.\n- Fix: for the `modal.input_events.*_time_us` metrics, add`.count` ,`.sum` ,`.min` ,`.max` , or`.bucket` to the name. For`modal.input_events.*` and`modal.function.*` , make sure that a Function ran, because Sandboxes do not report them.\n- Verify: query `modal.container.running` grouped by`object_type` , which both Functions and Sandboxes report.\n\n### A filter on function_name returns nothing\n\n- Likely cause: the two metric families spell the name differently.\n- Fix: use the module-qualified name such as `modal_load.worker` for the container and`modal.input_events.*` metrics, and the bare name such as`worker` for the`modal.function.*` metrics.\n- Verify: query the metric grouped by `function_name` with no filter, and read the spelling off the result.\n\n### No traces arrive from the caller\n\n- Likely cause: the exporter has no endpoint, or the header name is wrong.\n- Fix: make sure that `OTEL_EXPORTER_OTLP_ENDPOINT` and`OTEL_EXPORTER_OTLP_HEADERS` are set in the environment that runs the code. The header name here is`signoz-ingestion-key` , not`authorization` .\n- Verify: filter SigNoz Traces on the `service.name` you set.\n\n### Traces arrive under the name unknown_service\n\n- Likely cause: `OTEL_SERVICE_NAME` is not set in the environment that runs the code.\n- Fix: export it, or pass the name in code with `TracerProvider(resource=Resource.create({\"service.name\": \"<your-service-name>\"}))` . A name passed in code wins over the environment variable.\n- Verify: filter SigNoz Traces on your service name and confirm that `unknown_service` stops appearing.\n\n### Audit logs do not appear\n\n- Likely cause: Modal restricts audit logs to the Enterprise plan.\n- Fix: contact Modal to enable audit logs for your workspace. Function logs, Sandbox logs, and container metrics work on every plan.\n- Verify: open the audit logs page in your Modal settings.\n\n## ## Limitations\n\n- **No traces from Modal** : the integration sends logs and container metrics. Every span in SigNoz comes from code you instrument.\n- **Sandboxes report fewer metrics** : Sandboxes emit the container metrics, and nothing from`modal.input_events.*` or`modal.function.*` .\n- **No backfill** : the export covers what runs after you save the integration.\n- **Hyphens are not allowed in Secret key names** , which rules out the`signoz-ingestion-key` header. Use`authorization` .\n- **Modal's own collector is in beta** : Modal can route custom spans and metrics through`otlp-collector.modal.local` , but you must ask Modal to enable it for your workspace. Exporting straight to SigNoz needs no such request.\n\n## Next Steps\n\n- [Set up log-based alerts](https://signoz.io/docs/alerts-management/log-based-alerts/) on the error output of your Functions and Sandboxes.\n- [Set up metrics-based alerts](https://signoz.io/docs/alerts-management/metrics-based-alerts/) on cold start time or on GPU memory use.\n- [Import the Modal dashboard](https://signoz.io/docs/dashboards/dashboard-templates/modal-dashboard/) for throughput, cold starts, backlog, CPU, memory, and GPU in one view.\n- [Instrument the rest of your Python code](https://signoz.io/docs/instrumentation/opentelemetry-python/) so the work around your Sandbox calls appears in the same trace.\n\n## Get Help\n\nIf you need help with the steps in this topic, please reach out to us on [SigNoz Community Slack](https://signoz.io/slack/). If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at [cloud-support@signoz.io](mailto:cloud-support@signoz.io).", "url": "https://wpnews.pro/news/monitor-modal-functions-and-sandboxes-with-opentelemetry", "canonical_source": "https://signoz.io/docs/integrations/modal", "published_at": "2026-09-06 00:00:00+00:00", "updated_at": "2026-09-09 07:59:26.879548+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure"], "entities": ["Modal", "SigNoz", "OpenTelemetry"], "alternates": {"html": "https://wpnews.pro/news/monitor-modal-functions-and-sandboxes-with-opentelemetry", "markdown": "https://wpnews.pro/news/monitor-modal-functions-and-sandboxes-with-opentelemetry.md", "text": "https://wpnews.pro/news/monitor-modal-functions-and-sandboxes-with-opentelemetry.txt", "jsonld": "https://wpnews.pro/news/monitor-modal-functions-and-sandboxes-with-opentelemetry.jsonld"}}