Monitor Modal Functions and Sandboxes with OpenTelemetry

SigNoz has published a guide for monitoring Modal functions and Sandboxes with OpenTelemetry, detailing three non-overlapping telemetry paths: the Modal OpenTelemetry integration for workspace-wide logs and metrics, tracing in the caller for per-command latency and failures, and tracing inside the Sandbox for code-level spans. The integration requires creating a Modal Secret with an authorization header containing the SigNoz ingestion key, setting the OpenTelemetry push URL to the SigNoz endpoint, and testing the connection with a test log that appears in SigNoz as 'Hello from Modal!' with service.name 'modal.test_logs'.

Overview Modal https://modal.com runs Python functions and Sandboxes on demand. A Sandbox is a container that Modal creates at runtime to execute untrusted or AI-generated code. Your agent creates a Sandbox, runs commands in it, reads the output, and terminates it. Two things go wrong in that loop, and each needs its own telemetry. The code inside the Sandbox fails or hangs, and the Sandbox itself starts slowly or runs out of memory. Prerequisites - An instance of SigNoz either Cloud https://signoz.io/teams/ or Self-Hosted https://signoz.io/docs/install/self-host/ - A SigNoz ingestion key https://signoz.io/docs/ingestion/signoz-cloud/keys/ - A Modal account https://modal.com/signup , with permission to edit workspace settings - The modal CLI, installed with pip install modal and authenticated with modal setup How it works Modal offers three OpenTelemetry paths. They do not overlap, so pick the ones that answer your question. | Path | What you get | What it costs you | |---|---|---| | Modal OpenTelemetry integration | Function logs, Sandbox logs, and container metrics for the whole workspace | One workspace setting | | Tracing in the caller | Latency and failures for every Sandbox.create and every command you run | Code in the application that drives the Sandboxes | | Tracing inside the Sandbox | Spans and logs from the code your agent runs | Packages and a Secret in the Sandbox image | Start with the integration. It needs no code changes, and it covers every Function and every Sandbox in the workspace. It sends no traces, so add the second path when you need to know which command an agent ran, how long it took, and whether it failed. Monitoring Modal The Modal OpenTelemetry integration sends Function logs, Sandbox logs, and container metrics to any backend that accepts OTLP over HTTP. You give Modal two things: the base URL of your SigNoz endpoint, and a Modal Secret that holds the authentication header. Step 1: Create the Modal Secret Modal builds the request headers from a Secret. Each key starts with OTEL HEADER , and the rest of the key is the header name. Use the authorization header, which SigNoz accepts with the ingestion key as its value: modal secret create signoz-otel \ OTEL HEADER authorization=<your-ingestion-key You can create the same Secret from the Modal Secrets page https://modal.com/secrets with the OpenTelemetry template. Verify these values: - <your-ingestion-key : Your SigNoz ingestion key https://signoz.io/docs/ingestion/signoz-cloud/keys/ . Step 2: Point Modal at your SigNoz endpoint 1. Go to the Modal metrics settings page https://modal.com/settings/metrics . 2. Set the OpenTelemetry push URL to https://ingest.<region .signoz.cloud . 3. Select the signoz-otel Secret that you created in Step 1. 4. Save the changes. Verify these values: - <region : Your SigNoz Cloud region https://signoz.io/docs/ingestion/signoz-cloud/overview/ endpoint . Step 3: Test the connection Click Send Test on the metrics settings page. Modal sends one log line to your endpoint and reports the result. Open Logs in SigNoz and search for Hello from Modal . The test log arrives with service.name = modal.test logs . If you see it, the push URL and the ingestion key both work. Modal starts the export when you save the integration. Modal does not backfill older logs. The integration tells you that a Sandbox ran. It does not tell you which command an agent executed, how long each step took, or which run failed. Add the OpenTelemetry SDK to the code that drives the Sandboxes to get that. This path instruments the caller, which is the script or service that calls Sandbox.create . The Sandbox itself needs no changes. Step 1: Install the packages pip install modal opentelemetry-sdk opentelemetry-exporter-otlp-proto-http Step 2: Configure the exporter The exporter reads its endpoint and headers from the environment: export OTEL EXPORTER OTLP ENDPOINT="https://ingest.<region .signoz.cloud:443" export OTEL EXPORTER OTLP HEADERS="signoz-ingestion-key=<your-ingestion-key " export OTEL SERVICE NAME="<your-service-name " Verify these values: - <region : Your SigNoz Cloud region https://signoz.io/docs/ingestion/signoz-cloud/overview/ endpoint . - <your-ingestion-key : Your SigNoz ingestion key https://signoz.io/docs/ingestion/signoz-cloud/keys/ . - <your-service-name : The name this code appears under in SigNoz, such as modal-sandbox-runner . The Python SDK reads all three, so the code below passes no endpoint, header, or service name of its own. Step 3: Wrap the Sandbox calls in spans Create one span for the Sandbox and one span for each command it runs. Record the exit code, and mark the span as an error when the command fails: python import modal from opentelemetry import trace from opentelemetry.exporter.otlp.proto.http.trace exporter import OTLPSpanExporter from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.trace import Status, StatusCode The endpoint, headers, and service name all come from the environment provider = TracerProvider provider.add span processor BatchSpanProcessor OTLPSpanExporter trace.set tracer provider provider tracer = trace.get tracer "modal.sandbox" def run in sandbox app, command : with tracer.start as current span "sandbox.create" as span: sandbox = modal.Sandbox.create app=app, timeout=120 span.set attribute "modal.sandbox.id", sandbox.object id try: with tracer.start as current span "sandbox.exec" as span: span.set attribute "modal.sandbox.id", sandbox.object id span.set attribute "modal.sandbox.command", " ".join command process = sandbox.exec command output = process.stdout.read process.wait span.set attribute "modal.sandbox.exit code", process.returncode if process.returncode = 0: span.set status Status StatusCode.ERROR, f"exit code {process.returncode}" return output finally: sandbox.terminate if name == " main ": app = modal.App.lookup "agent-sandboxes", create if missing=True with tracer.start as current span "agent.run" : print run in sandbox app, "python", "-c", "print 'hello' " print run in sandbox app, "python", "-c", "raise SystemExit 3 " provider.shutdown sandbox.object id is the Sandbox ID, such as sb-QHSNZOpAvp2kkGxd9gVaG0 . The logs carry the same ID in their sandbox id attribute, so this one attribute joins a trace to the output of that Sandbox. Run the script: python agent.py Validate Run a Modal Function or start a Sandbox first, so that Modal has telemetry to send. For the integration: 1. Open Logs in SigNoz and filter on service.name = modal.function-logs . Your Function and Sandbox output arrives within a minute. 2. Open Metrics in SigNoz and search for modal. . Query modal.cpu.utilization and group by object type to see Functions and Sandboxes side by side. For the SDK: 1. Open Traces in SigNoz and filter on the service.name you set in OTEL SERVICE NAME . The screenshots below use modal-sandbox-runner . 2. Open the agent.run trace. It holds a sandbox.create span and a sandbox.exec span for each run. 3. The second sandbox.exec span is red, with the status message exit code 3 . 4. Copy a modal.sandbox.id value and search Logs for sandbox id = '<that value ' to read what the Sandbox printed. Run the Caller as a Modal Function When the code that creates Sandboxes runs on Modal itself, pass the exporter settings through a Secret: modal secret create signoz-otlp \ OTEL EXPORTER OTLP ENDPOINT=https://ingest.<region .signoz.cloud:443 \ "OTEL EXPORTER OTLP HEADERS=signoz-ingestion-key=<your-ingestion-key " \ OTEL SERVICE NAME=<your-service-name The Secret key restriction applies only to key names. All three names here are plain, and the hyphens in signoz-ingestion-key sit inside the value, which Modal accepts. Attach the Secret and install the packages in the image: image = modal.Image.debian slim .pip install "opentelemetry-sdk", "opentelemetry-exporter-otlp-proto-http", @app.function image=image, secrets= modal.Secret.from name "signoz-otlp" def run agent : ... Trace the Code Inside a Sandbox The spans in the caller time the Sandbox from outside. To see what the code inside it did, install the packages in the Sandbox image and pass the same Secret to Sandbox.create : image = modal.Image.debian slim .pip install "opentelemetry-sdk", "opentelemetry-exporter-otlp-proto-http", sandbox = modal.Sandbox.create app=app, image=image, secrets= modal.Secret.from name "signoz-otlp" , timeout=180, The code inside then configures a TracerProvider the same way the caller does, and calls provider.force flush before it finishes. Attribute and Metric Reference Logs Function output and Sandbox output both arrive under service.name = modal.function-logs . The test log is the only exception, and it uses service.name = modal.test logs . Every log record carries app id , app name , container id , environment , file descriptor , level , workspace , and workspace id . The rest depend on what produced the log: | Attribute | Present on | |---|---| | function name , function id , function call id , input id | Function logs | | sandbox id | Sandbox logs | To read Sandbox output only, filter on sandbox id EXISTS . To read one Sandbox, filter on the ID that Sandbox.create returned, for example sandbox id = 'sb-QHSNZOpAvp2kkGxd9gVaG0' . file descriptor is 1 for stdout and 2 for stderr. Modal sends both with severity text = INFO , so use file descriptor = '2' to find error output. Metrics | Metric | What it measures | |---|---| | modal.cpu.utilization | CPU use of the container, where 1 means one full core | | modal.memory.usage | Memory use of the container, in bytes | | modal.container.running | 1 while the container runs | | modal.container.terminations | Count of container terminations | | modal.gpu.compute.utilization | GPU compute use, as a fraction between 0 and 1 | | modal.gpu.memory.usage | GPU memory use, in bytes | | modal.gpu.clock | GPU SM clock frequency, in MHz | | modal.gpu.power.usage | GPU power draw, in watts | | modal.gpu.temperature | GPU temperature, in degrees Celsius | | modal.input events.elapsed time us | Time to handle one input, in microseconds | | modal.input events.input queue time us | Time an input waited in the queue, in microseconds | | modal.input events.coldstart time us | Cold start time, in microseconds | | modal.input events.successes | Count of inputs that succeeded | | modal.input events.total inputs | Count of inputs received | | modal.function.pending inputs | Inputs that wait for a container | | modal.function.running inputs | Inputs that a container handles now | Divide modal.input events.successes by modal.input events.total inputs to get a success rate for a Function. Most metrics carry app id , app name , container id , environment id , environment name , object type , workspace id , and workspace name . modal.container.terminations is the exception. It drops function name , function id , and environment id , and it adds object id and reason . reason is finished when the container ended on its own and USER CANCELLED when you stopped it, so group on it to separate normal exits from ones you caused. A filter on function name returns nothing for this metric. object type tells you what the container was doing. It takes three values: | Value | Container | |---|---| | function | Runs one of your Functions | | sandbox | Runs a Sandbox | | image | Builds an image, before any of your code runs | Functions and Sandboxes both report the container metrics. Only Functions report the modal.input events. and modal.function. metrics, because Sandboxes take no inputs. Every container reports the modal.gpu. metrics, and the containers without a GPU report zero. Filter on a Function you know uses a GPU before you average these, or the idle containers drag the number down. modal.cpu.utilization counts cores rather than a share of the container. A value of 1 means one full core, so a container with several cores goes above 1, and a chart in percent shows more than 100. Troubleshooting Modal reports 401 Unauthorized Modal shows an error like this one: Bad Request: Failed to send logs to OTEL: HTTPStatusError "Client error '401 Unauthorized' for url 'https://ingest.<region .signoz.cloud/v1/logs'" - Likely cause: the Secret key is not OTEL HEADER authorization , or the ingestion key is wrong. - Fix: create the Secret again with the key OTEL HEADER authorization . Modal sends the header exactly as you name it, and SigNoz rejects any other header name. - Verify: click Send Test again, then search SigNoz Logs for modal.test logs . Modal rejects the Secret key name Modal shows this error when you save the Secret: Secret key name 'OTEL HEADER signoz-ingestion-key' is invalid for environment variables. Only letters, numbers, and underscores are allowed. - Likely cause: the header name contains a hyphen. - Fix: use OTEL HEADER authorization . See Step 1. - Verify: run modal secret list and confirm that signoz-otel appears. The test log arrives but Function logs do not - Likely cause: nothing ran in Modal after you saved the integration. - Fix: run a Function or start a Sandbox. Modal exports only what runs after you turn the integration on. - Verify: filter SigNoz Logs on service.name = modal.function-logs . A metric name returns no data - Likely cause: the metric is a histogram, or the object type does not report it. - Fix: for the modal.input events. time us metrics, add .count , .sum , .min , .max , or .bucket to the name. For modal.input events. and modal.function. , make sure that a Function ran, because Sandboxes do not report them. - Verify: query modal.container.running grouped by object type , which both Functions and Sandboxes report. A filter on function name returns nothing - Likely cause: the two metric families spell the name differently. - Fix: use the module-qualified name such as modal load.worker for the container and modal.input events. metrics, and the bare name such as worker for the modal.function. metrics. - Verify: query the metric grouped by function name with no filter, and read the spelling off the result. No traces arrive from the caller - Likely cause: the exporter has no endpoint, or the header name is wrong. - Fix: make sure that OTEL EXPORTER OTLP ENDPOINT and OTEL EXPORTER OTLP HEADERS are set in the environment that runs the code. The header name here is signoz-ingestion-key , not authorization . - Verify: filter SigNoz Traces on the service.name you set. Traces arrive under the name unknown service - Likely cause: OTEL SERVICE NAME is not set in the environment that runs the code. - Fix: export it, or pass the name in code with TracerProvider resource=Resource.create {"service.name": "<your-service-name "} . A name passed in code wins over the environment variable. - Verify: filter SigNoz Traces on your service name and confirm that unknown service stops appearing. Audit logs do not appear - Likely cause: Modal restricts audit logs to the Enterprise plan. - Fix: contact Modal to enable audit logs for your workspace. Function logs, Sandbox logs, and container metrics work on every plan. - Verify: open the audit logs page in your Modal settings. Limitations - No traces from Modal : the integration sends logs and container metrics. Every span in SigNoz comes from code you instrument. - Sandboxes report fewer metrics : Sandboxes emit the container metrics, and nothing from modal.input events. or modal.function. . - No backfill : the export covers what runs after you save the integration. - Hyphens are not allowed in Secret key names , which rules out the signoz-ingestion-key header. Use authorization . - Modal's own collector is in beta : Modal can route custom spans and metrics through otlp-collector.modal.local , but you must ask Modal to enable it for your workspace. Exporting straight to SigNoz needs no such request. Next Steps - Set up log-based alerts https://signoz.io/docs/alerts-management/log-based-alerts/ on the error output of your Functions and Sandboxes. - Set up metrics-based alerts https://signoz.io/docs/alerts-management/metrics-based-alerts/ on cold start time or on GPU memory use. - Import the Modal dashboard https://signoz.io/docs/dashboards/dashboard-templates/modal-dashboard/ for throughput, cold starts, backlog, CPU, memory, and GPU in one view. - Instrument the rest of your Python code https://signoz.io/docs/instrumentation/opentelemetry-python/ so the work around your Sandbox calls appears in the same trace. Get Help If you need help with the steps in this topic, please reach out to us on SigNoz Community Slack https://signoz.io/slack/ . If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at cloud-support@signoz.io mailto:cloud-support@signoz.io .