Monitor Modal Functions and Sandboxes with OpenTelemetry

Overview #

Modal runs Python functions and Sandboxes on demand. A Sandbox is a container that Modal creates at runtime to execute untrusted or AI-generated code. Your agent creates a Sandbox, runs commands in it, reads the output, and terminates it.

Two things go wrong in that loop, and each needs its own telemetry. The code inside the Sandbox fails or hangs, and the Sandbox itself starts slowly or runs out of memory.

Prerequisites #

• An instance of SigNoz (either Cloud orSelf-Hosted )
• A SigNoz ingestion key
• A Modal account , with permission to edit workspace settings
• The modal CLI, installed withpip install modal and authenticated withmodal setup

How it works #

Modal offers three OpenTelemetry paths. They do not overlap, so pick the ones that answer your question.

Path	What you get	What it costs you
Modal OpenTelemetry integration	Function logs, Sandbox logs, and container metrics for the whole workspace	One workspace setting
Tracing in the caller	Latency and failures for every Sandbox.create and every command you run	Code in the application that drives the Sandboxes
Tracing inside the Sandbox	Spans and logs from the code your agent runs	Packages and a Secret in the Sandbox image

Start with the integration. It needs no code changes, and it covers every Function and every Sandbox in the workspace. It sends no traces, so add the second path when you need to know which command an agent ran, how long it took, and whether it failed.

Monitoring Modal #

The Modal OpenTelemetry integration sends Function logs, Sandbox logs, and container metrics to any backend that accepts OTLP over HTTP. You give Modal two things: the base URL of your SigNoz endpoint, and a Modal Secret that holds the authentication header.

Step 1: Create the Modal Secret

Modal builds the request headers from a Secret. Each key starts with OTEL_HEADER_, and the rest of the key is the header name. Use the authorization header, which SigNoz accepts with the ingestion key as its value:

modal secret create signoz-otel \
  OTEL_HEADER_authorization=<your-ingestion-key>

You can create the same Secret from the Modal Secrets page with the OpenTelemetry template.

Verify these values:

• <your-ingestion-key> : Your SigNozingestion key .

Step 2: Point Modal at your SigNoz endpoint

1. Go to the Modal metrics settings page .
2. Set the OpenTelemetry push URL to https://ingest.<region>.signoz.cloud .
3. Select the signoz-otel Secret that you created in Step 1.
4. Save the changes.

Verify these values:

• <region> : YourSigNoz Cloud region .

Step 3: Test the connection

Click Send Test on the metrics settings page. Modal sends one log line to your endpoint and reports the result.

Open Logs in SigNoz and search for Hello from Modal!. The test log arrives with service.name = modal.test_logs. If you see it, the push URL and the ingestion key both work.

Modal starts the export when you save the integration. Modal does not backfill older logs.

The integration tells you that a Sandbox ran. It does not tell you which command an agent executed, how long each step took, or which run failed. Add the OpenTelemetry SDK to the code that drives the Sandboxes to get that.

This path instruments the caller, which is the script or service that calls Sandbox.create. The Sandbox itself needs no changes.

Step 1: Install the packages

pip install modal opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

Step 2: Configure the exporter

The exporter reads its endpoint and headers from the environment:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.<region>.signoz.cloud:443"
export OTEL_EXPORTER_OTLP_HEADERS="signoz-ingestion-key=<your-ingestion-key>"
export OTEL_SERVICE_NAME="<your-service-name>"

Verify these values:

• <region> : YourSigNoz Cloud region .
• <your-ingestion-key> : Your SigNozingestion key .
• <your-service-name> : The name this code appears under in SigNoz, such asmodal-sandbox-runner .

The Python SDK reads all three, so the code below passes no endpoint, header, or service name of its own.

Step 3: Wrap the Sandbox calls in spans

Create one span for the Sandbox and one span for each command it runs. Record the exit code, and mark the span as an error when the command fails:

import modal
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.trace import Status, StatusCode
 
provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("modal.sandbox")
 
 
def run_in_sandbox(app, command):
    with tracer.start_as_current_span("sandbox.create") as span:
        sandbox = modal.Sandbox.create(app=app, timeout=120)
        span.set_attribute("modal.sandbox.id", sandbox.object_id)
 
    try:
        with tracer.start_as_current_span("sandbox.exec") as span:
            span.set_attribute("modal.sandbox.id", sandbox.object_id)
            span.set_attribute("modal.sandbox.command", " ".join(command))
            process = sandbox.exec(*command)
            output = process.stdout.read()
            process.wait()
            span.set_attribute("modal.sandbox.exit_code", process.returncode)
            if process.returncode != 0:
                span.set_status(
                    Status(StatusCode.ERROR, f"exit code {process.returncode}")
                )
            return output
    finally:
        sandbox.terminate()
 
 
if __name__ == "__main__":
    app = modal.App.lookup("agent-sandboxes", create_if_missing=True)
    with tracer.start_as_current_span("agent.run"):
        print(run_in_sandbox(app, ["python", "-c", "print('hello')"]))
        print(run_in_sandbox(app, ["python", "-c", "raise SystemExit(3)"]))
    provider.shutdown()

sandbox.object_id is the Sandbox ID, such as sb-QHSNZOpAvp2kkGxd9gVaG0. The logs carry the same ID in their sandbox_id attribute, so this one attribute joins a trace to the output of that Sandbox.

Run the script:

python agent.py

Validate #

Run a Modal Function or start a Sandbox first, so that Modal has telemetry to send.

For the integration:

1. Open Logs in SigNoz and filter onservice.name = modal.function-logs . Your Function and Sandbox output arrives within a minute.
2. Open Metrics in SigNoz and search formodal. . Querymodal.cpu.utilization and group byobject_type to see Functions and Sandboxes side by side.

For the SDK:

1. Open Traces in SigNoz and filter on theservice.name you set inOTEL_SERVICE_NAME . The screenshots below usemodal-sandbox-runner .
2. Open the agent.run trace. It holds asandbox.create span and asandbox.exec span for each run.
3. The second sandbox.exec span is red, with the status messageexit code 3 .
4. Copy a modal.sandbox.id value and searchLogs forsandbox_id = '<that value>' to read what the Sandbox printed.

## Run the Caller as a Modal Function #

When the code that creates Sandboxes runs on Modal itself, pass the exporter settings through a Secret:

modal secret create signoz-otlp \
  OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.<region>.signoz.cloud:443 \
  "OTEL_EXPORTER_OTLP_HEADERS=signoz-ingestion-key=<your-ingestion-key>" \
  OTEL_SERVICE_NAME=<your-service-name>

The Secret key restriction applies only to key names. All three names here are plain, and the hyphens in signoz-ingestion-key sit inside the value, which Modal accepts.

Attach the Secret and install the packages in the image:

image = modal.Image.debian_slim().pip_install(
    "opentelemetry-sdk",
    "opentelemetry-exporter-otlp-proto-http",
)
 
@app.function(image=image, secrets=[modal.Secret.from_name("signoz-otlp")])
def run_agent():
    ...

## Trace the Code Inside a Sandbox #

The spans in the caller time the Sandbox from outside. To see what the code inside it did, install the packages in the Sandbox image and pass the same Secret to Sandbox.create:

image = modal.Image.debian_slim().pip_install(
    "opentelemetry-sdk",
    "opentelemetry-exporter-otlp-proto-http",
)
 
sandbox = modal.Sandbox.create(
    app=app,
    image=image,
    secrets=[modal.Secret.from_name("signoz-otlp")],
    timeout=180,
)

The code inside then configures a TracerProvider the same way the caller does, and calls provider.force_flush() before it finishes.

## Attribute and Metric Reference #

Logs

Function output and Sandbox output both arrive under service.name = modal.function-logs. The test log is the only exception, and it uses service.name = modal.test_logs.

Every log record carries app_id, app_name, container_id, environment, file_descriptor, level, workspace, and workspace_id. The rest depend on what produced the log:

Attribute	Present on
function_name ,function_id ,function_call_id ,input_id	Function logs
sandbox_id	Sandbox logs

To read Sandbox output only, filter on sandbox_id EXISTS. To read one Sandbox, filter on the ID that Sandbox.create returned, for example sandbox_id = 'sb-QHSNZOpAvp2kkGxd9gVaG0'.

file_descriptor is 1 for stdout and 2 for stderr. Modal sends both with severity_text = INFO, so use file_descriptor = '2' to find error output.

Metrics

Metric	What it measures
modal.cpu.utilization	CPU use of the container, where 1 means one full core
modal.memory.usage	Memory use of the container, in bytes
modal.container.running	1 while the container runs
modal.container.terminations	Count of container terminations
modal.gpu.compute.utilization	GPU compute use, as a fraction between 0 and 1
modal.gpu.memory.usage	GPU memory use, in bytes
modal.gpu.clock	GPU SM clock frequency, in MHz
modal.gpu.power.usage	GPU power draw, in watts
modal.gpu.temperature	GPU temperature, in degrees Celsius
modal.input_events.elapsed_time_us	Time to handle one input, in microseconds
modal.input_events.input_queue_time_us	Time an input waited in the queue, in microseconds
modal.input_events.coldstart_time_us	Cold start time, in microseconds
modal.input_events.successes	Count of inputs that succeeded
modal.input_events.total_inputs	Count of inputs received
modal.function.pending_inputs	Inputs that wait for a container
modal.function.running_inputs	Inputs that a container handles now

Divide modal.input_events.successes by modal.input_events.total_inputs to get a success rate for a Function.

Most metrics carry app_id, app_name, container_id, environment_id, environment_name, object_type, workspace_id, and workspace_name.

modal.container.terminations is the exception. It drops function_name, function_id, and environment_id, and it adds object_id and reason. reason is finished when the container ended on its own and USER_CANCELLED when you stopped it, so group on it to separate normal exits from ones you caused. A filter on function_name returns nothing for this metric.

object_type tells you what the container was doing. It takes three values:

Value	Container
function	Runs one of your Functions
sandbox	Runs a Sandbox
image	Builds an image, before any of your code runs

Functions and Sandboxes both report the container metrics. Only Functions report the modal.input_events.* and modal.function.* metrics, because Sandboxes take no inputs.

Every container reports the modal.gpu.* metrics, and the containers without a GPU report zero. Filter on a Function you know uses a GPU before you average these, or the idle containers drag the number down.

modal.cpu.utilization counts cores rather than a share of the container. A value of 1 means one full core, so a container with several cores goes above 1, and a chart in percent shows more than 100.

## Troubleshooting #

Modal reports 401 Unauthorized

Modal shows an error like this one:

Bad Request: Failed to send logs to OTEL: HTTPStatusError("Client error
'401 Unauthorized' for url 'https://ingest.<region>.signoz.cloud/v1/logs'")

• Likely cause: the Secret key is not OTEL_HEADER_authorization , or the ingestion key is wrong.
• Fix: create the Secret again with the key OTEL_HEADER_authorization . Modal sends the header exactly as you name it, and SigNoz rejects any other header name.
• Verify: click Send Test again, then search SigNoz Logs formodal.test_logs .

Modal rejects the Secret key name

Modal shows this error when you save the Secret:

Secret key name 'OTEL_HEADER_signoz-ingestion-key' is invalid for environment
variables. Only letters, numbers, and underscores are allowed.

• Likely cause: the header name contains a hyphen.
• Fix: use OTEL_HEADER_authorization . See Step 1.
• Verify: run modal secret list and confirm thatsignoz-otel appears.

The test log arrives but Function logs do not

• Likely cause: nothing ran in Modal after you saved the integration.
• Fix: run a Function or start a Sandbox. Modal exports only what runs after you turn the integration on.
• Verify: filter SigNoz Logs on service.name = modal.function-logs .

A metric name returns no data

• Likely cause: the metric is a histogram, or the object type does not report it.
• Fix: for the modal.input_events.*_time_us metrics, add.count ,.sum ,.min ,.max , or.bucket to the name. Formodal.input_events.* andmodal.function.* , make sure that a Function ran, because Sandboxes do not report them.
• Verify: query modal.container.running grouped byobject_type , which both Functions and Sandboxes report.

A filter on function_name returns nothing

• Likely cause: the two metric families spell the name differently.
• Fix: use the module-qualified name such as modal_load.worker for the container andmodal.input_events.* metrics, and the bare name such asworker for themodal.function.* metrics.
• Verify: query the metric grouped by function_name with no filter, and read the spelling off the result.

No traces arrive from the caller

• Likely cause: the exporter has no endpoint, or the header name is wrong.
• Fix: make sure that OTEL_EXPORTER_OTLP_ENDPOINT andOTEL_EXPORTER_OTLP_HEADERS are set in the environment that runs the code. The header name here issignoz-ingestion-key , notauthorization .
• Verify: filter SigNoz Traces on the service.name you set.

Traces arrive under the name unknown_service

• Likely cause: OTEL_SERVICE_NAME is not set in the environment that runs the code.
• Fix: export it, or pass the name in code with TracerProvider(resource=Resource.create({"service.name": "<your-service-name>"})) . A name passed in code wins over the environment variable.
• Verify: filter SigNoz Traces on your service name and confirm that unknown_service stops appearing.

Audit logs do not appear

• Likely cause: Modal restricts audit logs to the Enterprise plan.
• Fix: contact Modal to enable audit logs for your workspace. Function logs, Sandbox logs, and container metrics work on every plan.
• Verify: open the audit logs page in your Modal settings.

## Limitations #

• No traces from Modal : the integration sends logs and container metrics. Every span in SigNoz comes from code you instrument.
• Sandboxes report fewer metrics : Sandboxes emit the container metrics, and nothing frommodal.input_events.* ormodal.function.* .
• No backfill : the export covers what runs after you save the integration.
• Hyphens are not allowed in Secret key names , which rules out thesignoz-ingestion-key header. Useauthorization .
• Modal's own collector is in beta : Modal can route custom spans and metrics throughotlp-collector.modal.local , but you must ask Modal to enable it for your workspace. Exporting straight to SigNoz needs no such request.

Next Steps #

• Set up log-based alerts on the error output of your Functions and Sandboxes.
• Set up metrics-based alerts on cold start time or on GPU memory use.
• Import the Modal dashboard for throughput, cold starts, backlog, CPU, memory, and GPU in one view.
• Instrument the rest of your Python code so the work around your Sandbox calls appears in the same trace.

Get Help #

If you need help with the steps in this topic, please reach out to us on SigNoz Community Slack. If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at cloud-support@signoz.io.

source & further reading

signoz.io — original article Daytona Sandbox Monitoring & Tracing with OpenTelemetry Antigravity CLI Dashboard Google Antigravity CLI Monitoring with OpenTelemetry