# Monitor AWS Lambda MicroVMs with OpenTelemetry Collector > Source: > Published: 2026-08-20 00:00:00+00:00 ## Overview [AWS Lambda MicroVMs](https://aws.amazon.com/lambda/lambda-microvms/) are Firecracker-based compute environments with VM-level isolation, snapshot start and resume, and full OS capabilities. They suit long-running, stateful sandboxes such as AI agent code execution, interactive development environments, CI jobs, and vulnerability scanners. This guide runs an OpenTelemetry Collector inside the MicroVM image. The Collector scrapes host metrics, tails application logs, receives OTLP from your application, and exports everything to SigNoz. ## Prerequisites - The AWS CLI with the `lambda-microvms` command set. Run`aws lambda-microvms help` to confirm it lists`create-microvm-image` and`run-microvm` . - An S3 bucket in the same region for the code artifact. - A **build role** that Lambda assumes during the image build. It needs`s3:GetObject` on the artifact and`logs:CreateLogGroup` ,`logs:CreateLogStream` , and`logs:PutLogEvents` . - An **execution role** for the running MicroVM. Without it, application stdout and stderr never reach CloudWatch. It needs`logs:CreateLogGroup` ,`logs:CreateLogStream` , and`logs:PutLogEvents` . - Both roles need a trust policy that lets Lambda assume them, with `Principal: {"Service": "lambda.amazonaws.com"}` and`Action: ["sts:AssumeRole", "sts:TagSession"]` . Your own caller needs`lambda:CreateMicrovmImage` ,`lambda:RunMicrovm` , and`iam:PassRole` for both roles. - An instance of SigNoz (either [Cloud](https://signoz.io/teams/)or[Self-Hosted](https://signoz.io/docs/install/self-host/)) ## How monitoring works | Signal | Source inside the MicroVM | Path to SigNoz | |---|---|---| Metrics (host CPU, memory, disk, network) | `hostmetrics` | **Traces****Logs**[receiver, or OTLP from the SDK](https://signoz.io/docs/userguide/collect_logs_from_file/)`filelog` **Lifecycle and audit**## How sizing works MicroVMs use a baseline and peak model. You set the baseline with the memory value when you create the image, and vCPU scales with it at 2 GB per vCPU. During activity the MicroVM scales vertically up to four times the baseline. | Baseline | Peak | Max disk | |---|---|---| | 0.5 GB, 0.25 vCPU | 2 GB, 1 vCPU | 8 GB | | 1 GB, 0.5 vCPU | 4 GB, 2 vCPU | 8 GB | | 2 GB, 1 vCPU (default) | 8 GB, 4 vCPU | 8 GB | | 4 GB, 2 vCPU | 16 GB, 8 vCPU | 16 GB | | 8 GB, 4 vCPU | 32 GB, 16 vCPU | 32 GB | A MicroVM runs for at most 8 hours. Set the limit with `--maximum-duration-in-seconds` , which accepts 1 to 28,800 seconds. ## Send telemetry to SigNoz Lambda builds the MicroVM image from a ZIP you upload to S3, then snapshots the running result. The Collector is baked into that snapshot, so it is already running the moment a MicroVM starts. Your application exports to it on `localhost` instead of reaching SigNoz directly. ### Step 1: Package the application and Dockerfile The code artifact is a ZIP that contains a `Dockerfile` at the archive root plus your application files. Lambda pulls the ZIP from S3, runs your `Dockerfile` on top of the managed base image, starts your application, and snapshots the result. Two different base images are involved. The **MicroVM base image** is the operating system environment, passed as `--base-image-arn` . The **container base image** is what your `Dockerfile` uses in its `FROM` instruction. ``` FROM public.ecr.aws/lambda/microvms:al2023-minimal # al2023-minimal ships without tar, gzip, or procps. Install what you need # before extracting anything. RUN dnf install -y tar gzip python3 python3-pip && dnf clean all # Release assets are version-named, and MicroVMs are ARM64 only. Pin a version # and use the arm64 tarball. ARG OTELCOL_VERSION=0.159.0 ADD https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${OTELCOL_VERSION}/otelcol-contrib_${OTELCOL_VERSION}_linux_arm64.tar.gz /tmp/otelcol.tar.gz RUN mkdir -p /usr/local/bin \ && tar -xzf /tmp/otelcol.tar.gz -C /usr/local/bin otelcol-contrib \ && rm /tmp/otelcol.tar.gz COPY app.py /app/app.py COPY otel-collector-config.yaml /etc/otelcol/config.yaml COPY entrypoint.sh /entrypoint.sh RUN chmod +x /entrypoint.sh && mkdir -p /var/log/app CMD ["/entrypoint.sh"] ``` The image build has outbound internet access, so pulling the Collector with `ADD` works. Start the Collector in the background and your application in the foreground, which keeps the MicroVM alive: ``` entrypoint.sh bash #!/usr/bin/env bash set -euo pipefail # Lambda forwards only stdout and stderr to CloudWatch, so do not redirect the # Collector to a file. Writing it into /var/log/app would also make the filelog # receiver tail the Collector's own output. /usr/local/bin/otelcol-contrib --config /etc/otelcol/config.yaml 2>&1 & exec python3 -u /app/app.py ``` ### Step 2: Configure the OpenTelemetry Collector ``` otel-collector-config.yaml receivers: # On Collector v0.151.0 and newer, use "host_metrics" to avoid a deprecation warning. hostmetrics: collection_interval: 30s scrapers: cpu: metrics: # system.cpu.utilization is an optional metric and off by default. # Enable it explicitly or it never reaches SigNoz. system.cpu.utilization: enabled: true memory: {} load: {} filesystem: {} network: {} # On Collector v0.149.0 and newer, use "file_log" to avoid a deprecation warning. # This only collects anything if your application writes to this path. An # application that logs to stdout alone produces no records here. # # Keep the Collector's own log out of this glob. A Collector that tails its own # output re-exports every line it writes. filelog: include: [/var/log/app/*.log] exclude: [/var/log/app/otelcol*.log] start_at: beginning otlp: protocols: http: endpoint: localhost:4318 processors: batch: {} # On Collector v0.153.0 and newer, use "resource_detection" to avoid a deprecation warning. resourcedetection: detectors: [env, system] resource: attributes: # Lambda injects this into every MicroVM. It is the same value the # CloudWatch Agent uses for its ImageName dimension. - key: aws.lambda.microvm.image_name value: ${env:AWS_LAMBDA_MICROVM_IMAGE_NAME} action: upsert - key: aws.lambda.microvm.image_version value: ${env:AWS_LAMBDA_MICROVM_IMAGE_VERSION} action: upsert exporters: # On Collector v0.144.0 and newer, use "otlp_http" to avoid a deprecation warning. otlphttp: endpoint: ${env:SIGNOZ_ENDPOINT} headers: signoz-ingestion-key: ${env:SIGNOZ_INGESTION_KEY} service: pipelines: metrics: receivers: [hostmetrics, otlp] processors: [resourcedetection, resource, batch] exporters: [otlphttp] traces: receivers: [otlp] processors: [resourcedetection, resource, batch] exporters: [otlphttp] logs: receivers: [filelog, otlp] processors: [resourcedetection, resource, batch] exporters: [otlphttp] ``` Lambda injects `AWS_LAMBDA_MICROVM_IMAGE_NAME` , `AWS_LAMBDA_MICROVM_IMAGE_ARN` , `AWS_LAMBDA_MICROVM_IMAGE_VERSION` , and `AWS_REGION` into every MicroVM. Using the image name as a resource attribute lets you filter and group all three signals by image in SigNoz. ### Step 3: Instrument your application Because the MicroVM is long-lived, use standard service-style instrumentation rather than a Lambda layer. Point the SDK at the embedded Collector with these variables: ``` OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318" OTEL_SERVICE_NAME="my-microvm-app" OTEL_RESOURCE_ATTRIBUTES="deployment.environment=production" ``` There is no environment variable flag on `run-microvm` , so pass these to `--environment-variables` when you build the image in [Step 4](#step-4-build-the-microvm-image). Setting them anywhere else has no effect. See the [SigNoz instrumentation guides](https://signoz.io/docs/instrumentation/) for your language. Write your application logs to `/var/log/app/` for the `filelog` receiver from [Step 2](#step-2-configure-the-opentelemetry-collector) to pick them up, or remove that receiver and export logs over OTLP from the SDK. Your directory now holds everything the build needs. Zip it with the `Dockerfile` at the archive root and upload it: ``` zip -r app.zip Dockerfile entrypoint.sh otel-collector-config.yaml app.py aws s3 cp app.zip s3:///app.zip ``` **Verify these values:** `` : The S3 bucket you created for the code artifact. ### Step 4: Build the MicroVM image Export your SigNoz destination so the build command can read it: ``` export SIGNOZ_ENDPOINT="https://ingest..signoz.cloud:443" export SIGNOZ_INGESTION_KEY="" ``` Then create the image: ``` aws lambda-microvms create-microvm-image \ --name my-monitored-app \ --code-artifact uri=s3:///app.zip \ --base-image-arn arn:aws:lambda::aws:microvm-image:al2023-1 \ --build-role-arn arn:aws:iam:::role/MicrovmBuildRole \ --cpu-configurations '[{"architecture":"ARM_64"}]' \ --resources '[{"minimumMemoryInMiB":2048}]' \ --environment-variables "SIGNOZ_ENDPOINT=$SIGNOZ_ENDPOINT,SIGNOZ_INGESTION_KEY=$SIGNOZ_INGESTION_KEY,OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318,OTEL_SERVICE_NAME=my-microvm-app" \ --logging '{"cloudWatch":{"logGroup":"/aws/lambda-microvms/my-monitored-app"}}' ``` **Verify these values:** `` : Your[SigNoz Cloud region](https://signoz.io/docs/ingestion/signoz-cloud/overview/#endpoint).`` : Your SigNoz[ingestion key](https://signoz.io/docs/ingestion/signoz-cloud/keys/).`` : The S3 bucket holding your code artifact.`` : The AWS region you are deploying to, for example`us-east-1` . Discover available base images with`aws lambda-microvms list-managed-microvm-images` .`` : Your AWS account ID. The build is asynchronous. Poll until the image reports `CREATED` : ``` aws lambda-microvms get-microvm-image \ --image-identifier arn:aws:lambda:::microvm-image:my-monitored-app ``` ### Step 5: Run the MicroVM ``` aws lambda-microvms run-microvm \ --image-identifier arn:aws:lambda:::microvm-image:my-monitored-app \ --execution-role-arn arn:aws:iam:::role/MicrovmExecutionRole \ --ingress-network-connectors "arn:aws:lambda::aws:network-connector:aws-network-connector:ALL_INGRESS" \ --idle-policy '{"maxIdleDurationSeconds":900,"suspendedDurationSeconds":600,"autoResumeEnabled":true}' \ --maximum-duration-in-seconds 1800 ``` MicroVMs have public internet access on the egress path by default, so the Collector reaches SigNoz without extra configuration. Attach a customer-managed egress connector when you want outbound traffic to route through your VPC instead: ``` --egress-network-connectors "" ``` **Verify these values:** `` : The ARN of a`Lambda Network Connector` in the`ACTIVE` state. Create that connector with `aws lambda-core create-network-connector` and wait for it to reach `ACTIVE` before you reference it. See [Networking for Lambda MicroVMs](https://docs.aws.amazon.com/lambda/latest/dg/microvms-networking.html). `run-microvm` returns a `microvmId` and a dedicated HTTPS `endpoint` for that MicroVM. Every request to the endpoint needs a token in the `X-aws-proxy-auth` header, and there is no unauthenticated access: ``` aws lambda-microvms create-microvm-auth-token \ --microvm-identifier \ --expiration-in-minutes 30 \ --allowed-ports '[{"port":8080}]' ``` **Verify these values:** `` : The`microvmId` returned by`run-microvm` . ## Validate Give the Collector a minute, then open SigNoz: **Traces**: your`OTEL_SERVICE_NAME` appears as a service with spans.**Metrics Explorer**:`system.memory.usage` and`system.cpu.utilization` are present and filterable by`aws.lambda.microvm.image_name` .**Logs Explorer**: your application log lines appear with the same resource attribute. To confirm the Collector itself started, check the runtime log group `/aws/lambda-microvms/` for the line `Everything is ready. Begin running and processing data.` ## Troubleshooting ### Image build fails with `tar: command not found` **Symptom:** The build stops with exit code 127 and `/bin/sh: line 1: tar: command not found` . - Likely cause: `al2023-minimal` ships without`tar` and`gzip` . - Fix: Add `RUN dnf install -y tar gzip` before any step that extracts an archive. The same applies to`pgrep` and other`procps` tools. - Verify: The build reaches `CREATED` . ### Image build fails for another reason **Symptom:** The image reports `CREATE_FAILED` , or `latestFailedImageVersion` is set. - Likely cause: A `Dockerfile` instruction failed. - Fix: Read the build output in the CloudWatch log group you passed to `--logging` , which defaults to`/aws/lambda-microvms/` . - Verify: The failing instruction appears with its exit code. ### The endpoint returns HTTP 403 **Symptom:** Requests to the MicroVM endpoint return `403 Forbidden` . - Likely cause: The token is missing, expired, or invalid, or the target port is not in the token's `allowedPorts` . - Fix: Mint a new token. Requests route to port 8080 unless you send an `X-aws-proxy-port` header, and whichever port you target must appear in`allowedPorts` . - Verify: The endpoint returns your application's response. ### The endpoint returns HTTP 502 **Symptom:** Requests to the MicroVM endpoint return `502 Bad Gateway` with an empty body. - Likely cause: Your application is not listening on the target port, it crashed handling the request, or auto-resume did not finish within the retry limit. - Fix: Check the runtime log group for a stack trace. Confirm your application listens on port 8080, or send `X-aws-proxy-port` for a different one. - Verify: The endpoint returns your application's response. ### No `system.cpu.utilization` in SigNoz **Symptom:** `system.memory.usage` appears but `system.cpu.utilization` returns no data. - Likely cause: `system.cpu.utilization` is an optional`hostmetrics` metric, disabled by default. - Fix: Enable it explicitly under the `cpu` scraper as shown in[Step 2](#step-2-configure-the-opentelemetry-collector). - Verify: The metric appears in **Metrics Explorer**. ### No logs in SigNoz **Symptom:** Traces and metrics arrive but the Logs Explorer is empty. - Likely cause: The `filelog` receiver watches`/var/log/app/*.log` , and your application writes to stdout only. - Fix: Write logs to a file under `/var/log/app/` , or drop`filelog` and export logs over OTLP from the SDK. - Verify: Records appear in **Logs Explorer**. ### The Collector logs `Configuration references unset environment variable` **Symptom:** The Collector cannot resolve `${env:SIGNOZ_ENDPOINT}` or `${env:SIGNOZ_INGESTION_KEY}` . - Likely cause: An `update-microvm-image` call omitted`--environment-variables` , which drops them from the new version. - Fix: Pass `--environment-variables` on every update. - Verify: The Collector starts and exports without warnings. ### The API returns HTTP 502 intermittently **Symptom:** `run-microvm` , `get-microvm` , or `list-microvm-images` fails with `Bad Gateway` . - Likely cause: A transient service error. AWS does not document 502 for these APIs, but it occurs in practice. - Fix: Retry with exponential backoff. Scripts that poll these APIs need retry handling. AWS documents `ThrottlingException` and`InternalServerException` as retryable, and`ResourceNotFoundException` when the image is not yet`CREATED` . - Verify: The call succeeds on a later attempt. ## Limitations **The CloudWatch Agent cannot forward to SigNoz.** AWS's recommended metrics agent for MicroVMs runs on Telegraf and the OpenTelemetry Collector. Its Collector build has no`otlp` exporter. Its only OTLP exporter,`otlphttp` , rejects non-AWS endpoints with`invalid AWS endpoint` . The agent exits on either config, which also stops the CloudWatch metrics you already had. Run a separate OpenTelemetry Collector as described above. To get CloudWatch data into SigNoz, pull the`/aws/lambda-microvms/` log group and the`LambdaMicroVms/Application` namespace through the[AWS monitoring integration](https://signoz.io/docs/aws-monitoring/overview/).**Host metrics reflect guest-visible resources.**`hostmetrics` reads what the guest kernel reports. Those totals differ from the memory and vCPU baseline you configured, so read them as relative signals rather than capacity numbers.**Metrics stop while a MicroVM is suspended.** The embedded Collector survives suspend and resume without restarting. No host metrics are produced during the suspended window, so expect gaps in dashboards. ## Optional: CloudTrail lifecycle and audit events MicroVM lifecycle actions are CloudTrail **data events**, and CloudTrail does not log them by default. Enable them with an advanced event selector on the `AWS::Lambda::MicrovmImage` resource type. The selectors below keep management events and add MicroVM data events, so they are safe to apply to a trail that was logging management events by default: ``` aws cloudtrail put-event-selectors \ --trail-name \ --advanced-event-selectors '[ { "Name": "Keep management events", "FieldSelectors": [ { "Field": "eventCategory", "Equals": ["Management"] } ] }, { "Name": "Log Lambda MicroVM data events", "FieldSelectors": [ { "Field": "eventCategory", "Equals": ["Data"] }, { "Field": "resources.type", "Equals": ["AWS::Lambda::MicrovmImage"] } ] } ]' ``` **Verify these values:** `` : The name of an existing CloudTrail trail in the same region. The second selector captures `RunMicrovm` , `TerminateMicrovm` , `SuspendMicrovm` , `ResumeMicrovm` , `CreateMicrovmAuthToken` , and `CreateMicrovmShellAuthToken` . Delivery to S3 takes a few minutes. Forward the stream into SigNoz using the existing [AWS logs ingestion patterns](https://signoz.io/docs/aws-monitoring/one-click-vs-manual/) to build suspend and resume timelines or security audit dashboards. Image and MicroVM management calls such as `CreateMicrovmImage` and `ListMicrovms` are management events. CloudTrail logs them by default, and the first selector above preserves that. See [Monitoring for Lambda MicroVMs](https://docs.aws.amazon.com/lambda/latest/dg/microvms-monitoring.html) for the full event list. ## Next steps [Set up alerts](https://signoz.io/docs/alerts-management/metrics-based-alerts/)on host metrics and error rates.[Build dashboards](https://signoz.io/docs/userguide/manage-dashboards/)grouped by`aws.lambda.microvm.image_name` .[Correlate traces and logs](https://signoz.io/docs/traces-management/guides/correlate-traces-and-logs/)to move between signals during an investigation.[Monitor AWS Lambda functions](https://signoz.io/docs/aws-monitoring/lambda/)for classic, per-invocation Lambda. ## Get Help If you need help with the steps in this topic, please reach out to us on [SigNoz Community Slack](https://signoz.io/slack/). If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at [cloud-support@signoz.io](mailto:cloud-support@signoz.io).