Monitor AWS Lambda MicroVMs with OpenTelemetry Collector

SigNoz published a guide for monitoring AWS Lambda MicroVMs with the OpenTelemetry Collector, which scrapes host metrics, tails logs, and exports telemetry to SigNoz. The Collector is baked into the MicroVM image snapshot, so it runs at startup. MicroVMs scale up to four times baseline, run up to 8 hours, and support baseline sizes from 0.5 GB to 8 GB.

Overview AWS Lambda MicroVMs https://aws.amazon.com/lambda/lambda-microvms/ are Firecracker-based compute environments with VM-level isolation, snapshot start and resume, and full OS capabilities. They suit long-running, stateful sandboxes such as AI agent code execution, interactive development environments, CI jobs, and vulnerability scanners. This guide runs an OpenTelemetry Collector inside the MicroVM image. The Collector scrapes host metrics, tails application logs, receives OTLP from your application, and exports everything to SigNoz. Prerequisites - The AWS CLI with the lambda-microvms command set. Run aws lambda-microvms help to confirm it lists create-microvm-image and run-microvm . - An S3 bucket in the same region for the code artifact. - A build role that Lambda assumes during the image build. It needs s3:GetObject on the artifact and logs:CreateLogGroup , logs:CreateLogStream , and logs:PutLogEvents . - An execution role for the running MicroVM. Without it, application stdout and stderr never reach CloudWatch. It needs logs:CreateLogGroup , logs:CreateLogStream , and logs:PutLogEvents . - Both roles need a trust policy that lets Lambda assume them, with Principal: {"Service": "lambda.amazonaws.com"} and Action: "sts:AssumeRole", "sts:TagSession" . Your own caller needs lambda:CreateMicrovmImage , lambda:RunMicrovm , and iam:PassRole for both roles. - An instance of SigNoz either Cloud https://signoz.io/teams/ or Self-Hosted https://signoz.io/docs/install/self-host/ How monitoring works | Signal | Source inside the MicroVM | Path to SigNoz | |---|---|---| Metrics host CPU, memory, disk, network | hostmetrics | Traces Logs receiver, or OTLP from the SDK https://signoz.io/docs/userguide/collect logs from file/ filelog Lifecycle and audit How sizing works MicroVMs use a baseline and peak model. You set the baseline with the memory value when you create the image, and vCPU scales with it at 2 GB per vCPU. During activity the MicroVM scales vertically up to four times the baseline. | Baseline | Peak | Max disk | |---|---|---| | 0.5 GB, 0.25 vCPU | 2 GB, 1 vCPU | 8 GB | | 1 GB, 0.5 vCPU | 4 GB, 2 vCPU | 8 GB | | 2 GB, 1 vCPU default | 8 GB, 4 vCPU | 8 GB | | 4 GB, 2 vCPU | 16 GB, 8 vCPU | 16 GB | | 8 GB, 4 vCPU | 32 GB, 16 vCPU | 32 GB | A MicroVM runs for at most 8 hours. Set the limit with --maximum-duration-in-seconds , which accepts 1 to 28,800 seconds. Send telemetry to SigNoz Lambda builds the MicroVM image from a ZIP you upload to S3, then snapshots the running result. The Collector is baked into that snapshot, so it is already running the moment a MicroVM starts. Your application exports to it on localhost instead of reaching SigNoz directly. Step 1: Package the application and Dockerfile The code artifact is a ZIP that contains a Dockerfile at the archive root plus your application files. Lambda pulls the ZIP from S3, runs your Dockerfile on top of the managed base image, starts your application, and snapshots the result. Two different base images are involved. The MicroVM base image is the operating system environment, passed as --base-image-arn . The container base image is what your Dockerfile uses in its FROM instruction. FROM public.ecr.aws/lambda/microvms:al2023-minimal al2023-minimal ships without tar, gzip, or procps. Install what you need before extracting anything. RUN dnf install -y tar gzip python3 python3-pip && dnf clean all Release assets are version-named, and MicroVMs are ARM64 only. Pin a version and use the arm64 tarball. ARG OTELCOL VERSION=0.159.0 ADD https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${OTELCOL VERSION}/otelcol-contrib ${OTELCOL VERSION} linux arm64.tar.gz /tmp/otelcol.tar.gz RUN mkdir -p /usr/local/bin \ && tar -xzf /tmp/otelcol.tar.gz -C /usr/local/bin otelcol-contrib \ && rm /tmp/otelcol.tar.gz COPY app.py /app/app.py COPY otel-collector-config.yaml /etc/otelcol/config.yaml COPY entrypoint.sh /entrypoint.sh RUN chmod +x /entrypoint.sh && mkdir -p /var/log/app CMD "/entrypoint.sh" The image build has outbound internet access, so pulling the Collector with ADD works. Start the Collector in the background and your application in the foreground, which keeps the MicroVM alive: entrypoint.sh bash /usr/bin/env bash set -euo pipefail Lambda forwards only stdout and stderr to CloudWatch, so do not redirect the Collector to a file. Writing it into /var/log/app would also make the filelog receiver tail the Collector's own output. /usr/local/bin/otelcol-contrib --config /etc/otelcol/config.yaml 2 &1 & exec python3 -u /app/app.py Step 2: Configure the OpenTelemetry Collector otel-collector-config.yaml receivers: On Collector v0.151.0 and newer, use "host metrics" to avoid a deprecation warning. hostmetrics: collection interval: 30s scrapers: cpu: metrics: system.cpu.utilization is an optional metric and off by default. Enable it explicitly or it never reaches SigNoz. system.cpu.utilization: enabled: true memory: {} load: {} filesystem: {} network: {} On Collector v0.149.0 and newer, use "file log" to avoid a deprecation warning. This only collects anything if your application writes to this path. An application that logs to stdout alone produces no records here. Keep the Collector's own log out of this glob. A Collector that tails its own output re-exports every line it writes. filelog: include: /var/log/app/ .log exclude: /var/log/app/otelcol .log start at: beginning otlp: protocols: http: endpoint: localhost:4318 processors: batch: {} On Collector v0.153.0 and newer, use "resource detection" to avoid a deprecation warning. resourcedetection: detectors: env, system resource: attributes: Lambda injects this into every MicroVM. It is the same value the CloudWatch Agent uses for its ImageName dimension. - key: aws.lambda.microvm.image name value: ${env:AWS LAMBDA MICROVM IMAGE NAME} action: upsert - key: aws.lambda.microvm.image version value: ${env:AWS LAMBDA MICROVM IMAGE VERSION} action: upsert exporters: On Collector v0.144.0 and newer, use "otlp http" to avoid a deprecation warning. otlphttp: endpoint: ${env:SIGNOZ ENDPOINT} headers: signoz-ingestion-key: ${env:SIGNOZ INGESTION KEY} service: pipelines: metrics: receivers: hostmetrics, otlp processors: resourcedetection, resource, batch exporters: otlphttp traces: receivers: otlp processors: resourcedetection, resource, batch exporters: otlphttp logs: receivers: filelog, otlp processors: resourcedetection, resource, batch exporters: otlphttp Lambda injects AWS LAMBDA MICROVM IMAGE NAME , AWS LAMBDA MICROVM IMAGE ARN , AWS LAMBDA MICROVM IMAGE VERSION , and AWS REGION into every MicroVM. Using the image name as a resource attribute lets you filter and group all three signals by image in SigNoz. Step 3: Instrument your application Because the MicroVM is long-lived, use standard service-style instrumentation rather than a Lambda layer. Point the SDK at the embedded Collector with these variables: OTEL EXPORTER OTLP ENDPOINT="http://localhost:4318" OTEL SERVICE NAME="my-microvm-app" OTEL RESOURCE ATTRIBUTES="deployment.environment=production" There is no environment variable flag on run-microvm , so pass these to --environment-variables when you build the image in Step 4 step-4-build-the-microvm-image . Setting them anywhere else has no effect. See the SigNoz instrumentation guides https://signoz.io/docs/instrumentation/ for your language. Write your application logs to /var/log/app/ for the filelog receiver from Step 2 step-2-configure-the-opentelemetry-collector to pick them up, or remove that receiver and export logs over OTLP from the SDK. Your directory now holds everything the build needs. Zip it with the Dockerfile at the archive root and upload it: zip -r app.zip Dockerfile entrypoint.sh otel-collector-config.yaml app.py aws s3 cp app.zip s3://<your-bucket /app.zip Verify these values: <your-bucket : The S3 bucket you created for the code artifact. Step 4: Build the MicroVM image Export your SigNoz destination so the build command can read it: export SIGNOZ ENDPOINT="https://ingest.<region .signoz.cloud:443" export SIGNOZ INGESTION KEY="<your-ingestion-key " Then create the image: aws lambda-microvms create-microvm-image \ --name my-monitored-app \ --code-artifact uri=s3://<your-bucket /app.zip \ --base-image-arn arn:aws:lambda:<aws-region :aws:microvm-image:al2023-1 \ --build-role-arn arn:aws:iam::<account-id :role/MicrovmBuildRole \ --cpu-configurations ' {"architecture":"ARM 64"} ' \ --resources ' {"minimumMemoryInMiB":2048} ' \ --environment-variables "SIGNOZ ENDPOINT=$SIGNOZ ENDPOINT,SIGNOZ INGESTION KEY=$SIGNOZ INGESTION KEY,OTEL EXPORTER OTLP ENDPOINT=http://localhost:4318,OTEL SERVICE NAME=my-microvm-app" \ --logging '{"cloudWatch":{"logGroup":"/aws/lambda-microvms/my-monitored-app"}}' Verify these values: <region : Your SigNoz Cloud region https://signoz.io/docs/ingestion/signoz-cloud/overview/ endpoint . <your-ingestion-key : Your SigNoz ingestion key https://signoz.io/docs/ingestion/signoz-cloud/keys/ . <your-bucket : The S3 bucket holding your code artifact. <aws-region : The AWS region you are deploying to, for example us-east-1 . Discover available base images with aws lambda-microvms list-managed-microvm-images . <account-id : Your AWS account ID. The build is asynchronous. Poll until the image reports CREATED : aws lambda-microvms get-microvm-image \ --image-identifier arn:aws:lambda:<aws-region :<account-id :microvm-image:my-monitored-app Step 5: Run the MicroVM aws lambda-microvms run-microvm \ --image-identifier arn:aws:lambda:<aws-region :<account-id :microvm-image:my-monitored-app \ --execution-role-arn arn:aws:iam::<account-id :role/MicrovmExecutionRole \ --ingress-network-connectors "arn:aws:lambda:<aws-region :aws:network-connector:aws-network-connector:ALL INGRESS" \ --idle-policy '{"maxIdleDurationSeconds":900,"suspendedDurationSeconds":600,"autoResumeEnabled":true}' \ --maximum-duration-in-seconds 1800 MicroVMs have public internet access on the egress path by default, so the Collector reaches SigNoz without extra configuration. Attach a customer-managed egress connector when you want outbound traffic to route through your VPC instead: --egress-network-connectors "<your-vpc-connector-arn " Verify these values: <your-vpc-connector-arn : The ARN of a Lambda Network Connector in the ACTIVE state. Create that connector with aws lambda-core create-network-connector and wait for it to reach ACTIVE before you reference it. See Networking for Lambda MicroVMs https://docs.aws.amazon.com/lambda/latest/dg/microvms-networking.html . run-microvm returns a microvmId and a dedicated HTTPS endpoint for that MicroVM. Every request to the endpoint needs a token in the X-aws-proxy-auth header, and there is no unauthenticated access: aws lambda-microvms create-microvm-auth-token \ --microvm-identifier <microvm-id \ --expiration-in-minutes 30 \ --allowed-ports ' {"port":8080} ' Verify these values: <microvm-id : The microvmId returned by run-microvm . Validate Give the Collector a minute, then open SigNoz: Traces : your OTEL SERVICE NAME appears as a service with spans. Metrics Explorer : system.memory.usage and system.cpu.utilization are present and filterable by aws.lambda.microvm.image name . Logs Explorer : your application log lines appear with the same resource attribute. To confirm the Collector itself started, check the runtime log group /aws/lambda-microvms/<image-name for the line Everything is ready. Begin running and processing data. Troubleshooting Image build fails with tar: command not found Symptom: The build stops with exit code 127 and /bin/sh: line 1: tar: command not found . - Likely cause: al2023-minimal ships without tar and gzip . - Fix: Add RUN dnf install -y tar gzip before any step that extracts an archive. The same applies to pgrep and other procps tools. - Verify: The build reaches CREATED . Image build fails for another reason Symptom: The image reports CREATE FAILED , or latestFailedImageVersion is set. - Likely cause: A Dockerfile instruction failed. - Fix: Read the build output in the CloudWatch log group you passed to --logging , which defaults to /aws/lambda-microvms/<image-name . - Verify: The failing instruction appears with its exit code. The endpoint returns HTTP 403 Symptom: Requests to the MicroVM endpoint return 403 Forbidden . - Likely cause: The token is missing, expired, or invalid, or the target port is not in the token's allowedPorts . - Fix: Mint a new token. Requests route to port 8080 unless you send an X-aws-proxy-port header, and whichever port you target must appear in allowedPorts . - Verify: The endpoint returns your application's response. The endpoint returns HTTP 502 Symptom: Requests to the MicroVM endpoint return 502 Bad Gateway with an empty body. - Likely cause: Your application is not listening on the target port, it crashed handling the request, or auto-resume did not finish within the retry limit. - Fix: Check the runtime log group for a stack trace. Confirm your application listens on port 8080, or send X-aws-proxy-port for a different one. - Verify: The endpoint returns your application's response. No system.cpu.utilization in SigNoz Symptom: system.memory.usage appears but system.cpu.utilization returns no data. - Likely cause: system.cpu.utilization is an optional hostmetrics metric, disabled by default. - Fix: Enable it explicitly under the cpu scraper as shown in Step 2 step-2-configure-the-opentelemetry-collector . - Verify: The metric appears in Metrics Explorer . No logs in SigNoz Symptom: Traces and metrics arrive but the Logs Explorer is empty. - Likely cause: The filelog receiver watches /var/log/app/ .log , and your application writes to stdout only. - Fix: Write logs to a file under /var/log/app/ , or drop filelog and export logs over OTLP from the SDK. - Verify: Records appear in Logs Explorer . The Collector logs Configuration references unset environment variable Symptom: The Collector cannot resolve ${env:SIGNOZ ENDPOINT} or ${env:SIGNOZ INGESTION KEY} . - Likely cause: An update-microvm-image call omitted --environment-variables , which drops them from the new version. - Fix: Pass --environment-variables on every update. - Verify: The Collector starts and exports without warnings. The API returns HTTP 502 intermittently Symptom: run-microvm , get-microvm , or list-microvm-images fails with Bad Gateway . - Likely cause: A transient service error. AWS does not document 502 for these APIs, but it occurs in practice. - Fix: Retry with exponential backoff. Scripts that poll these APIs need retry handling. AWS documents ThrottlingException and InternalServerException as retryable, and ResourceNotFoundException when the image is not yet CREATED . - Verify: The call succeeds on a later attempt. Limitations The CloudWatch Agent cannot forward to SigNoz. AWS's recommended metrics agent for MicroVMs runs on Telegraf and the OpenTelemetry Collector. Its Collector build has no otlp exporter. Its only OTLP exporter, otlphttp , rejects non-AWS endpoints with invalid AWS endpoint . The agent exits on either config, which also stops the CloudWatch metrics you already had. Run a separate OpenTelemetry Collector as described above. To get CloudWatch data into SigNoz, pull the /aws/lambda-microvms/<image-name log group and the LambdaMicroVms/Application namespace through the AWS monitoring integration https://signoz.io/docs/aws-monitoring/overview/ . Host metrics reflect guest-visible resources. hostmetrics reads what the guest kernel reports. Those totals differ from the memory and vCPU baseline you configured, so read them as relative signals rather than capacity numbers. Metrics stop while a MicroVM is suspended. The embedded Collector survives suspend and resume without restarting. No host metrics are produced during the suspended window, so expect gaps in dashboards. Optional: CloudTrail lifecycle and audit events MicroVM lifecycle actions are CloudTrail data events , and CloudTrail does not log them by default. Enable them with an advanced event selector on the AWS::Lambda::MicrovmImage resource type. The selectors below keep management events and add MicroVM data events, so they are safe to apply to a trail that was logging management events by default: aws cloudtrail put-event-selectors \ --trail-name <your-trail \ --advanced-event-selectors ' { "Name": "Keep management events", "FieldSelectors": { "Field": "eventCategory", "Equals": "Management" } }, { "Name": "Log Lambda MicroVM data events", "FieldSelectors": { "Field": "eventCategory", "Equals": "Data" }, { "Field": "resources.type", "Equals": "AWS::Lambda::MicrovmImage" } } ' Verify these values: <your-trail : The name of an existing CloudTrail trail in the same region. The second selector captures RunMicrovm , TerminateMicrovm , SuspendMicrovm , ResumeMicrovm , CreateMicrovmAuthToken , and CreateMicrovmShellAuthToken . Delivery to S3 takes a few minutes. Forward the stream into SigNoz using the existing AWS logs ingestion patterns https://signoz.io/docs/aws-monitoring/one-click-vs-manual/ to build suspend and resume timelines or security audit dashboards. Image and MicroVM management calls such as CreateMicrovmImage and ListMicrovms are management events. CloudTrail logs them by default, and the first selector above preserves that. See Monitoring for Lambda MicroVMs https://docs.aws.amazon.com/lambda/latest/dg/microvms-monitoring.html for the full event list. Next steps Set up alerts https://signoz.io/docs/alerts-management/metrics-based-alerts/ on host metrics and error rates. Build dashboards https://signoz.io/docs/userguide/manage-dashboards/ grouped by aws.lambda.microvm.image name . Correlate traces and logs https://signoz.io/docs/traces-management/guides/correlate-traces-and-logs/ to move between signals during an investigation. Monitor AWS Lambda functions https://signoz.io/docs/aws-monitoring/lambda/ for classic, per-invocation Lambda. Get Help If you need help with the steps in this topic, please reach out to us on SigNoz Community Slack https://signoz.io/slack/ . If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at cloud-support@signoz.io mailto:cloud-support@signoz.io .