Monitor AWS Lambda MicroVMs with OpenTelemetry Collector

Overview #

AWS Lambda MicroVMs are Firecracker-based compute environments with VM-level isolation, snapshot start and resume, and full OS capabilities. They suit long-running, stateful sandboxes such as AI agent code execution, interactive development environments, CI jobs, and vulnerability scanners.

This guide runs an OpenTelemetry Collector inside the MicroVM image. The Collector scrapes host metrics, tails application logs, receives OTLP from your application, and exports everything to SigNoz.

Prerequisites #

• The AWS CLI with the lambda-microvms

command set. Runaws lambda-microvms help

to confirm it listscreate-microvm-image

andrun-microvm

. - An S3 bucket in the same region for the code artifact.

• A build role that Lambda assumes during the image build. It needss3:GetObject

on the artifact andlogs:CreateLogGroup

,logs:CreateLogStream

, andlogs:PutLogEvents

. - An execution role for the running MicroVM. Without it, application stdout and stderr never reach CloudWatch. It needslogs:CreateLogGroup

,logs:CreateLogStream

, andlogs:PutLogEvents

. - Both roles need a trust policy that lets Lambda assume them, with Principal: {"Service": "lambda.amazonaws.com"}

andAction: ["sts:AssumeRole", "sts:TagSession"]

. Your own caller needslambda:CreateMicrovmImage

,lambda:RunMicrovm

, andiam:PassRole

for both roles. - An instance of SigNoz (either CloudorSelf-Hosted)

How monitoring works #

Signal	Source inside the MicroVM	Path to SigNoz
Metrics (host CPU, memory, disk, network)
hostmetrics

Traces****Logsreceiver, or OTLP from the SDKfilelog

Lifecycle and audit## How sizing works

MicroVMs use a baseline and peak model. You set the baseline with the memory value when you create the image, and vCPU scales with it at 2 GB per vCPU. During activity the MicroVM scales vertically up to four times the baseline.

Baseline	Peak	Max disk
0.5 GB, 0.25 vCPU	2 GB, 1 vCPU	8 GB
1 GB, 0.5 vCPU	4 GB, 2 vCPU	8 GB
2 GB, 1 vCPU (default)	8 GB, 4 vCPU	8 GB
4 GB, 2 vCPU	16 GB, 8 vCPU	16 GB
8 GB, 4 vCPU	32 GB, 16 vCPU	32 GB

A MicroVM runs for at most 8 hours. Set the limit with --maximum-duration-in-seconds

, which accepts 1 to 28,800 seconds.

Send telemetry to SigNoz #

Lambda builds the MicroVM image from a ZIP you upload to S3, then snapshots the running result. The Collector is baked into that snapshot, so it is already running the moment a MicroVM starts. Your application exports to it on localhost

instead of reaching SigNoz directly.

Step 1: Package the application and Dockerfile

The code artifact is a ZIP that contains a Dockerfile

at the archive root plus your application files. Lambda pulls the ZIP from S3, runs your Dockerfile

on top of the managed base image, starts your application, and snapshots the result.

Two different base images are involved. The MicroVM base image is the operating system environment, passed as --base-image-arn

. The container base image is what your Dockerfile

uses in its FROM

instruction.

FROM public.ecr.aws/lambda/microvms:al2023-minimal
 
RUN dnf install -y tar gzip python3 python3-pip && dnf clean all
 
ARG OTELCOL_VERSION=0.159.0
ADD https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${OTELCOL_VERSION}/otelcol-contrib_${OTELCOL_VERSION}_linux_arm64.tar.gz /tmp/otelcol.tar.gz
RUN mkdir -p /usr/local/bin \
 && tar -xzf /tmp/otelcol.tar.gz -C /usr/local/bin otelcol-contrib \
 && rm /tmp/otelcol.tar.gz
 
COPY app.py                     /app/app.py
COPY otel-collector-config.yaml /etc/otelcol/config.yaml
COPY entrypoint.sh              /entrypoint.sh
RUN chmod +x /entrypoint.sh && mkdir -p /var/log/app
 
CMD ["/entrypoint.sh"]

The image build has outbound internet access, so pulling the Collector with ADD

works. Start the Collector in the background and your application in the foreground, which keeps the MicroVM alive:

entrypoint.sh
bash
#!/usr/bin/env bash
set -euo pipefail
 
/usr/local/bin/otelcol-contrib --config /etc/otelcol/config.yaml 2>&1 &
 
exec python3 -u /app/app.py

Step 2: Configure the OpenTelemetry Collector

otel-collector-config.yaml
receivers:
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
      memory: {}
      load: {}
      filesystem: {}
      network: {}
 
  #
  filelog:
    include: [/var/log/app/*.log]
    exclude: [/var/log/app/otelcol*.log]
    start_at: beginning
 
  otlp:
    protocols:
      http:
        endpoint: localhost:4318
 
processors:
  batch: {}
  resourcedetection:
    detectors: [env, system]
  resource:
    attributes:
      - key: aws.lambda.microvm.image_name
        value: ${env:AWS_LAMBDA_MICROVM_IMAGE_NAME}
        action: upsert
      - key: aws.lambda.microvm.image_version
        value: ${env:AWS_LAMBDA_MICROVM_IMAGE_VERSION}
        action: upsert
 
exporters:
  otlphttp:
    endpoint: ${env:SIGNOZ_ENDPOINT}
    headers:
      signoz-ingestion-key: ${env:SIGNOZ_INGESTION_KEY}
 
service:
  pipelines:
    metrics:
      receivers: [hostmetrics, otlp]
      processors: [resourcedetection, resource, batch]
      exporters: [otlphttp]
    traces:
      receivers: [otlp]
      processors: [resourcedetection, resource, batch]
      exporters: [otlphttp]
    logs:
      receivers: [filelog, otlp]
      processors: [resourcedetection, resource, batch]
      exporters: [otlphttp]

Lambda injects AWS_LAMBDA_MICROVM_IMAGE_NAME

, AWS_LAMBDA_MICROVM_IMAGE_ARN

, AWS_LAMBDA_MICROVM_IMAGE_VERSION

, and AWS_REGION

into every MicroVM. Using the image name as a resource attribute lets you filter and group all three signals by image in SigNoz.

Step 3: Instrument your application

Because the MicroVM is long-lived, use standard service-style instrumentation rather than a Lambda layer. Point the SDK at the embedded Collector with these variables:

OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
OTEL_SERVICE_NAME="my-microvm-app"
OTEL_RESOURCE_ATTRIBUTES="deployment.environment=production"

There is no environment variable flag on run-microvm

, so pass these to --environment-variables

when you build the image in Step 4. Setting them anywhere else has no effect.

See the SigNoz instrumentation guides for your language.

Write your application logs to /var/log/app/

for the filelog

receiver from Step 2 to pick them up, or remove that receiver and export logs over OTLP from the SDK.

Your directory now holds everything the build needs. Zip it with the Dockerfile

at the archive root and upload it:

zip -r app.zip Dockerfile entrypoint.sh otel-collector-config.yaml app.py
aws s3 cp app.zip s3://<your-bucket>/app.zip

Verify these values:

<your-bucket>

: The S3 bucket you created for the code artifact.

Step 4: Build the MicroVM image

Export your SigNoz destination so the build command can read it:

export SIGNOZ_ENDPOINT="https://ingest.<region>.signoz.cloud:443"
export SIGNOZ_INGESTION_KEY="<your-ingestion-key>"

Then create the image:

aws lambda-microvms create-microvm-image \
  --name my-monitored-app \
  --code-artifact uri=s3://<your-bucket>/app.zip \
  --base-image-arn arn:aws:lambda:<aws-region>:aws:microvm-image:al2023-1 \
  --build-role-arn arn:aws:iam::<account-id>:role/MicrovmBuildRole \
  --cpu-configurations '[{"architecture":"ARM_64"}]' \
  --resources '[{"minimumMemoryInMiB":2048}]' \
  --environment-variables "SIGNOZ_ENDPOINT=$SIGNOZ_ENDPOINT,SIGNOZ_INGESTION_KEY=$SIGNOZ_INGESTION_KEY,OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318,OTEL_SERVICE_NAME=my-microvm-app" \
  --logging '{"cloudWatch":{"logGroup":"/aws/lambda-microvms/my-monitored-app"}}'

Verify these values:

<region>

: YourSigNoz Cloud region.<your-ingestion-key>

: Your SigNozingestion key.<your-bucket>

: The S3 bucket holding your code artifact.<aws-region>

: The AWS region you are deploying to, for exampleus-east-1

. Discover available base images withaws lambda-microvms list-managed-microvm-images

.<account-id>

: Your AWS account ID.

The build is asynchronous. Poll until the image reports CREATED

:

aws lambda-microvms get-microvm-image \
  --image-identifier arn:aws:lambda:<aws-region>:<account-id>:microvm-image:my-monitored-app

Step 5: Run the MicroVM

aws lambda-microvms run-microvm \
  --image-identifier arn:aws:lambda:<aws-region>:<account-id>:microvm-image:my-monitored-app \
  --execution-role-arn arn:aws:iam::<account-id>:role/MicrovmExecutionRole \
  --ingress-network-connectors "arn:aws:lambda:<aws-region>:aws:network-connector:aws-network-connector:ALL_INGRESS" \
  --idle-policy '{"maxIdleDurationSeconds":900,"suspendedDurationSeconds":600,"autoResumeEnabled":true}' \
  --maximum-duration-in-seconds 1800

MicroVMs have public internet access on the egress path by default, so the Collector reaches SigNoz without extra configuration. Attach a customer-managed egress connector when you want outbound traffic to route through your VPC instead:

--egress-network-connectors "<your-vpc-connector-arn>"

Verify these values:

<your-vpc-connector-arn>

: The ARN of aLambda Network Connector

in theACTIVE

state.

Create that connector with aws lambda-core create-network-connector

and wait for it to reach ACTIVE

before you reference it. See Networking for Lambda MicroVMs.

run-microvm

returns a microvmId

and a dedicated HTTPS endpoint

for that MicroVM. Every request to the endpoint needs a token in the X-aws-proxy-auth

header, and there is no unauthenticated access:

aws lambda-microvms create-microvm-auth-token \
  --microvm-identifier <microvm-id> \
  --expiration-in-minutes 30 \
  --allowed-ports '[{"port":8080}]'

Verify these values:

<microvm-id>

: ThemicrovmId

returned byrun-microvm

.

Validate #

Give the Collector a minute, then open SigNoz:

Traces: yourOTEL_SERVICE_NAME

appears as a service with spans.Metrics Explorer:system.memory.usage

andsystem.cpu.utilization

are present and filterable byaws.lambda.microvm.image_name

.Logs Explorer: your application log lines appear with the same resource attribute.

To confirm the Collector itself started, check the runtime log group /aws/lambda-microvms/<image-name>

for the line Everything is ready. Begin running and processing data.

Troubleshooting #

Image build fails with tar: command not found

Symptom: The build stops with exit code 127 and /bin/sh: line 1: tar: command not found

.

• Likely cause: al2023-minimal

ships withouttar

andgzip

. - Fix: Add RUN dnf install -y tar gzip

before any step that extracts an archive. The same applies topgrep

and otherprocps

tools. - Verify: The build reaches CREATED

.

Image build fails for another reason

Symptom: The image reports CREATE_FAILED

, or latestFailedImageVersion

is set.

• Likely cause: A Dockerfile

instruction failed. - Fix: Read the build output in the CloudWatch log group you passed to --logging

, which defaults to/aws/lambda-microvms/<image-name>

. - Verify: The failing instruction appears with its exit code.

The endpoint returns HTTP 403

Symptom: Requests to the MicroVM endpoint return 403 Forbidden

.

• Likely cause: The token is missing, expired, or invalid, or the target port is not in the token's allowedPorts

. - Fix: Mint a new token. Requests route to port 8080 unless you send an X-aws-proxy-port

header, and whichever port you target must appear inallowedPorts

. - Verify: The endpoint returns your application's response.

The endpoint returns HTTP 502

Symptom: Requests to the MicroVM endpoint return 502 Bad Gateway

with an empty body.

• Likely cause: Your application is not listening on the target port, it crashed handling the request, or auto-resume did not finish within the retry limit.
• Fix: Check the runtime log group for a stack trace. Confirm your application listens on port 8080, or send X-aws-proxy-port

for a different one. - Verify: The endpoint returns your application's response.

No system.cpu.utilization

in SigNoz

Symptom: system.memory.usage

appears but system.cpu.utilization

returns no data.

• Likely cause: system.cpu.utilization

is an optionalhostmetrics

metric, disabled by default. - Fix: Enable it explicitly under the cpu

scraper as shown inStep 2. - Verify: The metric appears in Metrics Explorer.

No logs in SigNoz

Symptom: Traces and metrics arrive but the Logs Explorer is empty.

• Likely cause: The filelog

receiver watches/var/log/app/*.log

, and your application writes to stdout only. - Fix: Write logs to a file under /var/log/app/

, or dropfilelog

and export logs over OTLP from the SDK. - Verify: Records appear in Logs Explorer.

The Collector logs Configuration references unset environment variable

Symptom: The Collector cannot resolve ${env:SIGNOZ_ENDPOINT}

or ${env:SIGNOZ_INGESTION_KEY}

.

• Likely cause: An update-microvm-image

call omitted--environment-variables

, which drops them from the new version. - Fix: Pass --environment-variables

on every update. - Verify: The Collector starts and exports without warnings.

The API returns HTTP 502 intermittently

Symptom: run-microvm

, get-microvm

, or list-microvm-images

fails with Bad Gateway

.

• Likely cause: A transient service error. AWS does not document 502 for these APIs, but it occurs in practice.
• Fix: Retry with exponential backoff. Scripts that poll these APIs need retry handling. AWS documents ThrottlingException

andInternalServerException

as retryable, andResourceNotFoundException

when the image is not yetCREATED

. - Verify: The call succeeds on a later attempt.

Limitations #

The CloudWatch Agent cannot forward to SigNoz. AWS's recommended metrics agent for MicroVMs runs on Telegraf and the OpenTelemetry Collector. Its Collector build has nootlp

exporter. Its only OTLP exporter,otlphttp

, rejects non-AWS endpoints withinvalid AWS endpoint

. The agent exits on either config, which also stops the CloudWatch metrics you already had. Run a separate OpenTelemetry Collector as described above. To get CloudWatch data into SigNoz, pull the/aws/lambda-microvms/<image-name>

log group and theLambdaMicroVms/Application

namespace through theAWS monitoring integration.Host metrics reflect guest-visible resources.hostmetrics

reads what the guest kernel reports. Those totals differ from the memory and vCPU baseline you configured, so read them as relative signals rather than capacity numbers.Metrics stop while a MicroVM is suspended. The embedded Collector survives suspend and resume without restarting. No host metrics are produced during the suspended window, so expect gaps in dashboards.

Optional: CloudTrail lifecycle and audit events #

MicroVM lifecycle actions are CloudTrail data events, and CloudTrail does not log them by default. Enable them with an advanced event selector on the AWS::Lambda::MicrovmImage

resource type.

The selectors below keep management events and add MicroVM data events, so they are safe to apply to a trail that was logging management events by default:

aws cloudtrail put-event-selectors \
  --trail-name <your-trail> \
  --advanced-event-selectors '[
    {
      "Name": "Keep management events",
      "FieldSelectors": [
        { "Field": "eventCategory", "Equals": ["Management"] }
      ]
    },
    {
      "Name": "Log Lambda MicroVM data events",
      "FieldSelectors": [
        { "Field": "eventCategory", "Equals": ["Data"] },
        { "Field": "resources.type", "Equals": ["AWS::Lambda::MicrovmImage"] }
      ]
    }
  ]'

Verify these values:

<your-trail>

: The name of an existing CloudTrail trail in the same region.

The second selector captures RunMicrovm

, TerminateMicrovm

, SuspendMicrovm

, ResumeMicrovm

, CreateMicrovmAuthToken

, and CreateMicrovmShellAuthToken

. Delivery to S3 takes a few minutes. Forward the stream into SigNoz using the existing AWS logs ingestion patterns to build suspend and resume timelines or security audit dashboards.

Image and MicroVM management calls such as CreateMicrovmImage

and ListMicrovms

are management events. CloudTrail logs them by default, and the first selector above preserves that. See Monitoring for Lambda MicroVMs for the full event list.

Next steps #

Set up alertson host metrics and error rates.Build dashboardsgrouped byaws.lambda.microvm.image_name

.Correlate traces and logsto move between signals during an investigation.Monitor AWS Lambda functionsfor classic, per-invocation Lambda.

Get Help #

If you need help with the steps in this topic, please reach out to us on SigNoz Community Slack. If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at cloud-support@signoz.io.

source & further reading

signoz.io — original article Antigravity CLI Dashboard Google Antigravity CLI Monitoring with OpenTelemetry Cline Monitoring with OpenTelemetry: Tokens and Cost