{"slug": "monitor-aws-lambda-microvms-with-opentelemetry-collector", "title": "Monitor AWS Lambda MicroVMs with OpenTelemetry Collector", "summary": "SigNoz published a guide for monitoring AWS Lambda MicroVMs with the OpenTelemetry Collector, which scrapes host metrics, tails logs, and exports telemetry to SigNoz. The Collector is baked into the MicroVM image snapshot, so it runs at startup. MicroVMs scale up to four times baseline, run up to 8 hours, and support baseline sizes from 0.5 GB to 8 GB.", "body_md": "## Overview\n\n[AWS Lambda MicroVMs](https://aws.amazon.com/lambda/lambda-microvms/) are\nFirecracker-based compute environments with VM-level isolation, snapshot start and resume, and full\nOS capabilities. They suit long-running, stateful sandboxes such as AI agent code execution,\ninteractive development environments, CI jobs, and vulnerability scanners.\n\nThis guide runs an OpenTelemetry Collector inside the MicroVM image. The Collector scrapes host metrics, tails application logs, receives OTLP from your application, and exports everything to SigNoz.\n\n## Prerequisites\n\n- The AWS CLI with the\n`lambda-microvms`\n\ncommand set. Run`aws lambda-microvms help`\n\nto confirm it lists`create-microvm-image`\n\nand`run-microvm`\n\n. - An S3 bucket in the same region for the code artifact.\n- A\n**build role** that Lambda assumes during the image build. It needs`s3:GetObject`\n\non the artifact and`logs:CreateLogGroup`\n\n,`logs:CreateLogStream`\n\n, and`logs:PutLogEvents`\n\n. - An\n**execution role** for the running MicroVM. Without it, application stdout and stderr never reach CloudWatch. It needs`logs:CreateLogGroup`\n\n,`logs:CreateLogStream`\n\n, and`logs:PutLogEvents`\n\n. - Both roles need a trust policy that lets Lambda assume them, with\n`Principal: {\"Service\": \"lambda.amazonaws.com\"}`\n\nand`Action: [\"sts:AssumeRole\", \"sts:TagSession\"]`\n\n. Your own caller needs`lambda:CreateMicrovmImage`\n\n,`lambda:RunMicrovm`\n\n, and`iam:PassRole`\n\nfor both roles. - An instance of SigNoz (either\n[Cloud](https://signoz.io/teams/)or[Self-Hosted](https://signoz.io/docs/install/self-host/))\n\n## How monitoring works\n\n| Signal | Source inside the MicroVM | Path to SigNoz |\n|---|---|---|\nMetrics (host CPU, memory, disk, network) |\n`hostmetrics` |\n\n**Traces****Logs**[receiver, or OTLP from the SDK](https://signoz.io/docs/userguide/collect_logs_from_file/)`filelog`\n\n**Lifecycle and audit**## How sizing works\n\nMicroVMs use a baseline and peak model. You set the baseline with the memory value when you create the image, and vCPU scales with it at 2 GB per vCPU. During activity the MicroVM scales vertically up to four times the baseline.\n\n| Baseline | Peak | Max disk |\n|---|---|---|\n| 0.5 GB, 0.25 vCPU | 2 GB, 1 vCPU | 8 GB |\n| 1 GB, 0.5 vCPU | 4 GB, 2 vCPU | 8 GB |\n| 2 GB, 1 vCPU (default) | 8 GB, 4 vCPU | 8 GB |\n| 4 GB, 2 vCPU | 16 GB, 8 vCPU | 16 GB |\n| 8 GB, 4 vCPU | 32 GB, 16 vCPU | 32 GB |\n\nA MicroVM runs for at most 8 hours. Set the limit with `--maximum-duration-in-seconds`\n\n, which\naccepts 1 to 28,800 seconds.\n\n## Send telemetry to SigNoz\n\nLambda builds the MicroVM image from a ZIP you upload to S3, then snapshots the running result. The\nCollector is baked into that snapshot, so it is already running the moment a MicroVM starts. Your\napplication exports to it on `localhost`\n\ninstead of reaching SigNoz directly.\n\n### Step 1: Package the application and Dockerfile\n\nThe code artifact is a ZIP that contains a `Dockerfile`\n\nat the archive root plus your application\nfiles. Lambda pulls the ZIP from S3, runs your `Dockerfile`\n\non top of the managed base image, starts\nyour application, and snapshots the result.\n\nTwo different base images are involved. The **MicroVM base image** is the operating system\nenvironment, passed as `--base-image-arn`\n\n. The **container base image** is what your `Dockerfile`\n\nuses in its `FROM`\n\ninstruction.\n\n```\nFROM public.ecr.aws/lambda/microvms:al2023-minimal\n \n# al2023-minimal ships without tar, gzip, or procps. Install what you need\n# before extracting anything.\nRUN dnf install -y tar gzip python3 python3-pip && dnf clean all\n \n# Release assets are version-named, and MicroVMs are ARM64 only. Pin a version\n# and use the arm64 tarball.\nARG OTELCOL_VERSION=0.159.0\nADD https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${OTELCOL_VERSION}/otelcol-contrib_${OTELCOL_VERSION}_linux_arm64.tar.gz /tmp/otelcol.tar.gz\nRUN mkdir -p /usr/local/bin \\\n && tar -xzf /tmp/otelcol.tar.gz -C /usr/local/bin otelcol-contrib \\\n && rm /tmp/otelcol.tar.gz\n \nCOPY app.py /app/app.py\nCOPY otel-collector-config.yaml /etc/otelcol/config.yaml\nCOPY entrypoint.sh /entrypoint.sh\nRUN chmod +x /entrypoint.sh && mkdir -p /var/log/app\n \nCMD [\"/entrypoint.sh\"]\n```\n\nThe image build has outbound internet access, so pulling the Collector with `ADD`\n\nworks. Start the\nCollector in the background and your application in the foreground, which keeps the MicroVM alive:\n\n```\nentrypoint.sh\nbash\n#!/usr/bin/env bash\nset -euo pipefail\n \n# Lambda forwards only stdout and stderr to CloudWatch, so do not redirect the\n# Collector to a file. Writing it into /var/log/app would also make the filelog\n# receiver tail the Collector's own output.\n/usr/local/bin/otelcol-contrib --config /etc/otelcol/config.yaml 2>&1 &\n \nexec python3 -u /app/app.py\n```\n\n### Step 2: Configure the OpenTelemetry Collector\n\n```\notel-collector-config.yaml\nreceivers:\n # On Collector v0.151.0 and newer, use \"host_metrics\" to avoid a deprecation warning.\n hostmetrics:\n collection_interval: 30s\n scrapers:\n cpu:\n metrics:\n # system.cpu.utilization is an optional metric and off by default.\n # Enable it explicitly or it never reaches SigNoz.\n system.cpu.utilization:\n enabled: true\n memory: {}\n load: {}\n filesystem: {}\n network: {}\n \n # On Collector v0.149.0 and newer, use \"file_log\" to avoid a deprecation warning.\n # This only collects anything if your application writes to this path. An\n # application that logs to stdout alone produces no records here.\n #\n # Keep the Collector's own log out of this glob. A Collector that tails its own\n # output re-exports every line it writes.\n filelog:\n include: [/var/log/app/*.log]\n exclude: [/var/log/app/otelcol*.log]\n start_at: beginning\n \n otlp:\n protocols:\n http:\n endpoint: localhost:4318\n \nprocessors:\n batch: {}\n # On Collector v0.153.0 and newer, use \"resource_detection\" to avoid a deprecation warning.\n resourcedetection:\n detectors: [env, system]\n resource:\n attributes:\n # Lambda injects this into every MicroVM. It is the same value the\n # CloudWatch Agent uses for its ImageName dimension.\n - key: aws.lambda.microvm.image_name\n value: ${env:AWS_LAMBDA_MICROVM_IMAGE_NAME}\n action: upsert\n - key: aws.lambda.microvm.image_version\n value: ${env:AWS_LAMBDA_MICROVM_IMAGE_VERSION}\n action: upsert\n \nexporters:\n # On Collector v0.144.0 and newer, use \"otlp_http\" to avoid a deprecation warning.\n otlphttp:\n endpoint: ${env:SIGNOZ_ENDPOINT}\n headers:\n signoz-ingestion-key: ${env:SIGNOZ_INGESTION_KEY}\n \nservice:\n pipelines:\n metrics:\n receivers: [hostmetrics, otlp]\n processors: [resourcedetection, resource, batch]\n exporters: [otlphttp]\n traces:\n receivers: [otlp]\n processors: [resourcedetection, resource, batch]\n exporters: [otlphttp]\n logs:\n receivers: [filelog, otlp]\n processors: [resourcedetection, resource, batch]\n exporters: [otlphttp]\n```\n\nLambda injects `AWS_LAMBDA_MICROVM_IMAGE_NAME`\n\n, `AWS_LAMBDA_MICROVM_IMAGE_ARN`\n\n,\n`AWS_LAMBDA_MICROVM_IMAGE_VERSION`\n\n, and `AWS_REGION`\n\ninto every MicroVM. Using the image name as a\nresource attribute lets you filter and group all three signals by image in SigNoz.\n\n### Step 3: Instrument your application\n\nBecause the MicroVM is long-lived, use standard service-style instrumentation rather than a Lambda layer. Point the SDK at the embedded Collector with these variables:\n\n```\nOTEL_EXPORTER_OTLP_ENDPOINT=\"http://localhost:4318\"\nOTEL_SERVICE_NAME=\"my-microvm-app\"\nOTEL_RESOURCE_ATTRIBUTES=\"deployment.environment=production\"\n```\n\nThere is no environment variable flag on `run-microvm`\n\n, so pass these to\n`--environment-variables`\n\nwhen you build the image in\n[Step 4](#step-4-build-the-microvm-image). Setting them anywhere else has no effect.\n\nSee the [SigNoz instrumentation guides](https://signoz.io/docs/instrumentation/) for your language.\n\nWrite your application logs to `/var/log/app/`\n\nfor the `filelog`\n\nreceiver from\n[Step 2](#step-2-configure-the-opentelemetry-collector) to pick them up, or remove that receiver and\nexport logs over OTLP from the SDK.\n\nYour directory now holds everything the build needs. Zip it with the `Dockerfile`\n\nat the archive\nroot and upload it:\n\n```\nzip -r app.zip Dockerfile entrypoint.sh otel-collector-config.yaml app.py\naws s3 cp app.zip s3:///app.zip\n```\n\n**Verify these values:**\n\n``\n\n: The S3 bucket you created for the code artifact.\n\n### Step 4: Build the MicroVM image\n\nExport your SigNoz destination so the build command can read it:\n\n```\nexport SIGNOZ_ENDPOINT=\"https://ingest..signoz.cloud:443\"\nexport SIGNOZ_INGESTION_KEY=\"\"\n```\n\nThen create the image:\n\n```\naws lambda-microvms create-microvm-image \\\n --name my-monitored-app \\\n --code-artifact uri=s3:///app.zip \\\n --base-image-arn arn:aws:lambda::aws:microvm-image:al2023-1 \\\n --build-role-arn arn:aws:iam:::role/MicrovmBuildRole \\\n --cpu-configurations '[{\"architecture\":\"ARM_64\"}]' \\\n --resources '[{\"minimumMemoryInMiB\":2048}]' \\\n --environment-variables \"SIGNOZ_ENDPOINT=$SIGNOZ_ENDPOINT,SIGNOZ_INGESTION_KEY=$SIGNOZ_INGESTION_KEY,OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318,OTEL_SERVICE_NAME=my-microvm-app\" \\\n --logging '{\"cloudWatch\":{\"logGroup\":\"/aws/lambda-microvms/my-monitored-app\"}}'\n```\n\n**Verify these values:**\n\n``\n\n: Your[SigNoz Cloud region](https://signoz.io/docs/ingestion/signoz-cloud/overview/#endpoint).``\n\n: Your SigNoz[ingestion key](https://signoz.io/docs/ingestion/signoz-cloud/keys/).``\n\n: The S3 bucket holding your code artifact.``\n\n: The AWS region you are deploying to, for example`us-east-1`\n\n. Discover available base images with`aws lambda-microvms list-managed-microvm-images`\n\n.``\n\n: Your AWS account ID.\n\nThe build is asynchronous. Poll until the image reports `CREATED`\n\n:\n\n```\naws lambda-microvms get-microvm-image \\\n --image-identifier arn:aws:lambda:::microvm-image:my-monitored-app\n```\n\n### Step 5: Run the MicroVM\n\n```\naws lambda-microvms run-microvm \\\n --image-identifier arn:aws:lambda:::microvm-image:my-monitored-app \\\n --execution-role-arn arn:aws:iam:::role/MicrovmExecutionRole \\\n --ingress-network-connectors \"arn:aws:lambda::aws:network-connector:aws-network-connector:ALL_INGRESS\" \\\n --idle-policy '{\"maxIdleDurationSeconds\":900,\"suspendedDurationSeconds\":600,\"autoResumeEnabled\":true}' \\\n --maximum-duration-in-seconds 1800\n```\n\nMicroVMs have public internet access on the egress path by default, so the Collector reaches SigNoz without extra configuration. Attach a customer-managed egress connector when you want outbound traffic to route through your VPC instead:\n\n```\n--egress-network-connectors \"\"\n```\n\n**Verify these values:**\n\n``\n\n: The ARN of a`Lambda Network Connector`\n\nin the`ACTIVE`\n\nstate.\n\nCreate that connector with `aws lambda-core create-network-connector`\n\nand wait for it to reach\n`ACTIVE`\n\nbefore you reference it. See\n[Networking for Lambda MicroVMs](https://docs.aws.amazon.com/lambda/latest/dg/microvms-networking.html).\n\n`run-microvm`\n\nreturns a `microvmId`\n\nand a dedicated HTTPS `endpoint`\n\nfor that MicroVM. Every request\nto the endpoint needs a token in the `X-aws-proxy-auth`\n\nheader, and there is no unauthenticated\naccess:\n\n```\naws lambda-microvms create-microvm-auth-token \\\n --microvm-identifier \\\n --expiration-in-minutes 30 \\\n --allowed-ports '[{\"port\":8080}]'\n```\n\n**Verify these values:**\n\n``\n\n: The`microvmId`\n\nreturned by`run-microvm`\n\n.\n\n## Validate\n\nGive the Collector a minute, then open SigNoz:\n\n**Traces**: your`OTEL_SERVICE_NAME`\n\nappears as a service with spans.**Metrics Explorer**:`system.memory.usage`\n\nand`system.cpu.utilization`\n\nare present and filterable by`aws.lambda.microvm.image_name`\n\n.**Logs Explorer**: your application log lines appear with the same resource attribute.\n\nTo confirm the Collector itself started, check the runtime log group\n`/aws/lambda-microvms/`\n\nfor the line `Everything is ready. Begin running and processing data.`\n\n## Troubleshooting\n\n### Image build fails with `tar: command not found`\n\n**Symptom:** The build stops with exit code 127 and `/bin/sh: line 1: tar: command not found`\n\n.\n\n- Likely cause:\n`al2023-minimal`\n\nships without`tar`\n\nand`gzip`\n\n. - Fix: Add\n`RUN dnf install -y tar gzip`\n\nbefore any step that extracts an archive. The same applies to`pgrep`\n\nand other`procps`\n\ntools. - Verify: The build reaches\n`CREATED`\n\n.\n\n### Image build fails for another reason\n\n**Symptom:** The image reports `CREATE_FAILED`\n\n, or `latestFailedImageVersion`\n\nis set.\n\n- Likely cause: A\n`Dockerfile`\n\ninstruction failed. - Fix: Read the build output in the CloudWatch log group you passed to\n`--logging`\n\n, which defaults to`/aws/lambda-microvms/`\n\n. - Verify: The failing instruction appears with its exit code.\n\n### The endpoint returns HTTP 403\n\n**Symptom:** Requests to the MicroVM endpoint return `403 Forbidden`\n\n.\n\n- Likely cause: The token is missing, expired, or invalid, or the target port is not in the token's\n`allowedPorts`\n\n. - Fix: Mint a new token. Requests route to port 8080 unless you send an\n`X-aws-proxy-port`\n\nheader, and whichever port you target must appear in`allowedPorts`\n\n. - Verify: The endpoint returns your application's response.\n\n### The endpoint returns HTTP 502\n\n**Symptom:** Requests to the MicroVM endpoint return `502 Bad Gateway`\n\nwith an empty body.\n\n- Likely cause: Your application is not listening on the target port, it crashed handling the request, or auto-resume did not finish within the retry limit.\n- Fix: Check the runtime log group for a stack trace. Confirm your application listens on port 8080, or send\n`X-aws-proxy-port`\n\nfor a different one. - Verify: The endpoint returns your application's response.\n\n### No `system.cpu.utilization`\n\nin SigNoz\n\n**Symptom:** `system.memory.usage`\n\nappears but `system.cpu.utilization`\n\nreturns no data.\n\n- Likely cause:\n`system.cpu.utilization`\n\nis an optional`hostmetrics`\n\nmetric, disabled by default. - Fix: Enable it explicitly under the\n`cpu`\n\nscraper as shown in[Step 2](#step-2-configure-the-opentelemetry-collector). - Verify: The metric appears in\n**Metrics Explorer**.\n\n### No logs in SigNoz\n\n**Symptom:** Traces and metrics arrive but the Logs Explorer is empty.\n\n- Likely cause: The\n`filelog`\n\nreceiver watches`/var/log/app/*.log`\n\n, and your application writes to stdout only. - Fix: Write logs to a file under\n`/var/log/app/`\n\n, or drop`filelog`\n\nand export logs over OTLP from the SDK. - Verify: Records appear in\n**Logs Explorer**.\n\n### The Collector logs `Configuration references unset environment variable`\n\n**Symptom:** The Collector cannot resolve `${env:SIGNOZ_ENDPOINT}`\n\nor `${env:SIGNOZ_INGESTION_KEY}`\n\n.\n\n- Likely cause: An\n`update-microvm-image`\n\ncall omitted`--environment-variables`\n\n, which drops them from the new version. - Fix: Pass\n`--environment-variables`\n\non every update. - Verify: The Collector starts and exports without warnings.\n\n### The API returns HTTP 502 intermittently\n\n**Symptom:** `run-microvm`\n\n, `get-microvm`\n\n, or `list-microvm-images`\n\nfails with `Bad Gateway`\n\n.\n\n- Likely cause: A transient service error. AWS does not document 502 for these APIs, but it occurs in practice.\n- Fix: Retry with exponential backoff. Scripts that poll these APIs need retry handling. AWS documents\n`ThrottlingException`\n\nand`InternalServerException`\n\nas retryable, and`ResourceNotFoundException`\n\nwhen the image is not yet`CREATED`\n\n. - Verify: The call succeeds on a later attempt.\n\n## Limitations\n\n**The CloudWatch Agent cannot forward to SigNoz.** AWS's recommended metrics agent for MicroVMs runs on Telegraf and the OpenTelemetry Collector. Its Collector build has no`otlp`\n\nexporter. Its only OTLP exporter,`otlphttp`\n\n, rejects non-AWS endpoints with`invalid AWS endpoint`\n\n. The agent exits on either config, which also stops the CloudWatch metrics you already had. Run a separate OpenTelemetry Collector as described above. To get CloudWatch data into SigNoz, pull the`/aws/lambda-microvms/`\n\nlog group and the`LambdaMicroVms/Application`\n\nnamespace through the[AWS monitoring integration](https://signoz.io/docs/aws-monitoring/overview/).**Host metrics reflect guest-visible resources.**`hostmetrics`\n\nreads what the guest kernel reports. Those totals differ from the memory and vCPU baseline you configured, so read them as relative signals rather than capacity numbers.**Metrics stop while a MicroVM is suspended.** The embedded Collector survives suspend and resume without restarting. No host metrics are produced during the suspended window, so expect gaps in dashboards.\n\n## Optional: CloudTrail lifecycle and audit events\n\nMicroVM lifecycle actions are CloudTrail **data events**, and CloudTrail does not log them by\ndefault. Enable them with an advanced event selector on the `AWS::Lambda::MicrovmImage`\n\nresource\ntype.\n\nThe selectors below keep management events and add MicroVM data events, so they are safe to apply to a trail that was logging management events by default:\n\n```\naws cloudtrail put-event-selectors \\\n --trail-name \\\n --advanced-event-selectors '[\n {\n \"Name\": \"Keep management events\",\n \"FieldSelectors\": [\n { \"Field\": \"eventCategory\", \"Equals\": [\"Management\"] }\n ]\n },\n {\n \"Name\": \"Log Lambda MicroVM data events\",\n \"FieldSelectors\": [\n { \"Field\": \"eventCategory\", \"Equals\": [\"Data\"] },\n { \"Field\": \"resources.type\", \"Equals\": [\"AWS::Lambda::MicrovmImage\"] }\n ]\n }\n ]'\n```\n\n**Verify these values:**\n\n``\n\n: The name of an existing CloudTrail trail in the same region.\n\nThe second selector captures `RunMicrovm`\n\n, `TerminateMicrovm`\n\n, `SuspendMicrovm`\n\n, `ResumeMicrovm`\n\n,\n`CreateMicrovmAuthToken`\n\n, and `CreateMicrovmShellAuthToken`\n\n. Delivery to S3 takes a few minutes.\nForward the stream into SigNoz using the existing\n[AWS logs ingestion patterns](https://signoz.io/docs/aws-monitoring/one-click-vs-manual/) to build\nsuspend and resume timelines or security audit dashboards.\n\nImage and MicroVM management calls such as `CreateMicrovmImage`\n\nand `ListMicrovms`\n\nare management\nevents. CloudTrail logs them by default, and the first selector above preserves that. See\n[Monitoring for Lambda MicroVMs](https://docs.aws.amazon.com/lambda/latest/dg/microvms-monitoring.html)\nfor the full event list.\n\n## Next steps\n\n[Set up alerts](https://signoz.io/docs/alerts-management/metrics-based-alerts/)on host metrics and error rates.[Build dashboards](https://signoz.io/docs/userguide/manage-dashboards/)grouped by`aws.lambda.microvm.image_name`\n\n.[Correlate traces and logs](https://signoz.io/docs/traces-management/guides/correlate-traces-and-logs/)to move between signals during an investigation.[Monitor AWS Lambda functions](https://signoz.io/docs/aws-monitoring/lambda/)for classic, per-invocation Lambda.\n\n## Get Help\n\nIf you need help with the steps in this topic, please reach out to us on [SigNoz Community Slack](https://signoz.io/slack/). If you are a SigNoz Cloud user, please use in product chat support located at the bottom right corner of your SigNoz instance or contact us at [cloud-support@signoz.io](mailto:cloud-support@signoz.io).", "url": "https://wpnews.pro/news/monitor-aws-lambda-microvms-with-opentelemetry-collector", "canonical_source": "https://signoz.io/docs/aws-monitoring/lambda-microvms", "published_at": "2026-08-20 00:00:00+00:00", "updated_at": "2026-08-25 06:43:46.654627+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools"], "entities": ["AWS Lambda", "SigNoz", "OpenTelemetry Collector", "Firecracker"], "alternates": {"html": "https://wpnews.pro/news/monitor-aws-lambda-microvms-with-opentelemetry-collector", "markdown": "https://wpnews.pro/news/monitor-aws-lambda-microvms-with-opentelemetry-collector.md", "text": "https://wpnews.pro/news/monitor-aws-lambda-microvms-with-opentelemetry-collector.txt", "jsonld": "https://wpnews.pro/news/monitor-aws-lambda-microvms-with-opentelemetry-collector.jsonld"}}