{"slug": "running-the-opentelemetry-collector-as-a-lambda", "title": "Running the OpenTelemetry Collector as a Lambda", "summary": "The OpenTelemetry Collector can be packaged as a Lambda container image and fronted with a Function URL, enabling intermittent telemetry processing at near-zero cost. The architecture uses the AWS Lambda Web Adapter to bridge the Lambda Runtime API with the Collector's OTLP/HTTP receiver, supporting stateless processing for low-volume, asynchronous telemetry.", "body_md": "# Running the OpenTelemetry Collector as a Lambda\n\nThe OpenTelemetry Collector is usually deployed as a long-running process, but when telemetry is rare, it makes sense to run it as a Lambda function instead. Here's how to do it.\n\n#### Span or Attribute in OpenTelemetry Custom Instrumentation\n\nWhen you want to add some information to your tracing telemetry, you could emit a log, create a span, or add a piece of data to your current span. Adding a piece of data to your current span is the best! Usually.\n\n[Learn More](/blog/span-or-attribute-opentelemetry-custom-instrumentation)\n\nThe OpenTelemetry Collector is usually deployed as a long-running process: a sidecar, a DaemonSet, an EC2 instance, a docker container on my computer. It sits there listening for telemetry. That's fine when I want to send telemetry all day, but not when telemetry is rare. Like right now, when I have an agent defined on AgentCore, and it runs a few times a week maybe. Or my website that hardly sees any traffic.\n\nCan I run the OpenTelemetry Collector as a Lambda function? Sounds tricky, but hey, that's what my coding assistant is for! Here’s how we got it working.\n\n# Learn more about Honeycomb Intelligence\n\nConnect with our experts today.\n\n*(From here, this post is agent-written. Point your coding agent at this post and ask it, “How will this work in my environment?”)*\n\nYou can package the OTel Collector as a Lambda container image and front it with a Function URL. Producers send OTLP/HTTP to the URL; the collector runs whatever processors you configure; the result goes on to your backend.\n\nAt low volume this costs essentially nothing: a Lambda below the free tier rounds to zero. Cold start is around 4 seconds; warm invocations are 2–4 ms.\n\nThe full file set (Dockerfile, `config.yaml`\n\n, and bootstrap/build/deploy scripts) is available as a [companion gist](https://gist.github.com/jessitron/f2cc160cb5e635d1c60752b5f2d038a8). The rest of the post explains what each piece does.\n\n## When this fits\n\nThe Lambda shape works when:\n\n- Traffic is intermittent—single-digit requests per second peak, often zero.\n- The producer exports asynchronously (a\n`BatchSpanProcessor`\n\n, an AgentCore runtime, another Lambda) so cold start is invisible to the user. - The processing you want is stateless per request—OTTL\n`transform`\n\n,`filter`\n\n,`attributes`\n\n, redaction, routing.\n\nIt does not fit when:\n\n- Sustained throughput exceeds about 1 request per second. Lambda's per-invocation overhead and pricing stop being free.\n- You need queuing or retry. If the backend is briefly unavailable, in-Lambda retry state dies with the process. Producer-side retry is your only safety net.\n- You're scraping metrics or logs. This is only for logs and traces that are pushed to the collector.\n\n## Architecture\n\n```\nProducer (any OTLP/HTTP client)\n   │  OTLP/HTTP/protobuf, bearer token in Authorization header\n   ▼\nLambda Function URL  (auth_type=NONE; bearer enforced inside the collector)\n   │\nLambda container image\n   ├─ AWS Lambda Web Adapter (extension)\n   │    polls the Lambda Runtime API, forwards each invocation as an HTTP\n   │    request to localhost:4318\n   └─ otelcol-contrib  (CMD, not ENTRYPOINT)\n        ├─ otlp receiver (bearertokenauth)\n        ├─ your processors\n        └─ otlphttp exporter (sending_queue disabled)\n             → backend (e.g. api.honeycomb.io)\n```\n\n[AWS Lambda Web Adapter](https://github.com/awslabs/aws-lambda-web-adapter) is the load-bearing piece. It registers as a Lambda extension, exposes the Lambda Runtime API as an HTTP listener on a port of your choosing, and forwards each invocation to your container's HTTP server. The Collector's OTLP/HTTP receiver is an HTTP server. LWA bridges them.\n\n**A note on auth.** The bearer token in the diagram is optional but recommended. The Collector's OTLP receiver works without it. But a Function URL with `auth_type=NONE`\n\nis publicly reachable, so without a check inside the collector, anyone who learns the URL can send data through it to your backend—running up your ingest bill and polluting your telemetry. The hostname is random, but URLs leak (commit history, screenshots, packet captures). A static bearer token, shared between authorized producers and this Lambda, raises the bar from \"anyone with the URL\" to \"anyone with the URL and the token.\" Details in the Authentication section below.\n\n## The container image\n\nUse a multi-stage build. The official `otel/opentelemetry-collector-contrib`\n\nimage is distroless and runs as `USER 10001`\n\nwith no `/etc/passwd`\n\n, both of which the Lambda container runtime trips on. Copy the binary into Alpine and run from there.\n\n```\nFROM otel/opentelemetry-collector-contrib:0.151.0 AS collector\n\nFROM alpine:3.20\nRUN apk add --no-cache ca-certificates\n\nCOPY --from=collector /otelcol-contrib /app/otelcol-contrib\nCOPY --from=public.ecr.aws/awsguru/aws-lambda-adapter:1.0.0 /lambda-adapter /opt/extensions/lambda-adapter\nCOPY config.yaml /etc/otel/config.yaml\n\nENV AWS_LWA_PORT=4318\nENV AWS_LWA_INVOKE_MODE=buffered\nENV AWS_LWA_READINESS_CHECK_PATH=/\n\nCMD [\"/app/otelcol-contrib\", \"--config=/etc/otel/config.yaml\"]\n```\n\nWhat each piece does:\n\n—Alpine ships without trusted CA roots. The`ca-certificates`\n\n`otlphttp`\n\nexporter does HTTPS to the backend; without this it fails at TLS.**LWA at**—Lambda discovers extensions in this directory automatically at container start. LWA ships its own Runtime Interface Client, so you do not need a separate RIC.`/opt/extensions/`\n\n—The OTLP/HTTP receiver's default port. LWA forwards Function URL invocations here.`AWS_LWA_PORT=4318`\n\n—One response per request. The other option, response_stream, is for Server-Sent Events.`AWS_LWA_INVOKE_MODE=buffered`\n\n—LWA polls this path until something responds before forwarding traffic. The OTLP receiver returns 404 on /, which counts as ready (any HTTP response is enough).`AWS_LWA_READINESS_CHECK_PATH=/`\n\n—Lambda treats CMD-only and ENTRYPOINT+CMD images differently at boot. With ENTRYPOINT set, the main process never starts. See troubleshooting below.`CMD`\n\n, not`ENTRYPOINT`\n\n## The configuration\n\nThree things shape the config: the receiver needs bearer-token auth, the processing must be stateless per request, and the exporter must not queue.\n\n```\nextensions:\n  bearertokenauth/ingest:\n    scheme: Bearer\n    token: ${env:INGEST_BEARER_TOKEN}\n\nreceivers:\n  otlp:\n    protocols:\n      http:\n        endpoint: 0.0.0.0:4318\n        auth:\n          authenticator: bearertokenauth/ingest\n\nprocessors:\n  # Your processors here. Examples: transform/, filter/, attributes/.\n  # Do NOT include a batch processor.\n\nexporters:\n  otlphttp/honeycomb:\n    endpoint: https://api.honeycomb.io\n    headers:\n      x-honeycomb-team: ${env:HONEYCOMB_API_KEY}\n    sending_queue:\n      enabled: false\n\nservice:\n  extensions: [bearertokenauth/ingest]\n  pipelines:\n    traces:\n      receivers: [otlp]\n      processors: [...]\n      exporters: [otlphttp/honeycomb]\n```\n\nThis sends to Honeycomb. To send elsewhere, swap the exporter—any OTLP/HTTP backend works with the same shape (change the `endpoint`\n\nand the auth header name and value).\n\nTwo non-obvious settings:\n\n**No** Lambda freezes the container after the handler returns. Spans sitting in a`batch`\n\nprocessor.`batch`\n\nprocessor never flush—they stay in memory until the next cold start (which discards them) or the next invocation (which may or may not arrive). Symptom: the exporter reports 200 in milliseconds, the backend never sees the trace.on every exporter. Same reason. The default async queue holds spans in memory; the freeze strands them.`sending_queue.enabled: false`\n\nThe official fix for these is the `decouple`\n\nprocessor from the `opentelemetry-lambda`\n\ncollector distribution, which knows to flush before the handler returns. It is not in `otel/opentelemetry-collector-contrib`\n\n. If you stick with the contrib image, synchronous export is the correct shape.\n\n### Authentication\n\nBearer auth is optional but recommended; see the note in Architecture for why. This section covers how.\n\nFunction URLs support `auth_type=AWS_IAM`\n\nand `auth_type=NONE`\n\n. IAM would be the strict answer, but it requires Sigv4-signing the OTLP requests on the producer side, and the OpenTelemetry SDKs do not sign with Sigv4. Writing a Sigv4-signing OTLP exporter is more work than this whole pattern is worth.\n\nThe pragmatic answer is `auth_type=NONE`\n\non the Function URL plus the `bearertokenauth`\n\nextension inside the collector. The bearer token is a shared secret between the producer's environment and the Lambda's environment. If it leaks, rotate it.\n\nThe producer sets:\n\n```\nOTEL_EXPORTER_OTLP_HEADERS=authorization=Bearer <token>\n```\n\nThe header name is case-insensitive at the receiver. The token value is not.\n\nIf you skip bearer auth, drop the `extensions`\n\nblock, the `auth:`\n\nstanza on the receiver, and the `extensions: [bearertokenauth/ingest]`\n\nline under `service`\n\n. Everything else in the config is the same.\n\n## Building, pushing, deploying\n\nThe [companion gist](https://gist.github.com/jessitron/f2cc160cb5e635d1c60752b5f2d038a8) packages the commands below as `bootstrap.sh`\n\n, `build.sh`\n\n, and `deploy.sh`\n\nif you'd rather run scripts than read shell out of a blog post.\n\n### Build\n\n```\ndocker buildx build \\\n  --platform linux/arm64 \\\n  --provenance=false \\\n  --sbom=false \\\n  --load \\\n  -t collector:local .\n```\n\n`--provenance=false --sbom=false`\n\nare required. Default `buildx`\n\noutput is an OCI image manifest with attestations, which Lambda rejects with `InvalidParameterValueException: The image manifest, config or layer media type for the source image ... is not supported.`\n\n### Push to ECR\n\nPush the image to ECR. The ECR repository needs to exist first; see [Appendix: ECR setup](https://github.com/jessitron/cynditaylor-com-bot/blob/main/notes/blog/running-otel-collector-as-a-lambda.md#ecr-setup) for the one-time commands.\n\n### Create the Lambda\n\nThe Lambda needs an execution role with `AWSLambdaBasicExecutionRole`\n\nand a trust policy that lets `lambda.amazonaws.com`\n\nassume it; see [Appendix: Lambda execution role](https://github.com/jessitron/cynditaylor-com-bot/blob/main/notes/blog/running-otel-collector-as-a-lambda.md#lambda-execution-role) for the one-time setup.\n\n```\naws lambda create-function \\\n  --function-name collector \\\n  --package-type Image \\\n  --code \"ImageUri=$ACCOUNT.dkr.ecr.$REGION.amazonaws.com/$REPO:latest\" \\\n  --role \"arn:aws:iam::$ACCOUNT:role/CollectorLambda\" \\\n  --architectures arm64 \\\n  --memory-size 512 \\\n  --timeout 30 \\\n  --environment \"Variables={INGEST_BEARER_TOKEN=...,HONEYCOMB_API_KEY=...}\"\n\naws lambda wait function-active-v2 --function-name collector\n```\n\n512 MB and 30 s are comfortable defaults. The collector itself uses much less; allocating more memory gives Lambda proportionally more CPU, which shortens cold start.\n\n### Create the Function URL\n\n```\naws lambda create-function-url-config \\\n  --function-name collector \\\n  --auth-type NONE\n```\n\n### Permissions — both statements are required\n\nThis is general Lambda Function URL behavior as of October 2025, not specific to the collector. But it's the single most common silent failure when first deploying, so it lives here in the main flow rather than in an appendix. `lambda:InvokeFunctionUrl`\n\nalone is not sufficient to allow public Function URL invocations; you also need `lambda:InvokeFunction`\n\nwith `--invoked-via-function-url`\n\n:\n\n```\naws lambda add-permission \\\n  --function-name collector \\\n  --statement-id FunctionURLAllowInvokeUrl \\\n  --action lambda:InvokeFunctionUrl \\\n  --principal '*' \\\n  --function-url-auth-type NONE\n\naws lambda add-permission \\\n  --function-name collector \\\n  --statement-id FunctionURLAllowInvoke \\\n  --action lambda:InvokeFunction \\\n  --principal '*' \\\n  --invoked-via-function-url\n```\n\nWithout the second statement, every request gets a 403 `AccessDeniedException`\n\nat the URL gate and Lambda does not invoke the container—there are no CloudWatch log lines to debug from. AWS documents the dual-permission rule at [https://docs.aws.amazon.com/lambda/latest/dg/urls-auth.html](https://docs.aws.amazon.com/lambda/latest/dg/urls-auth.html).\n\n## Verification\n\nAfter deploy, send a synthetic OTLP request and check three things.\n\n**1. The URL gate accepts auth.** Send an invalid body with the correct bearer:\n\n```\ncurl -i -X POST \"${URL}v1/traces\" \\\n  -H 'Content-Type: application/x-protobuf' \\\n  -H \"Authorization: Bearer ${TOKEN}\" \\\n  --data-binary 'x'\n```\n\nExpected: **400**. The collector parses the invalid OTLP body and rejects it. A 400 here means the Function URL accepted the request, LWA forwarded it, and the collector ran. If you see 403, see troubleshooting. If you see 401, the bearer token in the producer doesn't match the one in the Lambda's environment.\n\n**2. A real OTLP request succeeds.** Use any OTel SDK with `OTLPSpanExporter`\n\nand a `SimpleSpanProcessor`\n\n(not `BatchSpanProcessor`\n\n—for verification you want the export's return value to reflect the actual export status). Capture the result of `span_exporter.export(...)`\n\n. It should be `SUCCESS`\n\n. For a curl-only alternative, the `sample-span.json`\n\n+ curl pattern in [Testing an OpenTelemetry Collector deployed as a Daemonset in Kubernetes](https://jessitron.com/2023/09/08/testing-an-opentelemetry-collector-deployed-as-a-daemonset-in-kubernetes/) works the same way against a Function URL—point the curl at `${URL}v1/traces`\n\nand add the bearer header.\n\n**3. The span lands in your backend.** Query by trace ID or service name. If you're sending to Honeycomb, the [Honeycomb MCP server](https://github.com/honeycombio/honeycomb-mcp) lets your editor or CLI query traces directly—`get_trace`\n\nwith the trace ID returns the span shape without opening the UI, which makes \"did my one test span land\" a one-line check. If the producer reported `SUCCESS`\n\nbut the backend shows nothing, the most likely cause is an enabled `sending_queue`\n\nor a `batch`\n\nprocessor—see troubleshooting.\n\n## CloudWatch volume\n\nApproximate per-invocation log output:\n\n**Cold start:**~12 lines (collector startup banners plus Lambda runtime START/END/REPORT).** Warm invocation:**3 lines (Lambda runtime only; the collector emits nothing during steady-state processing at`info`\n\nlevel).**Container retirement**(Lambda recycles containers after idle): 4 lines (graceful shutdown).\n\nAt 100 invocations per day with one cold start, this is roughly 315 lines and 10 KB per day—about 4 MB per year. At CloudWatch's $0.50/GB ingest rate, that's roughly **$0.002 per year**.\n\n## Troubleshooting\n\nSymptoms grouped by where the failure surfaces. Most of these are silent—they don't produce a useful error message—so the symptom-to-cause table matters more than usual.\n\n### 403 `AccessDeniedException`\n\nat the Function URL, no CloudWatch log lines\n\nThe dual-permission gotcha. `lambda:InvokeFunctionUrl`\n\nalone is no longer sufficient on `auth_type=NONE`\n\nFunction URLs. Add the second `lambda:InvokeFunction`\n\nstatement with `--invoked-via-function-url`\n\n. See the Permissions section above.\n\nThis one is especially confusing because Lambda doesn't invoke the container at all—there are no logs anywhere, the collector is fine, and you'll spend time investigating SCPs, account-level blocks, or the resource policy itself before realizing the gate rejected the request before reaching Lambda.\n\n### 401 from the collector\n\nThe bearer token in the request doesn't match `INGEST_BEARER_TOKEN`\n\nin the Lambda's environment. Compare them. The header name is case-insensitive but the token value is not.\n\n### Producer reports `SUCCESS`\n\n, backend never sees the trace\n\nA `batch`\n\nprocessor or an enabled `sending_queue`\n\nis holding spans in memory across the Lambda freeze. Remove the batch processor and set `sending_queue.enabled: false`\n\non every exporter.\n\nYou can confirm this is the cause by setting the collector's log level to `debug`\n\ntemporarily—you'll see the spans arrive at the exporter but no export attempt before the invocation ends. Set it in `config.yaml`\n\n:\n\n```\nservice:\n  telemetry:\n    logs:\n      level: debug\n```\n\nRebuild and redeploy. Switch back to `info`\n\nonce you've diagnosed the issue; `debug`\n\nis noisy enough to matter for CloudWatch volume.\n\n`InvalidParameterValueException: image manifest ... is not supported`\n\nwhen creating the function\n\nDefault `docker buildx`\n\noutput is an OCI image manifest with build attestations, which Lambda's image-pull path rejects. Rebuild with `--provenance=false --sbom=false`\n\n.\n\n### LWA logs `app is not ready after 2000ms`\n\nrepeatedly, then init times out at 10 s\n\nThe collector process never started. One cause is having both `ENTRYPOINT`\n\nand `CMD`\n\nin the Dockerfile—Lambda's container init handles CMD-only and ENTRYPOINT+CMD images differently, and with `ENTRYPOINT`\n\nset the main process doesn't come up. Use `CMD`\n\nonly. This reproduces across distroless, `provided:al2023`\n\n, and Alpine bases; it is a Lambda-runtime behavior, not a base-image issue.\n\nLess common: the collector started but bound to a different port than `AWS_LWA_PORT`\n\n. Confirm the OTLP receiver's HTTP endpoint matches.\n\n### Container won't start; logs mention permissions or `/etc/passwd`\n\nYou might be running the official `otel/opentelemetry-collector-contrib`\n\nimage directly. It is distroless and runs as `USER 10001`\n\nwith no `/etc/passwd`\n\n. Stage the binary into Alpine via a multi-stage build instead.\n\n`x509: certificate signed by unknown authority`\n\nfrom the exporter\n\nAlpine base without `ca-certificates`\n\n. Add `RUN apk add --no-cache ca-certificates`\n\nto the Dockerfile.\n\n### Cold start is much longer than 4 seconds\n\nIncrease `--memory-size`\n\n. Lambda allocates CPU proportional to memory; at 128 MB the cold start can run into double-digit seconds. 512 MB is a comfortable default for the contrib binary.\n\n### Function URL is reachable but every request returns 500 with no detail\n\nCheck the collector's logs in CloudWatch. The most common causes are a missing required env var (the config references `${env:FOO}`\n\nand `FOO`\n\nis not set) or an invalid config file that survived deploy because no one ran `otelcol validate`\n\non it. Run `otelcol-contrib validate --config=/etc/otel/config.yaml`\n\ninside the built image as part of CI.\n\n### The backend is briefly down and traces are lost\n\nExpected. Lambda has no cross-invocation buffer; if the backend returns 5xx, the in-process retry inside that invocation runs and then the container freezes. Producer-side `BatchSpanProcessor`\n\nretry covers most short outages—that is the only buffer you have in this shape. If you need durable buffering, this pattern is the wrong fit; use a persistent collector instead.\n\n## Appendix: one-time AWS setup\n\nThe main flow above assumes the ECR repository and Lambda execution role already exist. These are the one-time commands to create them.\n\n### ECR setup\n\n```\nACCOUNT=$(aws sts get-caller-identity --query Account --output text)\nREGION=us-west-2\nREPO=collector\n\naws ecr create-repository --repository-name \"$REPO\" --region \"$REGION\"\n```\n\n`create-repository`\n\nerrors if the repo already exists; safe to ignore.\n\nTo push to it (from the main flow):\n\n```\naws ecr get-login-password --region \"$REGION\" \\\n  | docker login --username AWS --password-stdin \"$ACCOUNT.dkr.ecr.$REGION.amazonaws.com\"\n\ndocker tag collector:local \"$ACCOUNT.dkr.ecr.$REGION.amazonaws.com/$REPO:latest\"\ndocker push \"$ACCOUNT.dkr.ecr.$REGION.amazonaws.com/$REPO:latest\"\n```\n\n### Lambda execution role\n\nThe role needs two pieces: a trust policy letting Lambda assume it, and a permissions policy letting it write CloudWatch logs. `AWSLambdaBasicExecutionRole`\n\nis the managed policy that covers the logs.\n\n```\ncat > trust.json <<'EOF'\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [{\n    \"Effect\": \"Allow\",\n    \"Principal\": {\"Service\": \"lambda.amazonaws.com\"},\n    \"Action\": \"sts:AssumeRole\"\n  }]\n}\nEOF\n\naws iam create-role \\\n  --role-name CollectorLambda \\\n  --assume-role-policy-document file://trust.json\n\naws iam attach-role-policy \\\n  --role-name CollectorLambda \\\n  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole\n```\n\nThe role ARN goes into `--role`\n\non `aws lambda create-function`\n\nin the main flow.\n\n## Want to know more?\n\nLearn how to get the most value out of your telemetry data with Honeycomb Telemetry Pipeline.", "url": "https://wpnews.pro/news/running-the-opentelemetry-collector-as-a-lambda", "canonical_source": "https://www.honeycomb.io/blog/running-opentelemetry-collector-lambda", "published_at": "2026-06-08 20:30:00+00:00", "updated_at": "2026-06-30 12:24:15.361357+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure"], "entities": ["OpenTelemetry", "AWS Lambda", "Honeycomb", "AWS Lambda Web Adapter", "OTLP"], "alternates": {"html": "https://wpnews.pro/news/running-the-opentelemetry-collector-as-a-lambda", "markdown": "https://wpnews.pro/news/running-the-opentelemetry-collector-as-a-lambda.md", "text": "https://wpnews.pro/news/running-the-opentelemetry-collector-as-a-lambda.txt", "jsonld": "https://wpnews.pro/news/running-the-opentelemetry-collector-as-a-lambda.jsonld"}}