{"slug": "trace-multi-agent-llm-pipelines-with-langfuse-and-opentelemetry", "title": "Trace Multi-Agent LLM Pipelines with Langfuse and OpenTelemetry", "summary": "Langfuse 4.14.4 and OpenTelemetry can be used to trace a two-agent Claude pipeline, capturing every LLM call, tool invocation, token count, and dollar cost in a single nested trace timeline. The tutorial, by Priya Nair, demonstrates instrumenting a research agent and a writer agent using the Langfuse v4 SDK and OpenLLMetry's AnthropicInstrumentor 0.62.3, with the Langfuse SDK acting as an OTel tracer provider to automatically collect spans. The setup requires Python 3.10+, a Langfuse account, and an Anthropic API key, with the LANGFUSE_BASE_URL env var matching the signup region to avoid silent trace loss.", "body_md": "# Trace Multi-Agent LLM Pipelines with Langfuse and OpenTelemetry\n\nInstrument a two-agent Claude pipeline so every LLM call, tool hop, and token cost lands in one trace.\n\n[Priya Nair](https://sourcefeed.dev/u/priya_nair)\n\n## What you'll build\n\nA two-agent LLM pipeline (a research agent that calls a tool, then a writer agent) fully instrumented with [Langfuse](https://langfuse.com) over [OpenTelemetry](https://opentelemetry.io), so every LLM call, tool invocation, token count, and dollar cost shows up as a nested span in one queryable trace timeline.\n\n## Prerequisites\n\n**Python 3.10+**(both`langfuse`\n\nand the instrumentor require ≥3.10). Verified with Python 3.12.**Package versions verified for this tutorial:**`langfuse`\n\n4.14.4 (the v4 SDK, rewritten on OpenTelemetry — note the env var is now`LANGFUSE_BASE_URL`\n\n, not v3's`LANGFUSE_HOST`\n\n),`opentelemetry-instrumentation-anthropic`\n\n0.62.3 (from[OpenLLMetry](https://github.com/traceloop/openllmetry)), and a recentSDK (0.116+).`anthropic`\n\n- A\n**Langfuse account**— the[Cloud free tier](https://cloud.langfuse.com)works; note whether you signed up in the EU or US region. Self-hosted works identically, just point`LANGFUSE_BASE_URL`\n\nat your instance. - An\n**Anthropic API key** from the[Claude Console](https://platform.claude.com). Any OTel-instrumented provider works the same way; this tutorial uses Claude. - OS: anything with a shell. Commands below are macOS/Linux; on Windows use\n`set`\n\ninstead of`export`\n\n.\n\n## 1. Create a Langfuse project and grab keys\n\nSign in at [cloud.langfuse.com](https://cloud.langfuse.com) (or `us.cloud.langfuse.com`\n\nfor the US region), create an organization and a project, then go to **Project Settings → API Keys → Create new API keys**. You get a public key (`pk-lf-...`\n\n) and a secret key (`sk-lf-...`\n\n). The secret is shown once — copy both now.\n\n## 2. Install dependencies and set environment variables\n\n```\npython -m venv .venv && source .venv/bin/activate\npip install \"langfuse>=4.14\" \"opentelemetry-instrumentation-anthropic>=0.62\" anthropic\nexport LANGFUSE_PUBLIC_KEY=\"pk-lf-...\"\nexport LANGFUSE_SECRET_KEY=\"sk-lf-...\"\nexport LANGFUSE_BASE_URL=\"https://cloud.langfuse.com\"   # or https://us.cloud.langfuse.com\nexport ANTHROPIC_API_KEY=\"sk-ant-...\"\n```\n\n`LANGFUSE_BASE_URL`\n\nmust match the region you signed up in — this is the single most common source of silent trace loss.\n\n## 3. Understand the two instrumentation layers\n\nYou're wiring together two things, and it helps to know who does what:\n\n(OpenLLMetry) monkey-patches the Anthropic SDK and emits a standard OTel span for every API call, carrying`AnthropicInstrumentor`\n\n`gen_ai.*`\n\nsemantic-convention attributes: model, prompt, completion, and token usage.**The Langfuse v4 SDK** is itself an OTel tracer provider. Any span emitted by any OTel instrumentation library while Langfuse is initialized lands inside your Langfuse trace tree automatically — no exporter config, no OTLP endpoint wrangling. Langfuse maps`gen_ai.usage.*`\n\nto token counts and multiplies them against its built-in model price list (updated daily against provider docs) to compute cost per generation.\n\nYour own agent and tool functions become spans via Langfuse's `@observe`\n\ndecorator, and the auto-instrumented LLM spans nest under whichever `@observe`\n\nfunction made the call.\n\n## 4. Build the instrumented pipeline\n\nSave this as `pipeline.py`\n\n. It's the complete, runnable file:\n\n``` python\nimport os\n\nfrom anthropic import Anthropic\nfrom langfuse import get_client, observe, propagate_attributes\nfrom opentelemetry.instrumentation.anthropic import AnthropicInstrumentor\n\n# 1. Patch the Anthropic SDK BEFORE making any calls.\nAnthropicInstrumentor().instrument()\n\n# 2. Initialize Langfuse (reads LANGFUSE_* env vars) and fail fast on bad creds.\nlangfuse = get_client()\nif not langfuse.auth_check():\n    raise SystemExit(\"Langfuse rejected credentials — check keys and LANGFUSE_BASE_URL region\")\n\nclient = Anthropic()\nMODEL = \"claude-opus-5\"\n\ndef ask(system: str, prompt: str) -> str:\n    # Opus 5 thinks by default and max_tokens caps thinking + answer together,\n    # so leave generous headroom.\n    response = client.messages.create(\n        model=MODEL,\n        max_tokens=16000,\n        system=system,\n        messages=[{\"role\": \"user\", \"content\": prompt}],\n    )\n    return \"\".join(b.text for b in response.content if b.type == \"text\")\n\n@observe()  # tool invocation -> its own span, input/output captured\ndef fetch_release_notes(project: str) -> str:\n    # Stub tool: swap in a real HTTP call or DB query.\n    return (\n        f\"{project} changelog: v4 SDK is OTel-native; ingestion adds \"\n        \"x-langfuse-ingestion-version=4; cost table now audited daily.\"\n    )\n\n@observe(name=\"research-agent\")\ndef research(topic: str) -> str:\n    notes = fetch_release_notes(topic)\n    return ask(\n        \"You are a research agent. Extract the three most important facts as bullets.\",\n        f\"Source material:\\n{notes}\",\n    )\n\n@observe(name=\"writer-agent\")\ndef write_summary(facts: str) -> str:\n    return ask(\n        \"You are a writing agent. Turn these facts into a two-sentence executive summary.\",\n        facts,\n    )\n\n@observe(name=\"research-pipeline\")\ndef run_pipeline(topic: str) -> str:\n    return write_summary(research(topic))\n\nif __name__ == \"__main__\":\n    # Attributes set here propagate to every span in the trace,\n    # so you can filter/group traces by session or user in the UI.\n    with propagate_attributes(session_id=\"demo-session-1\", user_id=\"tutorial-reader\"):\n        print(run_pipeline(\"Langfuse\"))\n    langfuse.flush()  # spans are buffered; short scripts must flush before exit\n```\n\nTwo details matter more than they look. `AnthropicInstrumentor().instrument()`\n\nruns before anything else so every SDK call is patched. And `langfuse.flush()`\n\nruns last — the SDK exports spans on a background thread, and a script that exits without flushing loses the trace.\n\nPrivacy note: the instrumentor records prompts and completions into span attributes by default. Set `TRACELOOP_TRACE_CONTENT=false`\n\nto keep payloads out of your traces.\n\n## 5. Run it\n\n```\npython pipeline.py\n```\n\nThe script prints a two-sentence summary, e.g.:\n\n```\nLangfuse's v4 SDK is now built natively on OpenTelemetry, with ingestion\ntagged via x-langfuse-ingestion-version=4. Its model cost table is audited\ndaily, keeping per-generation pricing accurate.\n```\n\n## Verify it works\n\nOpen your project in the Langfuse UI and click **Tracing → Traces**. You should see a trace named `research-pipeline`\n\nwithin a few seconds. Click it and check:\n\n**The timeline nests correctly:**`research-pipeline`\n\n→`research-agent`\n\n→ (`fetch_release_notes`\n\nspan + one`anthropic.chat`\n\ngeneration), then`writer-agent`\n\n→ a second generation. Two generations total.**Each generation shows**`claude-opus-5`\n\nas the model, the full prompt and completion, input/output token counts, latency, and a**USD cost** computed from Langfuse's price table.**Trace metadata** shows`user_id: tutorial-reader`\n\nand`session_id: demo-session-1`\n\n; under**Tracing → Sessions**,`demo-session-1`\n\ngroups this run (re-run the script and both traces appear in the session).\n\nIf all three hold, every hop of the pipeline is now queryable — you can filter traces by session, sort by cost, and drill into any slow or expensive generation.\n\n## Troubleshooting\n\n— nine times out of ten the keys are fine and`auth_check()`\n\nfails (or the script exits with \"Langfuse rejected credentials\")`LANGFUSE_BASE_URL`\n\npoints at the wrong region: US-region keys against`https://cloud.langfuse.com`\n\n(the EU default) return 401. Match the URL to where you created the project. Set`LANGFUSE_DEBUG=True`\n\nto see the failing request.**Script succeeds but no trace appears in the UI**— the process exited before the background exporter drained its buffer. Ensure`langfuse.flush()`\n\n(or`langfuse.shutdown()`\n\n) runs at the end of the script, including on exception paths; in serverless handlers, flush inside the handler before returning.— the Anthropic key is missing, mistyped, or was revoked. Re-export`anthropic.AuthenticationError: Error code: 401 ... 'invalid x-api-key'`\n\n`ANTHROPIC_API_KEY`\n\n; if you use a key manager, confirm the venv shell actually inherits it (`echo $ANTHROPIC_API_KEY`\n\n).**Generations show token counts but the cost column is blank**— the model string on the span didn't match any Langfuse model definition (common with brand-new or fine-tuned models). Add one under**Project Settings → Models** with a regex match pattern and per-token prices; custom definitions override built-ins and apply to new traces immediately.\n\n## Next steps\n\n- Add\n[scores](https://langfuse.com/docs/evaluation/overview)to traces (user feedback, LLM-as-a-judge) so you can correlate quality with cost per agent. - Instrument other layers of the stack — any OTel library (HTTP clients, databases, other LLM SDKs) drops its spans into the same trace; see\n[Langfuse's OpenTelemetry docs](https://langfuse.com/integrations/native/opentelemetry)for the attribute mapping and the OTLP endpoint if you'd rather bring your own collector. - Use\n`langfuse.start_as_current_observation(as_type=\"generation\", ...)`\n\nfor manual control when auto-instrumentation doesn't fit — for example, custom retry loops or providers without an instrumentor. - Wire the same env vars into CI and production; traces are cheap enough to leave on everywhere, and the session/user attributes you propagated make per-tenant cost accounting a saved table view.\n\n## Sources & further reading\n\n-\n[Observability for Anthropic with Langfuse Integration](https://langfuse.com/integrations/model-providers/anthropic)— langfuse.com -\n[Langfuse Python SDK Overview (v4)](https://langfuse.com/docs/observability/sdk/python/overview)— langfuse.com -\n[Langfuse Python SDK Instrumentation](https://langfuse.com/docs/observability/sdk/python/instrumentation)— langfuse.com -\n[Token and Cost Tracking](https://langfuse.com/docs/observability/features/token-and-cost-tracking)— langfuse.com -\n[OpenTelemetry (OTLP) Integration](https://langfuse.com/integrations/native/opentelemetry)— langfuse.com -\n[opentelemetry-instrumentation-anthropic](https://pypi.org/project/opentelemetry-instrumentation-anthropic/)— pypi.org\n\n[Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer\n\nPriya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/trace-multi-agent-llm-pipelines-with-langfuse-and-opentelemetry", "canonical_source": "https://sourcefeed.dev/a/trace-multi-agent-llm-pipelines-with-langfuse-and-opentelemetry", "published_at": "2026-08-16 17:40:21+00:00", "updated_at": "2026-08-16 18:11:08.908612+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "large-language-models", "mlops"], "entities": ["Langfuse", "OpenTelemetry", "Anthropic", "Claude", "OpenLLMetry", "Priya Nair", "AnthropicInstrumentor", "Python"], "alternates": {"html": "https://wpnews.pro/news/trace-multi-agent-llm-pipelines-with-langfuse-and-opentelemetry", "markdown": "https://wpnews.pro/news/trace-multi-agent-llm-pipelines-with-langfuse-and-opentelemetry.md", "text": "https://wpnews.pro/news/trace-multi-agent-llm-pipelines-with-langfuse-and-opentelemetry.txt", "jsonld": "https://wpnews.pro/news/trace-multi-agent-llm-pipelines-with-langfuse-and-opentelemetry.jsonld"}}