# Trace Multi-Agent LLM Pipelines with Langfuse and OpenTelemetry > Source: > Published: 2026-08-16 17:40:21+00:00 # Trace Multi-Agent LLM Pipelines with Langfuse and OpenTelemetry Instrument a two-agent Claude pipeline so every LLM call, tool hop, and token cost lands in one trace. [Priya Nair](https://sourcefeed.dev/u/priya_nair) ## What you'll build A two-agent LLM pipeline (a research agent that calls a tool, then a writer agent) fully instrumented with [Langfuse](https://langfuse.com) over [OpenTelemetry](https://opentelemetry.io), so every LLM call, tool invocation, token count, and dollar cost shows up as a nested span in one queryable trace timeline. ## Prerequisites **Python 3.10+**(both`langfuse` and the instrumentor require ≥3.10). Verified with Python 3.12.**Package versions verified for this tutorial:**`langfuse` 4.14.4 (the v4 SDK, rewritten on OpenTelemetry — note the env var is now`LANGFUSE_BASE_URL` , not v3's`LANGFUSE_HOST` ),`opentelemetry-instrumentation-anthropic` 0.62.3 (from[OpenLLMetry](https://github.com/traceloop/openllmetry)), and a recentSDK (0.116+).`anthropic` - A **Langfuse account**— the[Cloud free tier](https://cloud.langfuse.com)works; note whether you signed up in the EU or US region. Self-hosted works identically, just point`LANGFUSE_BASE_URL` at your instance. - An **Anthropic API key** from the[Claude Console](https://platform.claude.com). Any OTel-instrumented provider works the same way; this tutorial uses Claude. - OS: anything with a shell. Commands below are macOS/Linux; on Windows use `set` instead of`export` . ## 1. Create a Langfuse project and grab keys Sign in at [cloud.langfuse.com](https://cloud.langfuse.com) (or `us.cloud.langfuse.com` for the US region), create an organization and a project, then go to **Project Settings → API Keys → Create new API keys**. You get a public key (`pk-lf-...` ) and a secret key (`sk-lf-...` ). The secret is shown once — copy both now. ## 2. Install dependencies and set environment variables ``` python -m venv .venv && source .venv/bin/activate pip install "langfuse>=4.14" "opentelemetry-instrumentation-anthropic>=0.62" anthropic export LANGFUSE_PUBLIC_KEY="pk-lf-..." export LANGFUSE_SECRET_KEY="sk-lf-..." export LANGFUSE_BASE_URL="https://cloud.langfuse.com" # or https://us.cloud.langfuse.com export ANTHROPIC_API_KEY="sk-ant-..." ``` `LANGFUSE_BASE_URL` must match the region you signed up in — this is the single most common source of silent trace loss. ## 3. Understand the two instrumentation layers You're wiring together two things, and it helps to know who does what: (OpenLLMetry) monkey-patches the Anthropic SDK and emits a standard OTel span for every API call, carrying`AnthropicInstrumentor` `gen_ai.*` semantic-convention attributes: model, prompt, completion, and token usage.**The Langfuse v4 SDK** is itself an OTel tracer provider. Any span emitted by any OTel instrumentation library while Langfuse is initialized lands inside your Langfuse trace tree automatically — no exporter config, no OTLP endpoint wrangling. Langfuse maps`gen_ai.usage.*` to token counts and multiplies them against its built-in model price list (updated daily against provider docs) to compute cost per generation. Your own agent and tool functions become spans via Langfuse's `@observe` decorator, and the auto-instrumented LLM spans nest under whichever `@observe` function made the call. ## 4. Build the instrumented pipeline Save this as `pipeline.py` . It's the complete, runnable file: ``` python import os from anthropic import Anthropic from langfuse import get_client, observe, propagate_attributes from opentelemetry.instrumentation.anthropic import AnthropicInstrumentor # 1. Patch the Anthropic SDK BEFORE making any calls. AnthropicInstrumentor().instrument() # 2. Initialize Langfuse (reads LANGFUSE_* env vars) and fail fast on bad creds. langfuse = get_client() if not langfuse.auth_check(): raise SystemExit("Langfuse rejected credentials — check keys and LANGFUSE_BASE_URL region") client = Anthropic() MODEL = "claude-opus-5" def ask(system: str, prompt: str) -> str: # Opus 5 thinks by default and max_tokens caps thinking + answer together, # so leave generous headroom. response = client.messages.create( model=MODEL, max_tokens=16000, system=system, messages=[{"role": "user", "content": prompt}], ) return "".join(b.text for b in response.content if b.type == "text") @observe() # tool invocation -> its own span, input/output captured def fetch_release_notes(project: str) -> str: # Stub tool: swap in a real HTTP call or DB query. return ( f"{project} changelog: v4 SDK is OTel-native; ingestion adds " "x-langfuse-ingestion-version=4; cost table now audited daily." ) @observe(name="research-agent") def research(topic: str) -> str: notes = fetch_release_notes(topic) return ask( "You are a research agent. Extract the three most important facts as bullets.", f"Source material:\n{notes}", ) @observe(name="writer-agent") def write_summary(facts: str) -> str: return ask( "You are a writing agent. Turn these facts into a two-sentence executive summary.", facts, ) @observe(name="research-pipeline") def run_pipeline(topic: str) -> str: return write_summary(research(topic)) if __name__ == "__main__": # Attributes set here propagate to every span in the trace, # so you can filter/group traces by session or user in the UI. with propagate_attributes(session_id="demo-session-1", user_id="tutorial-reader"): print(run_pipeline("Langfuse")) langfuse.flush() # spans are buffered; short scripts must flush before exit ``` Two details matter more than they look. `AnthropicInstrumentor().instrument()` runs before anything else so every SDK call is patched. And `langfuse.flush()` runs last — the SDK exports spans on a background thread, and a script that exits without flushing loses the trace. Privacy note: the instrumentor records prompts and completions into span attributes by default. Set `TRACELOOP_TRACE_CONTENT=false` to keep payloads out of your traces. ## 5. Run it ``` python pipeline.py ``` The script prints a two-sentence summary, e.g.: ``` Langfuse's v4 SDK is now built natively on OpenTelemetry, with ingestion tagged via x-langfuse-ingestion-version=4. Its model cost table is audited daily, keeping per-generation pricing accurate. ``` ## Verify it works Open your project in the Langfuse UI and click **Tracing → Traces**. You should see a trace named `research-pipeline` within a few seconds. Click it and check: **The timeline nests correctly:**`research-pipeline` →`research-agent` → (`fetch_release_notes` span + one`anthropic.chat` generation), then`writer-agent` → a second generation. Two generations total.**Each generation shows**`claude-opus-5` as the model, the full prompt and completion, input/output token counts, latency, and a**USD cost** computed from Langfuse's price table.**Trace metadata** shows`user_id: tutorial-reader` and`session_id: demo-session-1` ; under**Tracing → Sessions**,`demo-session-1` groups this run (re-run the script and both traces appear in the session). If all three hold, every hop of the pipeline is now queryable — you can filter traces by session, sort by cost, and drill into any slow or expensive generation. ## Troubleshooting — nine times out of ten the keys are fine and`auth_check()` fails (or the script exits with "Langfuse rejected credentials")`LANGFUSE_BASE_URL` points at the wrong region: US-region keys against`https://cloud.langfuse.com` (the EU default) return 401. Match the URL to where you created the project. Set`LANGFUSE_DEBUG=True` to see the failing request.**Script succeeds but no trace appears in the UI**— the process exited before the background exporter drained its buffer. Ensure`langfuse.flush()` (or`langfuse.shutdown()` ) runs at the end of the script, including on exception paths; in serverless handlers, flush inside the handler before returning.— the Anthropic key is missing, mistyped, or was revoked. Re-export`anthropic.AuthenticationError: Error code: 401 ... 'invalid x-api-key'` `ANTHROPIC_API_KEY` ; if you use a key manager, confirm the venv shell actually inherits it (`echo $ANTHROPIC_API_KEY` ).**Generations show token counts but the cost column is blank**— the model string on the span didn't match any Langfuse model definition (common with brand-new or fine-tuned models). Add one under**Project Settings → Models** with a regex match pattern and per-token prices; custom definitions override built-ins and apply to new traces immediately. ## Next steps - Add [scores](https://langfuse.com/docs/evaluation/overview)to traces (user feedback, LLM-as-a-judge) so you can correlate quality with cost per agent. - Instrument other layers of the stack — any OTel library (HTTP clients, databases, other LLM SDKs) drops its spans into the same trace; see [Langfuse's OpenTelemetry docs](https://langfuse.com/integrations/native/opentelemetry)for the attribute mapping and the OTLP endpoint if you'd rather bring your own collector. - Use `langfuse.start_as_current_observation(as_type="generation", ...)` for manual control when auto-instrumentation doesn't fit — for example, custom retry loops or providers without an instrumentor. - Wire the same env vars into CI and production; traces are cheap enough to leave on everywhere, and the session/user attributes you propagated make per-tenant cost accounting a saved table view. ## Sources & further reading - [Observability for Anthropic with Langfuse Integration](https://langfuse.com/integrations/model-providers/anthropic)— langfuse.com - [Langfuse Python SDK Overview (v4)](https://langfuse.com/docs/observability/sdk/python/overview)— langfuse.com - [Langfuse Python SDK Instrumentation](https://langfuse.com/docs/observability/sdk/python/instrumentation)— langfuse.com - [Token and Cost Tracking](https://langfuse.com/docs/observability/features/token-and-cost-tracking)— langfuse.com - [OpenTelemetry (OTLP) Integration](https://langfuse.com/integrations/native/opentelemetry)— langfuse.com - [opentelemetry-instrumentation-anthropic](https://pypi.org/project/opentelemetry-instrumentation-anthropic/)— pypi.org [Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to. ## Discussion 0 No comments yet Be the first to weigh in.