Langfuse is an open-source observability and evaluation platform for LLM applications. It handles trace ingestion, prompt versioning, dataset snapshots, and eval workflows in a single stack. For financial agents and regulated environments, this means audit-grade trace storage, reproducible eval datasets, and self-hosted deployment that keeps sensitive data inside your perimeter.
The architecture splits into three layers: ingestion (traces, spans, generations), management (prompts, datasets, scores), and evaluation (playground, LLM-as-judge). The boundary between logging and evaluation is explicit. Traces flow in via SDK or API, prompts are versioned independently, and datasets are frozen snapshots of production traces or synthetic examples.
Langfuse uses a three-tier hierarchy:
Each entity carries metadata (user ID, session ID, tags), timestamps, and optional scores. Generations link to prompt versions, so you can correlate eval results with the exact prompt text and parameters used in production.
Ingestion happens via:
The SDK batches events and sends them asynchronously to avoid blocking agent execution. Traces land in PostgreSQL with JSONB columns for flexible metadata queries.
Prompts are first-class objects with version history. Each version has:
fraud-detection-v3)
When you log a generation, you pass the prompt name and version. Langfuse resolves the template, stores the rendered text, and links the generation to that prompt version. This makes it possible to:
The prompt registry lives in the database, not in code. You fetch the latest version at runtime or pin to a specific label. This decouples prompt iteration from application deployment.
Datasets are collections of input-output pairs or input-only examples. You create them by:
Each dataset is versioned. When you run an eval, you snapshot the dataset state so results are reproducible even if you later add or remove examples.
Evals run in the Langfuse UI or via SDK. You define:
Results are stored as scores linked to the generation. You can compare prompt versions, model providers, or parameter sweeps in a single view.
Langfuse offers cloud-hosted and self-hosted options. For financial or healthcare agents, self-hosting is the default:
The self-hosted stack includes:
You control the network boundary. Traces never leave your VPC. You can encrypt the database at rest, rotate API keys per environment, and audit all access via PostgreSQL logs.
For high-throughput agents, the ingestion path is the bottleneck. Langfuse batches writes and uses upsert logic to handle duplicate trace IDs. If you exceed 10,000 generations per minute, consider:
Langfuse exposes metrics via the web UI and optional Prometheus endpoints:
Common failure modes:
| Failure | Symptom | Mitigation |
|---|---|---|
| SDK batch timeout | Traces dropped silently | Increase flush interval or reduce batch size |
| Database connection pool exhaustion | 500 errors on ingestion | Scale worker replicas or tune pool size |
| Prompt version mismatch | Eval results link to wrong template | Pin prompt versions in code, not labels |
| Dataset drift | Eval scores degrade over time | Snapshot datasets before each eval run |
The SDK retries failed requests with exponential backoff. If the server is unreachable, traces are logged locally and uploaded when connectivity resumes. This works for short outages but can cause memory pressure in long-running agents.
Langfuse integrates natively with LangChain, LlamaIndex, and Autogen. For LangChain:
from langfuse.callback import CallbackHandler
from langchain.chains import LLMChain
handler = CallbackHandler(
public_key="pk-...",
secret_key="sk-...",
trace_name="fraud-detection",
user_id="user-123"
)
chain = LLMChain(llm=llm, prompt=prompt, callbacks=[handler])
result = chain.run(input="Check this transaction")
The callback handler automatically creates a trace, logs the prompt and completion as a generation, and links metadata. You can add custom scores or tags in the same callback.
For custom agents, use the SDK directly:
from langfuse import Langfuse
langfuse = Langfuse()
trace = langfuse.trace(name="agent-session", user_id="user-123")
span = trace.span(name="tool-call", input={"tool": "search"})
generation = trace.generation(
name="reasoning",
prompt="fraud-detection-v3",
model="gpt-4",
input=prompt_text,
output=completion_text
)
span.end(output={"results": search_results})
trace.score(name="accuracy", value=0.95)
This gives you full control over the trace structure and lets you instrument multi-step workflows without framework lock-in.
Langfuse uses project-level API keys and role-based access. Each project is isolated:
For financial agents, you typically run one project per environment (dev, staging, prod) and rotate keys quarterly. The database schema enforces project isolation via foreign keys, so a compromised key cannot leak traces from other projects.
Sensitive data in traces (PII, account numbers) should be redacted before ingestion. Langfuse does not auto-redact. You handle this in the SDK callback or via a proxy layer that strips fields before they reach the database.
Use Langfuse when:
Avoid Langfuse when:
Langfuse is mature infrastructure for teams that treat evals as a first-class concern. The prompt versioning and dataset snapshot model makes it possible to reproduce eval results months later, which matters when regulators ask why your fraud detection agent approved a specific transaction. The self-hosted option and PostgreSQL backend give you full control over data residency and retention policies.