The Problem: The High Cost of Blind AI Requests
When you build a standard CRUD application, performance bottlenecks are predictable: it's almost always a missing database index or a heavy payload. With AI agents and LLMs, performance is wildly unpredictable.
A single user prompt can trigger multiple API calls, prompt formatting, vector database lookups (RAG), and streaming responses. If a user complains that the app is slow, you can't just check your server CPU usage. You need to know:
Which specific LLM call took the longest?
Did our vector database search slow down the context retrieval?
How many tokens did that request consume (so we don't go broke)?
To answer this, I instrumented a Python-based AI agent backend using OpenTelemetry and hooked it up to SigNoz. Here is exactly how to do it.
Show Your Work: The Setup & Instrumentation
Instead of relying on generic auto-instrumentation that only tells you an HTTP request happened, we are going to write explicit, semantic inline tracing. This gives us deep, domain-specific insights like token counts and model names right inside our telemetry.
Python
`
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
def init_tracer(service_name="ai-agent-service"):
resource = Resource.create(attributes={"service.name": service_name})
provider = TracerProvider(resource=resource)
otlp_exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
processor = BatchSpanProcessor(otlp_exporter)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
return trace.get_tracer(service_name)
tracer = init_tracer()
Python
import time
from tracing import tracer
from opentelemetry.trace import StatusCode
def fake_vector_db_search(query):
with tracer.start_as_current_span("vector_db_search") as span:
span.set_attribute("db.system", "chromadb")
span.set_attribute("db.query", query)
time.sleep(0.4)
return "Context: OpenTelemetry is an open-source observability framework."
def call_llm_agent(user_prompt):
with tracer.start_as_current_span("ai_agent_pipeline") as parent_span:
parent_span.set_attribute("user.prompt_length", len(user_prompt))
context = fake_vector_db_search(user_prompt)
with tracer.start_as_current_span("llm_generation") as llm_span:
llm_span.set_attribute("llm.model_name", "gpt-4o")
llm_span.set_attribute("llm.temperature", 0.7)
try:
start_time = time.time()
time.sleep(1.8)
generation_time = time.time() - start_time
prompt_tokens = 120
completion_tokens = 250
llm_span.set_attribute("llm.prompt_tokens", prompt_tokens)
llm_span.set_attribute("llm.completion_tokens", completion_tokens)
llm_span.set_attribute("llm.total_tokens", prompt_tokens + completion_tokens)
llm_span.set_attribute("llm.latency_seconds", generation_time)
llm_span.set_status(StatusCode.OK)
return "AI Response: Steps completed successfully."
except Exception as e:
llm_span.record_exception(e)
llm_span.set_status(StatusCode.ERROR, description=str(e))
raise e
if name == "main":
print("Running instrumented AI Agent...")
call_llm_agent("How do I properly monitor my pipeline?")
`
Tracking the Output in SigNoz
Once you spin up SigNoz via Docker and run the script above, the exact execution flow transfers from your terminal into a visual powerhouse.
Analyzing the Flame Graph
When you open the Traces tab in SigNoz and click into ai_agent_pipeline, you instantly see the breakdown of the 2.2-second request:
You see the total ai_agent_pipeline bar spans across the entire duration.
Directly underneath it, you see vector_db_search taking exactly 400ms.
Immediately following that, llm_generation takes up 1.8 seconds of the timeline.
If a request spikes to 5 seconds, you no longer have to guess. The flame graph visually exposes exactly which child span shifted right.
Creating a Custom Token & Cost Dashboard
Because we explicitly attached attributes like llm.total_tokens and llm.model_name directly to our spans, we don't just get charts for network speeds—we can build business-critical dashboards.
In SigNoz, you can build a custom panel using ClickHouse queries on your span attributes:
SQL
SELECT
sum(attributes_number_value[indexOf(attributes_string_key, 'llm.total_tokens')]) as total_tokens_consumed,
attributes_string_value[indexOf(attributes_string_key, 'llm.model_name')] as model
FROM signoz_traces.spans
WHERE attributes_string_key = 'llm.model_name'
GROUP BY model
This transforms raw engineering trace telemetry into an executive-level dashboard showing real-time token spend per AI model.
Key Gotchas & What I Learned
Writing this implementation exposed a few critical engineering details that the generic setup guides completely skip over:
📌 Don't Over-Trace Async Streams: If you are streaming responses token-by-token (using stream=True in OpenAI/LangChain), wrapping the entire loop will artifically inflate your generation latency metrics because it includes the client's network read speed. Instead, explicitly measure the time from the initial request to the first byte received to calculate accurate Time-To-First-Token (TTFT).
📌 Mind the Batch Processor: Always use BatchSpanProcessor in staging and production environments. Using a synchronous SimpleSpanProcessor will force your backend application to wait for the SigNoz collector to acknowledge receipt of data before responding to your user, completely destroying application performance.
Conclusion
Observability shouldn't stop at checking if your server container is healthy. By mapping OpenTelemetry custom spans directly to our AI pipelines, we can isolate external vendor API slowdowns, track precise token resource consumption, and debug complex agent architectures with zero guesswork.
To learn more about OpenTelemetry semantic conventions for AI architectures, check out the Official OpenTelemetry Documentation.
To deploy your own observability stack locally, clone the SigNoz GitHub Repository.