What is DSPy Observability?
DSPy observability gives you real-time visibility into your DSPy programs by collecting traces using OpenTelemetry. The OpenInference DSPy instrumentation automatically captures every DSPy module call (Predict
, ChainOfThought
, ReAct
, custom Module
pipelines) and the underlying language model request as spans, tagged with DSPy-native attributes such as openinference.span.kind
(CHAIN, LLM, TOOL), llm.model_name
, and llm.provider
.
With full DSPy monitoring in SigNoz, you can trace every program run end to end, break down module and tool activity, compare model usage and latency, set alerts on latency and errors, and continuously improve the reliability of your DSPy applications.
Prerequisites
- A SigNoz Cloud accountwith an active ingestion key orSelf Hosted SigNoz instance - Python 3.9 or later
- A DSPy program and a configured language model provider (for example, an Anthropic or OpenAI API key)
Monitoring DSPy with OpenTelemetry
DSPy does not ship with OpenTelemetry built in, so you add the OpenInference DSPy instrumentation and export traces to SigNoz over OTLP/HTTP. You can wire this up two ways. For more details, refer to the DSPy documentation.
No-code auto-instrumentation is recommended for quick setup with minimal code changes. It is ideal when you want observability up and running without modifying your application code, using the OpenInference DSPy instrumentor loaded automatically at startup.
Step 1: Install the necessary packages in your Python environment.
pip install \
dspy \
opentelemetry-distro \
opentelemetry-exporter-otlp \
openinference-instrumentation-dspy
Step 2: Add automatic instrumentation.
opentelemetry-bootstrap --action=install
Step 3: Run an example.
import dspy
dspy.configure(lm=dspy.LM("anthropic/claude-sonnet-5"))
predict = dspy.Predict("question -> answer")
result = predict(question="What is OpenTelemetry in one sentence?")
print(result.answer)
Step 4: Run your application with auto-instrumentation.
OTEL_RESOURCE_ATTRIBUTES="service.name=<service_name>" \
OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.<region>.signoz.cloud:443" \
OTEL_EXPORTER_OTLP_HEADERS="signoz-ingestion-key=<your-ingestion-key>" \
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
OTEL_TRACES_EXPORTER=otlp \
opentelemetry-instrument <your_run_command>
Verify these values:
<service_name>
: The name your DSPy service appears under in SigNoz.<region>
: YourSigNoz Cloud region.<your-ingestion-key>
: Your SigNozingestion key.<your_run_command>
: The command you use to run your application, for examplepython main.py
.
Code-based instrumentation gives you fine-grained control over your telemetry configuration. Use this approach when you need to customize resource attributes, sampling strategies, or integrate with existing observability infrastructure.
Step 1: Install the necessary packages in your Python environment.
pip install \
dspy \
openinference-instrumentation-dspy \
opentelemetry-sdk \
opentelemetry-exporter-otlp-proto-http
Step 2: Configure the OpenTelemetry exporter.
Create a telemetry.py
module that sets up the tracer provider, wires the OTLP/HTTP exporter to SigNoz, and turns on the DSPy instrumentation:
from openinference.instrumentation.dspy import DSPyInstrumentor
from opentelemetry import trace as trace_api
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk import trace as trace_sdk
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace.export import BatchSpanProcessor
def setup_tracing() -> None:
resource = Resource.create({"service.name": "<service_name>"})
tracer_provider = trace_sdk.TracerProvider(resource=resource)
exporter = OTLPSpanExporter(
endpoint="https://ingest.<region>.signoz.cloud:443/v1/traces",
headers={"signoz-ingestion-key": "<your-ingestion-key>"},
)
tracer_provider.add_span_processor(BatchSpanProcessor(exporter))
trace_api.set_tracer_provider(tracer_provider)
DSPyInstrumentor().instrument(tracer_provider=tracer_provider)
Verify these values:
<service_name>
: The name your DSPy service appears under in SigNoz.<region>
: YourSigNoz Cloud region.<your-ingestion-key>
: Your SigNozingestion key.
Step 3: Instrument your DSPy program.
Call setup_tracing()
once, before you configure DSPy and run your modules. Every DSPy module call and each underlying model request is then captured automatically:
import dspy
from telemetry import setup_tracing
setup_tracing()
dspy.configure(lm=dspy.LM("anthropic/claude-sonnet-5"))
predict = dspy.Predict("question -> answer")
result = predict(question="What is OpenTelemetry in one sentence?")
print(result.answer)
Each run emits a trace whose spans carry openinference.span.kind
(CHAIN, LLM, TOOL) along with llm.model_name
and llm.provider
on the model spans. OpenTelemetry batches spans before sending, so allow a few seconds after a run for data to appear in SigNoz.
View DSPy Traces in SigNoz
Once configured, your DSPy program automatically emits traces for every run.
Traces are available in SigNoz under the Traces tab:
When you click on a trace in SigNoz, you'll see a detailed view of the trace, including all associated spans, along with their events and attributes. CHAIN spans represent DSPy module and pipeline steps, LLM spans carry llm.model_name
and llm.provider
, and TOOL spans capture ReAct tool calls.
DSPy Observability Dashboard
You can also check out our custom DSPy dashboard which provides specialized visualizations for monitoring your DSPy programs. The dashboard includes pre-built charts for module, chain, and tool activity, per-model usage, latency percentiles, and errors, along with import instructions to get started quickly.
Troubleshooting DSPy Observability
Troubleshooting DSPy Observability
No traces in SigNoz
- Confirm your program actually ran a DSPy module. Setting up tracing alone emits nothing.
- Verify
setup_tracing()
is called beforedspy.configure(...)
and before any module runs. - Check that the region in the OTLP endpoint matches your SigNoz account.
- OpenTelemetry batches data before sending, so wait 10-30 seconds after a run.
Verify the instrumentor
Make sure DSPyInstrumentor().instrument(...)
runs against the same tracer provider you registered with trace_api.set_tracer_provider(...)
. If you build multiple providers, spans can be dropped silently.
Auth errors (401 / 403)
Re-check the ingestion key in the signoz-ingestion-key
header. It must be the exact key from your SigNoz Ingestion Settings, with no extra spaces or quotes.
Endpoint issues
Keep the https://
scheme and :443
port on the endpoint, and the /v1/traces
path for OTLP/HTTP. Do not send OTLP/HTTP traffic to the gRPC port.
Setup OpenTelemetry Collector (Optional)
Setup OpenTelemetry Collector (Optional)
What is the OpenTelemetry Collector?
Think of the OTel Collector as a middleman between your app and SigNoz. Instead of your application sending data directly to SigNoz, it sends everything to the Collector first, which then forwards it along.
Why use it?
Cleaning up data- Filter out noisy traces you don't care about, or remove sensitive info before it leaves your servers.** Keeping your app lightweight**- Let the Collector handle batching, retries, and compression instead of your application code.** Adding context automatically**- The Collector can tag your data with useful info like which Kubernetes pod or cloud region it came from.** Future flexibility**- Want to send data to multiple backends later? The Collector makes that easy without changing your app.
See Switch from direct export to Collector for step-by-step instructions to convert your setup.
For more details, see Why use the OpenTelemetry Collector? and the Collector configuration guide.
Additional resources:
- Set up alertsfor high latency or error rates - Learn more about querying traces - Explore log correlation