Your LLM passed the vibe check. It is in production. Nothing is throwing exceptions, and users are still clicking. Silent failures can develop while everything appears healthy. Charity Majors argues that the source of truth is what the system is doing right now. Malte Ubl’s path from quick validation to shipped agents adds the other half: once business logic lives in prompts and tool loops, you need traces before you tune prompts or models.
If you only watch HTTP 500s, you may learn about model drift from support tickets. Drift is performance getting worse while the weights file stays the same: inputs shift, behavior shifts, errors do not. Pair that with instruction adherence, because a fluent answer that ignores your format still breaks downstream code. Alerts should treat obedience and quality as separate signals.
For a fuller map of how observability, evaluation, and infrastructure fit together, the Topics index on Chain of Thought is a useful anchor. This article is narrower: what to page on, and how each alert links back to one trace.
Pre-launch checks answer “does this work on yesterday’s examples?” Production answers “what is happening on this request, with this user, with this tool output?” Majors describes teams that debug by finding whoever last changed the code and asking the resident expert. Generated or agent-written paths break that habit: there may be no expert, and the failure is nondeterministic.
Like, the source of truth is what is happening right now.
Charity Majors, Honeycomb, on Chain of Thought ep 18
Malte Ubl frames the same shift for agents: core logic is partly prose, execution paths vary, so logging alone is not enough. You alert on aggregates, but you investigate on traces.
Decision rule: if the feature runs more than once per user session, or calls tools, default to trace-backed alerts. If it is a single-shot completion with a strict schema, lean on validators plus sampled trace review.
Drift alerts watch live distributions and outcomes rather than one offline score. The glossary's drift definition says to watch live inputs and outputs. These are the triggers I would propose:
Instruction adherence alerts are mechanical where you can be mechanical. Schema validation failures, missing required JSON fields, over-length responses, and regex checks on forbidden patterns should increment a counter and page when the rate crosses a threshold you set from staging traffic. For fuzzy rules (“formal tone”, “cite the doc”), use an LLM-as-judge on a sample, but page on validator failures first because those are cheap and unambiguous.
Do not merge these into one “quality score” alert. A model can stay on-distribution while ignoring instructions, or follow format while becoming less helpful.
Ubl points out that extra LLM calls add latency and cost. I would make p95 latency and dollars per successful task first-class SLOs for agents and multi-step chains. Alert when:
Tie both metrics to the same trace id you use for quality. My working hypotheses: a cost spike with flat quality may point to a prompt change or a model swap, and a latency spike with rising validator failures may mean the model is rambling before it breaks schema.
An alert without a trace is a mood. Minimum fields per LLM span (sketch, not a vendor schema):
def on_alert(alert_name: str, trace_id: str, store):
trace = store.get_trace(trace_id) # turns, tool I/O, model id, tokens
if alert_name == "schema_fail_rate":
return [s for s in trace.spans if s.get("validator") == "json_schema"]
if alert_name == "cost_baseline":
return trace.summary(tokens_in=..., tokens_out=..., model=...)
Workflow when a page fires:
Ubl’s production path assumes this loop exists before optimization:
you need tracing now. Because the trace tells you what the thing did.
Malte Ubl, Vercel, on Chain of Thought ep 39
He also notes traces are not identical run to run, which is why the alert should carry a bad example along with the metric delta.
Start small and explicit:
| Alert | Signal | First action |
|---|---|---|
instruction_break |
Validator failure rate | Inspect trace; check schema and tool descriptions |
output_shift |
Judge or human negative rate vs baseline | Sample inputs on failing traces for distribution shift |
latency_slo |
p95 trace duration | Count tool turns; look for retry loops |
cost_slo |
Tokens per successful trace | Compare model and prompt version on exemplar |
Run a weekly review: one drift-oriented alert, one adherence alert, one latency or cost alert. If you cannot name the trace id you would open for each, the alert is not ready.
More on this topic, with the related episodes, is on Chain of Thought. It draws on this episode.
Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.
Drafted with AI assistance from the episode transcripts.