Member-only story
The 2 a.m. page that tracing can’t explain #
A support bot answers a billing question with total confidence and gets the refund policy wrong. Nobody notices for three days because the response looked fine: grammatically correct, on-topic, politely worded. It just wasn’t true. When someone finally flags it, an engineer pulls up the logs and finds one line: a request ID, a 200 status code, a latency number. The actual conversation is gone. The retrieved documents, the system prompt, the tool calls, the reasoning steps, whatever confidence the model had in its own answer. None of it was ever captured.
That’s the moment most teams realize standard application logging was never built for this problem. A REST API either returns the right JSON or it doesn’t, and you can write an assertion for that. An LLM call can be syntactically perfect, fast, cheap, and still wrong in a way no HTTP status code will ever surface. You need visibility that treats “was this output actually good” as a first-class signal alongside latency and error rate. That’s what LLM observability is, and it’s why MLflow, Langfuse, and Confident AI all exist.
These three get lumped together constantly. All three trace LLM calls, all three touch evaluation, all three show up on the same…