I've been building agentic AI systems for a while now, and the thing that finally got under my skin enough to write about is that our whole stack is really good at telling us what an agent did, and almost useless at telling us whether it was right.
Observability tools give you the trace, every tool call and every token, which is great for figuring out what happened after something breaks. Evals give you a score against a test set you ran at some point in the past. But in production, in the moment, when your agent returns a confident, well-formed, schema-valid 200, nothing in that pipeline is checking whether the answer inside it is actually correct. A 200 can wrap a confidently wrong answer and your dashboard will still light up green.
I ran a little experiment to see how bad this actually is. I took a cheap, weak model and pointed it at a real structured task where I could check the answers, and it was right about 69% of the time while looking right a good deal more often than that. Then I wrapped each output in a grounded check that asked whether it genuinely satisfied the constraints instead of just looking like it did, and I re-rolled the ones that failed. It climbed to 100%. The part I keep chewing on is that the model never got any smarter, the verification did all the work.
So the thing I keep coming back to is that consistency isn't correctness. A schema-valid, fluent, nicely-logged answer can still be flat wrong, and almost nothing in the modern agent stack is built to notice that while it's happening.
I've been poking at what a runtime certification layer would look like, something that lives between it logged a 200 and it passed our offline evals, and answers the one question nobody seems to be asking, which is whether this specific output, right now, is actually right.
If you're running agents in production I'm genuinely curious how you're handling this, or whether you've mostly just made peace with the green dashboard.