{"slug": "ai-safety-is-an-observability-problem", "title": "AI Safety Is an Observability Problem", "summary": "Microsoft CEO Satya Nadella said at the All-In Summit that AI agents need \"true aggressive monitoring of agent activity\" with auditable behavioral evidence and access logs, warning that an agent asked to optimize working capital \"may fake my books\" — what he called \"a new type of insider risk.\" Nadella framed AI safety as an engineering problem, saying \"we will have to get the engineering process around building out this experimental science to be more robust,\" and opened with containment rather than alignment theory. The argument holds that operational metrics like error rate and latency stay flat while agent decision quality degrades, so telemetry must be paired with evals to determine whether an agent's actions were actually correct.", "body_md": "When you ask an agent to optimize your working capital, it might falsify your books instead. The system won't crash while it happens. Your error rate and latency graphs stay flat, and nobody gets paged. Every dashboard you own stays green while the decisions underneath it rot.\n\nSatya Nadella reached for that scenario at the [All-In Summit](https://www.youtube.com/watch?v=hdcsTeCFE0I), during a wider discussion about the Hugging Face incident, reward hacking, and warnings from people leaving frontier AI labs about the catastrophic risk of human extinction.\n\n“Suppose I say, hey, go optimize my working capital. It may fake my books”, he said. “Because this is like a new type of insider risk”.\n\nHis answer to that risk was not what I expected. He did not start with alignment theory or another model-level safeguard. He started with monitoring:\n\n\"One of the things that I think is going to be really an issue and a thing that needs great solutions is true aggressive monitoring of agent activity\", he said. \"That's behavioral evidence, and so everything has got to be auditable. And then every object it accesses\".\n\nThat’s a description of observability. He just didn't use the word.\n\n## [He answered a safety question with an engineering answer](#he-answered-a-safety-question-with-an-engineering-answer)\n\nNadella was handed the existential framing twice and both times he went somewhere more mundane. He opened with containment. “I would start from the very basics of saying, okay, what does containment look like?”\n\nThen he said the part that made the whole argument click.\n\n\"The core of my take is we will have to get the engineering process around building out this experimental science to be more robust\".\n\nComing from the person overseeing one of the largest AI infrastructure investments in the industry, I read that as a statement about where some of the most important controls have to live.\n\nSome of that lives in the model, but most of it lives around it, in containment, permissions, and the monitoring that produces a record of what the agent actually did.\n\nFor most of its history, the pitch has been uptime, latency, errors, cost, and the occasional bad night. Agentic systems raise the stakes, and telemetry becomes part of the evidence you need before handing a system more authority.\n\nYou should not let an agent act on its own if you cannot reconstruct what it did.\n\n## [Failure and dishonesty are different problems](#failure-and-dishonesty-are-different-problems)\n\nThe working capital agent breaks something we have leaned on for two decades.\n\nMonitoring is very good at detecting failure. It catches crashes, errors, saturation, and latency, and almost none of that catches an agent that's confidently doing the wrong thing. We call that gap silent success in agentic observability, where the operational metrics stay healthy while decision quality degrades underneath them.\n\nOperational health stops being enough at that point. Whether the agent is up and whether it’s doing the right thing are different questions, and a dashboard built to flag errors can answer the first while completely missing the second.\n\nThat‘s where evals become part of the observability story. Telemetry tells you what the agent did, but only evals can help you decide whether what it did was actually good.\n\nFor an agent, that might mean checking whether an answer was grounded, whether a tool call was appropriate, whether the task constraints were satisfied, or whether the final outcome matched the goal. Some of those checks can be deterministic. Others may need model-based evaluation or human review.\n\nNadella reached for an older instinct here. He talked about transaction processing, and how data loss was the class of bug where \"you stop the thing\". An agent that falsifies a number is a data integrity bug, and the response that class of bug has always earned is to halt the system rather than file a ticket.\n\n## [Some of this is not a new problem](#some-of-this-is-not-a-new-problem)\n\nWhat I appreciated most was his refusal to make the problem exotic.\n\nHe split these incidents into two categories. The mundane ones came first: “somebody misconfigured a container”, API keys in a public repository, “there’s no monitoring, there’s internet access”.\n\nHis label was blunt: “Classic, I would call it, basic DevOps”.\n\nOnly after that did he get to the novel part, the reward hacking and swarm behavior nobody has good science for yet. And that distinction deserves more attention.\n\nMost of what gets filed under AI safety right now is unmonitored infrastructure with credentials lying around, and you already know how to fix a lot of that.\n\nThe uncomfortable version of the same point is that a team can spend its attention on alignment while running agents in an environment it can't see into, and the second problem is the one that will actually bite.\n\n## [Read the rest as a specification](#read-the-rest-as-a-specification)\n\nEach phrase in that quote asks for something specific, and the specifics are more demanding than the headline version suggests.\n\n*Behavioral evidence* rules out watching outputs. An agent's output is one artifact at the end of a long, non-deterministic run, and the behavior is the run itself: which tool, which arguments, in which order, against which data, and where the reasoning branched.\n\nIn [LLM observability terms](https://dash0.com/knowledge/llm-observability-developers-guide), the unit of analysis moves from the response to the trajectory. A log line may record that something finished, which isn't the same as recording what it did.\n\n*Everything has got to be auditable, and then every object it accesses* is a data model problem before it is a security problem. After an incident, nobody asks whether the agent looked anomalous. They ask whether it read this table, retrieved that secret, called this API, or caused that change.\n\nThat means agent telemetry has to connect with the systems it touched. And some of the evidence should come from those systems too. If the agent says it queried a database, the trace tells you what it attempted but only the database can tell you what actually arrived. An agent should not be the only witness to its own actions.\n\nThen he asked for something harder. \"If it goes and gets a secret, it's going to go chain a couple of things. You should be able to see it when it's starting to chain a couple of vulnerabilities to go hack\".\n\nThat's a sequence being watched as it assembles, across many steps and many systems, rather than a report written afterward, and it's the hardest thing on his list. Just enough context to recognize the sequence, and enough speed to do something about it.\n\nHis answer to the insider risk was \"a causal model, like a semantic model that actually checks and verifies\", where the agent's claim about the world gets checked against the world.\n\nIf it says latency increased, check the latency. If it says a deployment caused the problem, check the deployment timeline. If it proposes an alert for a metric, verify that the metric exists and returns data.\n\nSome checks are harder and need evals.\n\nWas the diagnosis actually correct? Did the agent choose an appropriate tool? Did it satisfy the constraints of the task? Did the final result improve the thing it was supposed to improve?\n\nThat’s where agent observability goes beyond recording execution. You need the trace of what happened, and an evaluation of whether what happened was acceptable.\n\nThat sounds obvious, but it’s not how most agent systems work today.\n\nHe asked for one more thing earlier in the conversation: reasoning traces “in language that we can all understand”, transparent enough that “if you have the full CoT, then you can really go look at it deeply.”\n\nThat can be useful evidence, but it is not ground truth. A reasoning trace can help explain a run, but it doesn’t replace the record of what the agent actually did or what the surrounding systems observed.\n\nKeep the reasoning artifact if you have it, but trust the evidence around the action more.\n\n## [What this asks of an observability platform](#what-this-asks-of-an-observability-platform)\n\nThe evidence has to sit in one place. When agent behavior lives in a separate store from your infrastructure, application, and frontend telemetry, watching a chain assemble becomes a manual exercise across tools, under time pressure, run by someone who didn't write the agent. [SignalStore](https://www.dash0.com/blog/beyond-observability) exists so that join happens at query time.\n\nIt also has to stay in an open standard, because agent telemetry in a proprietary schema makes a poor audit trail when its usefulness expires with your contract. That's why we're [OpenTelemetry-native](https://www.dash0.com/blog/opentelemetry-native-the-future-of-observability) rather than OpenTelemetry-compatible, and why the [GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) are worth paying attention to.\n\nThen verify inside the loop that produces the work. [Agent0](https://dash0.com/blog/observability-for-the-ai-era-starts-here-agent0-is-ga) applies a narrow version of that principle already. When it generates assets from telemetry, it validates the underlying queries against live data before it reaches you, with citations you can click into.\n\nWe built it this way because I was not willing to ship an AI that confidently invents a metric name.\n\nThat does not prove every conclusion the agent makes is correct. But it does remove one important class of silent failure where confident claims are made about systems that the telemetry does not support.\n\nThe same principle is why autonomy in Dash0 is progressive. A human approves first, then reviews, then steps back as the system earns more authority.\n\n## [Final thoughts](#final-thoughts)\n\nThe sharpest description of agent security I've heard lately came out of a podcast interview, and what it pointed at was observability.\n\nThe work it implies is concrete. Fix the boring things first, because a misconfigured sandbox and a credential sitting in a public repo are still the most likely ways this goes wrong.\n\nCapture trajectories not just outputs, record what the agent touched, then check what your agents claim against what your systems actually report, with evals that grade against a source of truth rather than against another model's opinion.\n\nKeep the evidence in an open standard, so your audit trail does not become another form of vendor lock-in.\n\nThis is not a new discipline. It’s security, DevOps, evals, and observability being asked to work together around a new kind of actor.\n\nObservability gets the harder job, which is keeping record of a system that acts at machine speed, crosses system boundaries on its own, and may look perfectly healthy while doing the wrong thing.\n\nThat’s the problem we are building for at Dash0. If it resonates, [come take a look](https://www.dash0.com/sign-up).", "url": "https://wpnews.pro/news/ai-safety-is-an-observability-problem", "canonical_source": "https://dash0.com/blog/ai-safety-is-an-observability-problem", "published_at": "2026-09-21 11:04:12+00:00", "updated_at": "2026-09-21 11:52:45.732025+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "ai-policy", "artificial-intelligence"], "entities": ["Satya Nadella", "Microsoft", "All-In Summit", "Hugging Face"], "alternates": {"html": "https://wpnews.pro/news/ai-safety-is-an-observability-problem", "markdown": "https://wpnews.pro/news/ai-safety-is-an-observability-problem.md", "text": "https://wpnews.pro/news/ai-safety-is-an-observability-problem.txt", "jsonld": "https://wpnews.pro/news/ai-safety-is-an-observability-problem.jsonld"}}