Before we start, I’m a noob!
My experience with AI has mostly been things like, “Create an itinerary for a summer trip to Mallorca,” or “Write me a Python function to calculate this or that,” with a little bit of vibe coding on the side.
Well, maybe more than a little.
It’s now much easier to ask a simple question and get a complex answer. With AI, work can often be done faster and more… “efficiently.”
But not always.
I’ve lost count of the frustrations. In trying to get things done faster, we sometimes end up making them messier. Some people say, “You need to prompt the AI better,” or “You didn’t give the AI enough context,” or even, “You gave the AI too much context!”
Or should I say “LLM” instead?
So I started wondering: how does our close friend actually “think” and arrive at an answer?
Recently, I read an interesting framework proposed by Sisodia [5], which identifies five key taxonomies of AI observability. It completely shifted how I see AI, specifically LLMs, behind the scenes.
According to Sisodia [5], each of these layers represents a distinct level of abstraction for observing LLM system behavior, flowing from Layer 1 to Layer 5.
In this post, I’ll do my best to revisit the research and walk through each layer in a way that makes sense for us noobs.
Hang on tight!
The central issue is that Language Models (LMs) can be “unfaithful” to the facts provided in their input context.
For example, if you explicitly tell a model “Greg is a nurse,” it might still turn around and tell a user “Greg is a doctor.” Why? Because of deeply ingrained training biases, malicious prompt injections, or hidden backdoors learned during its training days. Therefore, when a model processes a sentence like “Greg is a nurse. Laura is a physicist,” it can’t just recognize the individual words. It has to physically “bind” them together. It needs to know that Greg goes with nurse, and Laura goes with physicist, not the other way around.
Even if the model outputs a lie, its brain still knows the truth.
This is where the research by Feng et al. [3] gets fascinating. They introduced something called propositional probes. Inside the model’s complex mathematical activations, there is a specific, low-dimensional area called the binding subspace.
Think of the binding subspace like a mental Velcro wall. It’s the dedicated corner of the AI’s brain where it physically slaps related concepts together.
The probe first looks at individual tokens (words) to see if they represent a name, an occupation, a country, etc. Once it identifies those pieces, it checks the binding subspace to see which pieces are connected.
As for the *,*If the math shows “Greg” and “nurse” are bound together, the probe extracts the logical statement (or proposition): WorksAs(Greg, nurse). So when the model’s output is “unfaithful” (e.g., it says “Greg is a physicist”), the propositional probe can look at the model’s internal activations and correctly extract WorksAs(Greg, nurse).
This means, according to Sisodia [5], the a way to use “probes” to look directly into the model’s neural network to extract its true beliefs, even if the model is outputting a lie or hallucination.
Imagine a customer service chatbot gets tricked by a user into selling a brand-new car for $1. Traditional cloud monitoring wouldn’t catch this because nothing crashed and no servers went down.
But Layer 1 observability is like a digital security guard standing over the AI’s shoulder. It catches the lie at the source. It looks at the Velcro wall, spots that the internal state knew the car cost $20,000, and immediately flags the fact that the output text says $1.
Second Layer: Confidence and Calibration
First things first: am I familiar with Reinforcement Learning with Verifiable Rewards?
Of course…
…not!
At least, until I found this paper [5]. Think of RLVR as an old-school training process that uses a strict “pass/fail” grading system. The model is given a prompt (like a tough math problem), spits out a long, rambling “chain of thought,” and finally guesses an answer.
The RLVR system checks if the final answer is factually correct. If the answer is correct, the model receives a reward (a positive signal), otherwise… nothing… and that’s the problem.
The model uses this binary feedback loop to adjust its parameters, basically telling itself, “Hey, whatever random thoughts led to that correct answer, let’s do more of that!”
Damani et al. [2] discovered a problem where models are incentivized to guess, it is not penalized for “guessing” or being overconfident. This incentivizes the model to generate responses even when it is uncertain, leading to hallucinations**.**
Hence, Damani et al. [2] proposed RLCR (Reinforcement Learning with Calibration Rewards) against the RLVR benchmark.
The model is trained to generate both an answer and a numerical confidence estimate. Instead of just a binary “correctness” reward, the training uses a combined reward function:
where:
In short, RLCR forces the model to treat “knowing what it doesn’t know” as an essential part of the reasoning process.
According to Sisodia [5], when models can accurately express how confident they are, they suddenly become a million times easier to manage. Because, when a standard LLM gives us an answer today, we have absolutely no idea if it’s spitting hard facts or completely winging it. If the model is properly calibrated, it hands us a reliable honesty gauge.
For example, an automated monitoring system can look at that gauge and explicitly say: “Whoa, the model is only 30% confident in this medical summary, let’s route this to a human doctor for review immediately.” (Though, let’s be honest, you should probably never skip a real doctor for LLM medical advice anyway!) In this way, the model’s confidence score becomes a useful signal that helps people decide what to do next.
In traditional software, you monitor behavior by looking at log files (e.g., “User clicked button A”). In AI, the model’s final action (“Button A”) does not tell you if the AI is trying to help the user or hack the system.
Modern AI agents often “think” out loud (Chain-of-Thought) before acting. While monitoring these CoTs has become a primary way to detect misbehavior, according to Guan et al. [4], ]it is unclear if this practice is robust.
Why? You asked.
Because we need to know if the AI might learn to hide its true intentions (obfuscation) as it gets smarter.
Guan et al. [4], “Evaluating chain-of-thought (CoT) monitorability” refers to measuring how effectively a model’s step-by-step reasoning process can reveal whether the model is behaving correctly or starting to hallucinate, fail, or act dangerously.
The research yielded several massive takeaways for how we should run AI systems in the real world:
According to Sisodia [5], this would serve as a foundation of deployment strategy, by trading inference compute for safety — instead of paying to run a massive, ultra-fast model that gives immediate (but opaque) answers, we might need to choose to run a slightly smaller model but force it to generate long, detailed reasoning chains.
This costs more compute power per query (inference compute), but the resulting transparency drastically improves the safety and monitorability of the system.
Moving down to Layer 4, we zoom out entirely!
We are no longer looking at the AI’s internal thoughts or its reasoning scratchpad. Instead, we are looking at the entire cloud environment where the AI lives and operates.
Cloud environments are chaotic, dynamic, and constantly changing based on user traffic. An AI that can read a static log file might completely panic in a live environment.
Chen et al. [1] introduces the concept of AgentOps. Rather than just using AI to detect an anomaly (e.g., “CPU is running hot”), AgentOps envisions compound AI systems that manage the entire incident lifecycle.
The research introduces a four-level taxonomy to categorize the complexity of AIOps tasks, progressing from Detection (Level 1) to Localization (Level 2), Root Cause Analysis (Level 3), and finally Mitigation (Level 4) — mirrors the complete Incident Management Lifecycle handled by a human Site Reliability Engineer (SRE).
At this foundational level, the core challenge is simply identifying that something has gone wrong in the system.
Once an incident is detected, the agent must pinpoint exactly where the fire is burning within a complex web of microservices. This requires structural understanding of microservice architectures and the ability to differentiate between a symptom (a downstream service failing) and the source (an upstream service timing out).
Finding where the issue is occurring isn’t enough; the agent must now figure out exactly why it is happening. This requires profound deductive reasoning. The agent must parse messy, highly verbose text files and reason about edge cases in software configuration.
The ultimate frontier of “AgentOps.” The agent is given authorization to actively modify the environment to restore system health. This carries the highest risk. An incorrect action at this stage could cascade an isolated fault into a total, hyper-scale cloud outage.
As we progress from Level 1 to Level 4, the complexity grows exponentially. By testing agents across all four levels, we can identify exactly where an AI agent’s reasoning breaks down. For example, a model might be a perfect “Level 1” watchman but lack the deep architectural logic required to step up to a “Level 4” autonomous cloud healer.
According to Sisodia [5], observability means having a system that actively correlates the crash with a specific server misconfiguration, reads the relevant logs, and provides a human engineer with a plain-English diagnosis of the root cause (Active).
This shifts monitoring from Passive Alerts to Active Investigation. Instead of a dashboard just flashing red and waking up a human engineer at 2:00 AM because an LLM is running slowly.
Okay, buckle up. We are diving into the absolute deepest, darkest basement of AI monitoring. Honestly? This section almost broke my brain.
When you look at a traditional app, tracking a user’s request is easy: they click a button, a server does a thing, and it logs the action. Done. But inside a high-performance AI cluster, the hardware is dealing with two massive nightmares: asynchrony and concurrency.
To understand what that actually means, let’s imagine a chaotic restaurant kitchen:
This creates an asynchrony problem: the time when the CPU launches an operation is not necessarily the time when the GPU actually runs it. As a result, a CPU-side log may show that an operation was issued at a certain moment, but the corresponding GPU-side event may occur several milliseconds later, possibly after other operations have already run.
Now throw in concurrency: imagine hundreds of customers ordering different meals all at the exact same time, and multiple kitchens (distributed GPU nodes) trying to share ingredients across the room. If a meal comes out late, how on earth do you figure out which cook dropped the ball?
Because of this, debugging performance problems is not as simple as checking whether a GPU server is online or busy. A request may be slow because a GPU kernel waited in a queue, a memory copy overlapped poorly with computation, a network transfer delayed the next step, or another request competed for the same hardware resources.
This is where the genius framework TRUFFLD by Xu et al. [6] comes to the rescue. Instead of forcing engineers to rewrite their AI code to add tracking, TRUFFLD acts like a passive wiretap hidden in the kitchen. It secretly records everything from two angles:
Once TRUFFLD collects these messy, asynchronous hardware events, it feeds them into a specialized merging algorithm. This algorithm normalizes every single event onto a unified nanosecond time base.
According to Xu et al. [6], by mapping out the exact parent-child relationships of these events, it builds a Per-Request Call-Chain Tree.
Once the call-chain tree is built, TRUFFLD uses a clever hybrid approach to find out if the hardware is misbehaving:
Without Layer 5, an AI company is completely flying blind. Their main cloud dashboard (Layer 4) might show a beautiful, glowing green light saying the servers are healthy, while the end-users are experiencing agonizing lag spikes (what engineers call tail-latency inflation).
According to Sisodia [5], this provides Hardware-Level Diagnostics. Sometimes an AI system fails or stutters not because the prompt was bad, but because the physical GPU got fragmented memory.
While Sisodia [5] has pulled together all 5 layers to abstract LLM observability, the gap remains that they are currently completely disconnected silos.
Some of these gaps identified from Sisodia [5] are:
The Disconnected Signals Gap: If an LLM suddenly outputs an answer with low confidence (Layer 2), nobody knows why. Is it because the prompt was too hard? Or is it because the physical GPU memory was overloaded at Layer 5, forcing the model to drop its context?
The Safety Budget Gap: Monitoring costs money and computing power. Forcing a model to think out loud (Layer 3) or tracing its GPU chips (Layer 5) slows down performance. There is no zero math or frameworks to help them decide where to spend their “safety budget.”
and a few others…
Twinkll Sisodia’s analysis makes one thing clear: the standard DevOps playbook is entirely blind to the unique failure modes of artificial intelligence.
A production LLM application can experience a catastrophic failure — silent hallucinations, confidence degradation, or catastrophic tail-latencies — while every traditional cloud monitoring dashboard remains perfectly green.
The future of enterprise AI doesn’t just require isolated breakthroughs like MIT’s calibration tracking, OpenAI’s behavioral diaries, or TRUFFLD’s low-level hardware tracing. It requires a central nervous system that binds them together.
[1] Chen, Y., Peng, H., Maddila, C., Bansal, C., Rajmohan, S., Zhang, Q., Lin, D., et al. (2025). “AIOpsLab: A holistic framework to evaluate AI agents for enabling autonomous clouds.” Proceedings of Machine Learning and Systems (MLSys).
[2] Damani, M., Puri, I., Slocum, S., Shenfeld, I., Choshen, L., Kim, Y., & Andreas, J. (2025). “Beyond binary rewards: Training LMs to reason about their uncertainty.” arXiv preprint arXiv:2507.16806 (Presented at ICLR 2026).
[3] Feng, J., Russell, S., & Steinhardt, J. (2025). “Monitoring latent world states in language models with propositional probes.” Proceedings of the International Conference on Learning Representations (ICLR) (Spotlight paper).
[4] Guan, M. Y., Wang, M., Carroll, M., et al. (2025). “Monitoring monitorability.” Technical report, OpenAI.
[5] Sisodia, T. (2026). “AI Observability for Large Language Model Systems: A Multi-Layer Analysis of Monitoring Approaches from Confidence Calibration to Infrastructure Tracing.” Red Hat.
**[6] **Xu, R., Li, J., Xie, Z., & Chen, P. (2026). TRUFFLD: Bridging non-intrusive tracing and fine-grained cross-layer representations for LLM inference diagnosis. Submitted to ICLR 2026.
How I Started to See Inside the LLM was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.