Your AI Agent's Logs Are Lying to You: A 4-Field Schema That Actually Works

A developer discovered that standard agent logging schemas hide failures by logging what the model says rather than what actually happened. After 12 weeks of iteration, they developed a four-field schema—expected_outcome, observed_outcome, verification, and uncertainty_signal—that surfaces fabrications and tool failures. The schema requires a success flag in observed_outcome, explicit verification via a second API call for state-changing steps, and an uncertainty signal to flag high-risk steps.

I shipped a logging schema to my production agent pipeline six months ago. It logged every prompt, every tool call, every response, and every latency. The dashboards looked great. The alerts never fired. Then one Tuesday morning, an agent ran a 14-step task and ended on a confident "Done." It had fabricated three of those steps. My logs had captured every call. They had just hidden the failure. The problem wasn't volume. It was that I was logging the wrong fields in the wrong shape. After 12 weeks of iterating on what actually surfaces failures, here's the schema that finally caught the fabrication before it shipped. Most agent logging looks like this: What most teams ship on day one log.info "agent step", extra={ "step id": uuid4 , "agent name": "research-agent", "prompt tokens": 1247, "completion tokens": 312, "latency ms": 1843, "tool calls": "search", "fetch url", "search" , "tool results": ... , truncated to 500 chars "status": "ok", } This looks comprehensive. It is useless for finding failures. Three things break: 1. status: ok is what the model said, not what happened. The HTTP call may have returned a 401. The tool may have raised an exception. The agent doesn't see that — it sees a return value, and if that value parses as a dict, the agent logs ok . 2. Tool results are truncated. Truncating to 500 chars hides exactly the errors you need to see. Stack traces and validation messages tend to live past the first 500 characters of a JSON response. 3. There's no notion of intent. You know the agent called search , but you don't know what The deeper issue: these logs are structured for cost analysis, not for catching failures. They were designed by people thinking about token spend, not by people trying to debug a fabrication at 11 PM. After three rewrites, I landed on a schema with four mandatory fields per step. Every step log now contains: expected outcome observed outcome verification uncertainty signal Here's what it looks like in Python: python import logging from dataclasses import dataclass, asdict from typing import Any logger = logging.getLogger "agent.step" @dataclass class StepLog: step id: str agent name: str expected outcome: str What success looks like observed outcome: dict What actually happened full, not truncated verification: dict Did we verify? How? uncertainty signal: str "none" | "low" | "high" tool name: str | None = None latency ms: int = 0 def log step step: StepLog - None: Severity is derived from verification + uncertainty if not step.observed outcome.get "success" : severity = "ERROR" elif step.uncertainty signal == "high": severity = "WARNING" else: severity = "INFO" logger.log getattr logging, severity , "agent step", extra={"step": asdict step }, The four fields work because they map to the three failure modes I kept hitting: observed outcome requiring a success flag verification requiring an explicit re-query uncertainty signal verification is the field I underestimated. It seemed redundant at first — the agent just calls the tool, the tool returns something, done. But the difference between "called the tool" and "the tool worked" is where every fabrication I've caught has lived. Concretely, every step that changes state needs a verification re-read. Closing a GitHub issue, sending an email, posting a comment, writing a file — all of these need a second API call to confirm the first one worked: php def log github close issue repo: str, issue number: int - None: 1. Execute close response = github.post f"/repos/{repo}/issues/{issue number}/close", headers={"Authorization": f"Bearer {GITHUB TOKEN}"}, 2. Verify separate API call verify response = github.get f"/repos/{repo}/issues/{issue number}", headers={"Authorization": f"Bearer {GITHUB TOKEN}"}, 3. Build the step log with verification embedded step = StepLog step id=str uuid4 , agent name="issue-closer", expected outcome=f"Issue {issue number} state == 'closed'", observed outcome={ "close call status": close response.status code, "verify call status": verify response.status code, "actual state": verify response.json .get "state" , "success": close response.status code == 200 and verify response.json .get "state" == "closed" , }, verification={ "method": "re-read-via-get", "passed": close response.status code == 200 and verify response.json .get "state" == "closed" , }, uncertainty signal="none", log step step If verification.passed is False , the log severity is ERROR even if the agent itself reports status: ok . That's how I catch the fabrication: the agent believes it succeeded, but the log fires an alert because the verification field saw the truth. uncertainty signal is the field I added last, and it's the one that catches the subtlest bugs. It's a string: "none" , "low" , or "high" . The model sets it explicitly after each step. Most teams try to derive uncertainty from logit values or token probabilities. That doesn't work — those values track next-token confidence, not task-level confidence. An agent can be very sure about which token comes next while being completely wrong about whether the tool returned useful data. Instead, I prompt the agent to self-report: SYSTEM PROMPT = """ After each tool call, append a single line to your reasoning: UNCERTAINTY: <none|low|high Use 'high' when: - The tool returned an unexpected format - You had to guess what a field meant - You are inferring success without explicit confirmation Use 'low' when: - The tool returned what you expected but with minor formatting differences Use 'none' when: - You received exactly the data structure you asked for """ Parsing this line gives the uncertainty signal . Combined with verification, it produces three alert classes: In my pipeline, about 4% of steps fire WARNING. About 0.6% fire ERROR. That 0.6% is where every fabrication has come from. The WARNINGs are where I find bugs in my tool definitions — cases where the tool returns technically-valid data that the agent misinterprets. Three months in, the four-field schema has done three things the old logs couldn't: Catching fabrications is automatic now. I no longer discover "the agent said done but didn't do it" the next morning. The verification step fires an ERROR within seconds, the agent's final report gets gated on the ERROR count, and a human gets pinged before anything ships. Bugs in tool definitions surface faster. When an agent logs uncertainty signal: high across many calls to the same tool, I know the tool schema or response format needs work. This was a huge blind spot before — I thought the agent was flaky when really the tool was ambiguous. Token cost became debuggable. Because expected outcome is structured, I can group steps by intent. Steps that share an expected outcome but use wildly different token counts usually indicate either a bug or a tool returning data the agent has to re-parse multiple times. The cost team loves this; they used to only see totals. The schema is not perfect. It still doesn't catch cases where the agent fabricates the verification itself — "yes I checked, the issue is closed" when the verification API was never called. For that, I added a separate layer that runs the verification logic outside the agent loop, in a watchdog process. That's a different article. If you're starting from scratch with agent logging, don't begin with prompt tokens and latency. Begin with: what does this step intend to do, what actually happened, did we verify the two match, and how unsure is the model? Those four questions catch almost every production failure I've seen in the last six months. The other fields are nice to have.