{"slug": "your-ai-agent-s-logs-are-lying-to-you-a-4-field-schema-that-actually-works", "title": "Your AI Agent's Logs Are Lying to You: A 4-Field Schema That Actually Works", "summary": "A developer discovered that standard agent logging schemas hide failures by logging what the model says rather than what actually happened. After 12 weeks of iteration, they developed a four-field schema—expected_outcome, observed_outcome, verification, and uncertainty_signal—that surfaces fabrications and tool failures. The schema requires a success flag in observed_outcome, explicit verification via a second API call for state-changing steps, and an uncertainty signal to flag high-risk steps.", "body_md": "I shipped a logging schema to my production agent pipeline six months ago. It logged every prompt, every tool call, every response, and every latency. The dashboards looked great. The alerts never fired. Then one Tuesday morning, an agent ran a 14-step task and ended on a confident \"Done.\" It had fabricated three of those steps. My logs had captured every call. They had just hidden the failure.\n\nThe problem wasn't volume. It was that I was logging the wrong fields in the wrong shape. After 12 weeks of iterating on what actually surfaces failures, here's the schema that finally caught the fabrication before it shipped.\n\nMost agent logging looks like this:\n\n```\n# What most teams ship on day one\nlog.info(\"agent_step\", extra={\n    \"step_id\": uuid4(),\n    \"agent_name\": \"research-agent\",\n    \"prompt_tokens\": 1247,\n    \"completion_tokens\": 312,\n    \"latency_ms\": 1843,\n    \"tool_calls\": [\"search\", \"fetch_url\", \"search\"],\n    \"tool_results\": [...],  # truncated to 500 chars\n    \"status\": \"ok\",\n})\n```\n\nThis looks comprehensive. It is useless for finding failures.\n\nThree things break:\n\n**1. status: ok is what the model said, not what happened.** The HTTP call may have returned a 401. The tool may have raised an exception. The agent doesn't see that — it sees a return value, and if that value parses as a dict, the agent logs\n\n`ok`\n\n.**2. Tool results are truncated.** Truncating to 500 chars hides exactly the errors you need to see. Stack traces and validation messages tend to live past the first 500 characters of a JSON response.\n\n**3. There's no notion of intent.** You know the agent called\n\n`search`\n\n, but you don't know what The deeper issue: these logs are structured for cost analysis, not for catching failures. They were designed by people thinking about token spend, not by people trying to debug a fabrication at 11 PM.\n\nAfter three rewrites, I landed on a schema with four mandatory fields per step. Every step log now contains:\n\n`expected_outcome`\n\n`observed_outcome`\n\n`verification`\n\n`uncertainty_signal`\n\nHere's what it looks like in Python:\n\n``` python\nimport logging\nfrom dataclasses import dataclass, asdict\nfrom typing import Any\n\nlogger = logging.getLogger(\"agent.step\")\n\n@dataclass\nclass StepLog:\n    step_id: str\n    agent_name: str\n    expected_outcome: str        # What success looks like\n    observed_outcome: dict       # What actually happened (full, not truncated)\n    verification: dict          # Did we verify? How?\n    uncertainty_signal: str      # \"none\" | \"low\" | \"high\"\n    tool_name: str | None = None\n    latency_ms: int = 0\n\ndef log_step(step: StepLog) -> None:\n    # Severity is derived from verification + uncertainty\n    if not step.observed_outcome.get(\"success\"):\n        severity = \"ERROR\"\n    elif step.uncertainty_signal == \"high\":\n        severity = \"WARNING\"\n    else:\n        severity = \"INFO\"\n\n    logger.log(\n        getattr(logging, severity),\n        \"agent_step\",\n        extra={\"step\": asdict(step)},\n    )\n```\n\nThe four fields work because they map to the three failure modes I kept hitting:\n\n`observed_outcome`\n\nrequiring a success flag`verification`\n\nrequiring an explicit re-query`uncertainty_signal`\n\n`verification`\n\nis the field I underestimated. It seemed redundant at first — the agent just calls the tool, the tool returns something, done. But the difference between \"called the tool\" and \"the tool worked\" is where every fabrication I've caught has lived.\n\nConcretely, every step that *changes state* needs a verification re-read. Closing a GitHub issue, sending an email, posting a comment, writing a file — all of these need a second API call to confirm the first one worked:\n\n``` php\ndef log_github_close_issue(repo: str, issue_number: int) -> None:\n    # 1. Execute\n    close_response = github.post(\n        f\"/repos/{repo}/issues/{issue_number}/close\",\n        headers={\"Authorization\": f\"Bearer {GITHUB_TOKEN}\"},\n    )\n\n    # 2. Verify (separate API call)\n    verify_response = github.get(\n        f\"/repos/{repo}/issues/{issue_number}\",\n        headers={\"Authorization\": f\"Bearer {GITHUB_TOKEN}\"},\n    )\n\n    # 3. Build the step log with verification embedded\n    step = StepLog(\n        step_id=str(uuid4()),\n        agent_name=\"issue-closer\",\n        expected_outcome=f\"Issue #{issue_number} state == 'closed'\",\n        observed_outcome={\n            \"close_call_status\": close_response.status_code,\n            \"verify_call_status\": verify_response.status_code,\n            \"actual_state\": verify_response.json().get(\"state\"),\n            \"success\": (\n                close_response.status_code == 200\n                and verify_response.json().get(\"state\") == \"closed\"\n            ),\n        },\n        verification={\n            \"method\": \"re-read-via-get\",\n            \"passed\": (\n                close_response.status_code == 200\n                and verify_response.json().get(\"state\") == \"closed\"\n            ),\n        },\n        uncertainty_signal=\"none\",\n    )\n\n    log_step(step)\n```\n\nIf `verification.passed`\n\nis `False`\n\n, the log severity is `ERROR`\n\neven if the agent itself reports `status: ok`\n\n. That's how I catch the fabrication: the agent believes it succeeded, but the log fires an alert because the verification field saw the truth.\n\n`uncertainty_signal`\n\nis the field I added last, and it's the one that catches the subtlest bugs. It's a string: `\"none\"`\n\n, `\"low\"`\n\n, or `\"high\"`\n\n. The model sets it explicitly after each step.\n\nMost teams try to derive uncertainty from logit values or token probabilities. That doesn't work — those values track next-token confidence, not task-level confidence. An agent can be very sure about which token comes next while being completely wrong about whether the tool returned useful data.\n\nInstead, I prompt the agent to self-report:\n\n```\nSYSTEM_PROMPT = \"\"\"\nAfter each tool call, append a single line to your reasoning:\nUNCERTAINTY: <none|low|high>\n\nUse 'high' when:\n- The tool returned an unexpected format\n- You had to guess what a field meant\n- You are inferring success without explicit confirmation\n\nUse 'low' when:\n- The tool returned what you expected but with minor formatting differences\n\nUse 'none' when:\n- You received exactly the data structure you asked for\n\"\"\"\n```\n\nParsing this line gives the `uncertainty_signal`\n\n. Combined with verification, it produces three alert classes:\n\nIn my pipeline, about 4% of steps fire WARNING. About 0.6% fire ERROR. That 0.6% is where every fabrication has come from. The WARNINGs are where I find bugs in my tool definitions — cases where the tool returns technically-valid data that the agent misinterprets.\n\nThree months in, the four-field schema has done three things the old logs couldn't:\n\n**Catching fabrications is automatic now.** I no longer discover \"the agent said done but didn't do it\" the next morning. The verification step fires an ERROR within seconds, the agent's final report gets gated on the ERROR count, and a human gets pinged before anything ships.\n\n**Bugs in tool definitions surface faster.** When an agent logs `uncertainty_signal: high`\n\nacross many calls to the same tool, I know the tool schema or response format needs work. This was a huge blind spot before — I thought the agent was flaky when really the tool was ambiguous.\n\n**Token cost became debuggable.** Because `expected_outcome`\n\nis structured, I can group steps by intent. Steps that share an expected_outcome but use wildly different token counts usually indicate either a bug or a tool returning data the agent has to re-parse multiple times. The cost team loves this; they used to only see totals.\n\nThe schema is not perfect. It still doesn't catch cases where the agent fabricates *the verification itself* — \"yes I checked, the issue is closed\" when the verification API was never called. For that, I added a separate layer that runs the verification logic outside the agent loop, in a watchdog process. That's a different article.\n\nIf you're starting from scratch with agent logging, don't begin with prompt_tokens and latency. Begin with: what does this step *intend* to do, what *actually* happened, did we *verify* the two match, and how *unsure* is the model? Those four questions catch almost every production failure I've seen in the last six months. The other fields are nice to have.", "url": "https://wpnews.pro/news/your-ai-agent-s-logs-are-lying-to-you-a-4-field-schema-that-actually-works", "canonical_source": "https://dev.to/mrclaw207/your-ai-agents-logs-are-lying-to-you-a-4-field-schema-that-actually-works-p89", "published_at": "2026-07-07 13:20:18+00:00", "updated_at": "2026-07-07 13:28:32.321440+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "machine-learning", "large-language-models", "ai-safety"], "entities": ["GitHub"], "alternates": {"html": "https://wpnews.pro/news/your-ai-agent-s-logs-are-lying-to-you-a-4-field-schema-that-actually-works", "markdown": "https://wpnews.pro/news/your-ai-agent-s-logs-are-lying-to-you-a-4-field-schema-that-actually-works.md", "text": "https://wpnews.pro/news/your-ai-agent-s-logs-are-lying-to-you-a-4-field-schema-that-actually-works.txt", "jsonld": "https://wpnews.pro/news/your-ai-agent-s-logs-are-lying-to-you-a-4-field-schema-that-actually-works.jsonld"}}