I shipped a logging schema to my production agent pipeline six months ago. It logged every prompt, every tool call, every response, and every latency. The dashboards looked great. The alerts never fired. Then one Tuesday morning, an agent ran a 14-step task and ended on a confident "Done." It had fabricated three of those steps. My logs had captured every call. They had just hidden the failure.
The problem wasn't volume. It was that I was logging the wrong fields in the wrong shape. After 12 weeks of iterating on what actually surfaces failures, here's the schema that finally caught the fabrication before it shipped.
Most agent logging looks like this:
log.info("agent_step", extra={
"step_id": uuid4(),
"agent_name": "research-agent",
"prompt_tokens": 1247,
"completion_tokens": 312,
"latency_ms": 1843,
"tool_calls": ["search", "fetch_url", "search"],
"tool_results": [...], # truncated to 500 chars
"status": "ok",
})
This looks comprehensive. It is useless for finding failures.
Three things break:
1. status: ok is what the model said, not what happened. The HTTP call may have returned a 401. The tool may have raised an exception. The agent doesn't see that — it sees a return value, and if that value parses as a dict, the agent logs
ok
.2. Tool results are truncated. Truncating to 500 chars hides exactly the errors you need to see. Stack traces and validation messages tend to live past the first 500 characters of a JSON response.
3. There's no notion of intent. You know the agent called
search
, but you don't know what The deeper issue: these logs are structured for cost analysis, not for catching failures. They were designed by people thinking about token spend, not by people trying to debug a fabrication at 11 PM.
After three rewrites, I landed on a schema with four mandatory fields per step. Every step log now contains:
expected_outcome
observed_outcome
verification
uncertainty_signal
Here's what it looks like in Python:
import logging
from dataclasses import dataclass, asdict
from typing import Any
logger = logging.getLogger("agent.step")
@dataclass
class StepLog:
step_id: str
agent_name: str
expected_outcome: str # What success looks like
observed_outcome: dict # What actually happened (full, not truncated)
verification: dict # Did we verify? How?
uncertainty_signal: str # "none" | "low" | "high"
tool_name: str | None = None
latency_ms: int = 0
def log_step(step: StepLog) -> None:
if not step.observed_outcome.get("success"):
severity = "ERROR"
elif step.uncertainty_signal == "high":
severity = "WARNING"
else:
severity = "INFO"
logger.log(
getattr(logging, severity),
"agent_step",
extra={"step": asdict(step)},
)
The four fields work because they map to the three failure modes I kept hitting:
observed_outcome
requiring a success flagverification
requiring an explicit re-queryuncertainty_signal
verification
is the field I underestimated. It seemed redundant at first — the agent just calls the tool, the tool returns something, done. But the difference between "called the tool" and "the tool worked" is where every fabrication I've caught has lived.
Concretely, every step that changes state needs a verification re-read. Closing a GitHub issue, sending an email, posting a comment, writing a file — all of these need a second API call to confirm the first one worked:
def log_github_close_issue(repo: str, issue_number: int) -> None:
close_response = github.post(
f"/repos/{repo}/issues/{issue_number}/close",
headers={"Authorization": f"Bearer {GITHUB_TOKEN}"},
)
verify_response = github.get(
f"/repos/{repo}/issues/{issue_number}",
headers={"Authorization": f"Bearer {GITHUB_TOKEN}"},
)
step = StepLog(
step_id=str(uuid4()),
agent_name="issue-closer",
expected_outcome=f"Issue #{issue_number} state == 'closed'",
observed_outcome={
"close_call_status": close_response.status_code,
"verify_call_status": verify_response.status_code,
"actual_state": verify_response.json().get("state"),
"success": (
close_response.status_code == 200
and verify_response.json().get("state") == "closed"
),
},
verification={
"method": "re-read-via-get",
"passed": (
close_response.status_code == 200
and verify_response.json().get("state") == "closed"
),
},
uncertainty_signal="none",
)
log_step(step)
If verification.passed
is False
, the log severity is ERROR
even if the agent itself reports status: ok
. That's how I catch the fabrication: the agent believes it succeeded, but the log fires an alert because the verification field saw the truth.
uncertainty_signal
is the field I added last, and it's the one that catches the subtlest bugs. It's a string: "none"
, "low"
, or "high"
. The model sets it explicitly after each step.
Most teams try to derive uncertainty from logit values or token probabilities. That doesn't work — those values track next-token confidence, not task-level confidence. An agent can be very sure about which token comes next while being completely wrong about whether the tool returned useful data.
Instead, I prompt the agent to self-report:
SYSTEM_PROMPT = """
After each tool call, append a single line to your reasoning:
UNCERTAINTY: <none|low|high>
Use 'high' when:
- The tool returned an unexpected format
- You had to guess what a field meant
- You are inferring success without explicit confirmation
Use 'low' when:
- The tool returned what you expected but with minor formatting differences
Use 'none' when:
- You received exactly the data structure you asked for
"""
Parsing this line gives the uncertainty_signal
. Combined with verification, it produces three alert classes:
In my pipeline, about 4% of steps fire WARNING. About 0.6% fire ERROR. That 0.6% is where every fabrication has come from. The WARNINGs are where I find bugs in my tool definitions — cases where the tool returns technically-valid data that the agent misinterprets.
Three months in, the four-field schema has done three things the old logs couldn't:
Catching fabrications is automatic now. I no longer discover "the agent said done but didn't do it" the next morning. The verification step fires an ERROR within seconds, the agent's final report gets gated on the ERROR count, and a human gets pinged before anything ships.
Bugs in tool definitions surface faster. When an agent logs uncertainty_signal: high
across many calls to the same tool, I know the tool schema or response format needs work. This was a huge blind spot before — I thought the agent was flaky when really the tool was ambiguous.
Token cost became debuggable. Because expected_outcome
is structured, I can group steps by intent. Steps that share an expected_outcome but use wildly different token counts usually indicate either a bug or a tool returning data the agent has to re-parse multiple times. The cost team loves this; they used to only see totals.
The schema is not perfect. It still doesn't catch cases where the agent fabricates the verification itself — "yes I checked, the issue is closed" when the verification API was never called. For that, I added a separate layer that runs the verification logic outside the agent loop, in a watchdog process. That's a different article.
If you're starting from scratch with agent logging, don't begin with prompt_tokens and latency. Begin with: what does this step intend to do, what actually happened, did we verify the two match, and how unsure is the model? Those four questions catch almost every production failure I've seen in the last six months. The other fields are nice to have.