# An Agent You Cannot Watch Is an Agent You Cannot Trust. > Source: > Published: 2026-07-11 09:35:25+00:00 This is the second stage of a build-in-public series, and it helps to know where the first one landed. Stage one built [PromptProof](https://github.com/HelloJahid/PromptProof), a small engine that makes a single prompt prove itself, with typed gates between steps, retries that carry their own error message, and a feedback loop that revises an output until it passes. You do not need to have read it to follow this piece, since everything here stands on its own, but if making an LLM trustworthy rather than merely clever appeals to you, that is the story it tells. Hold onto one word from it, proof, because this stage is about earning it again, one level up. PromptProof proved an output. This stage has to prove something slipperier, a system that chooses its own next move. And choosing a move is the easy part now. Anyone can wire a model to a couple of tools in sixty lines of Python, the same while True loop every tutorial shows, and watch something that looks alive answer a question. That demo is nearly free in 2026. What is not free, what has quietly become the real skill, is being able to say why the thing did what it did, and whether you can trust it to do the same tomorrow. Building an agent is cheap now. Proving it works is the skill. I know that thesis is true because I watched it happen on my own machine, late one night, in a file the agent did not know it was writing. Let me show you the moment it surprised me. I had asked a simple question, what did NVIDIA announce at its most recent GTC event, and the agent’s flight recorder was writing down every thought it had while I read along. ``` php [2] model wishes -> web_search("NVIDIA GTC 2025 announcements") web_search("NVIDIA GTC keynote latest news")[3] tools web_search [ok] -> "GTC 2025 announcements and live updates..."[4] model wishes -> web_search("NVIDIA GTC Taipei 2026 announcements") web_search('"GTC Taipei 2026" NVIDIA news') ``` Look at step 2 and step 4. The first search assumed the latest GTC was the 2025 event, a stale guess frozen in the model’s training data. The results came back hinting at something newer. And then, without me touching anything, the agent rewrote its own query to GTC Taipei 2026, in quotes, the way a person searches once they have stopped guessing and started verifying. I did not program that correction. I programmed the conditions for it. And I only know it happened because every step sits in a file I can replay, inspect, and show you. That file, and everything that earns its trust, is what this article builds. AgentProof is a from-scratch runtime, no LangGraph and no CrewAI, about 1,500 lines of readable Python, built to make one claim true. An agent you cannot watch, and cannot grade, is an agent you cannot trust. So the project comes in two movements, and so does this article. First we make the agent watchable, a state machine with a flight recorder wired into its spine, so nothing it does is hidden. Then we make it gradable, an evaluation harness that judges the recording and a CI gate that turns red when behaviour slips. By the end you will read that trace above like an investigator, and I will show you the two things it caught that I never saw coming. Some of what follows will look familiar if you read stage one, the same gates, the same retry-with-feedback, the same mocked-everything tests, and that is deliberate, because the two share a toolbox and only the driver changed. PromptProof followed a script I wrote. This thing picks its own route through the problem, and I find out afterwards. You cannot watch what has no shape. So before a single line of the recorder exists, the runtime has to settle two things, what the agent knows and how it carries that knowledge from one step to the next. Both begin with a problem every language model has. The first honest thing to know about a language model API is that it remembers nothing. Every call starts from a blank slate. You ask a question, you get an answer, and on the next call the model has no idea that either the question or the answer ever existed. It is not being difficult. It simply has no memory of its own from one request to the next. Think about how a busy hospital ward handles the same problem. Staff change shift, and the nurse who admitted you at noon is long gone by the evening, so no single person carries your case in their head. What holds everything together is the chart clipped to the end of the bed. Every reading, every medication, every instruction is written there, and whoever walks in next can pick it up and carry on without missing a thing. The chart is the memory. The people passing through are interchangeable. An agent needs exactly that chart. The tutorials skip it and scatter the case notes instead, a messages list in one variable, a tool_results in another, a counter somewhere else. It holds together right up until the night you are debugging at midnight and cannot answer the one question that matters, what did the agent actually know at step 3. So the first thing I built was the chart, a single typed object that every step reads from and writes back to. ``` class AgentState(BaseModel): model_config = ConfigDict(extra="forbid") query: str # why this run exists instructions: str = "" # who the agent is messages: list[Message] # the conversation so far pending_tool_calls: list[ToolCall] # what the model WANTS done tool_results: list[ToolResult] # what actually HAPPENED history: list[StepRecord] # which steps ran, in order input_tokens: int = 0 # the running bill output_tokens: int = 0 final_answer: str | None = None # the exit condition ``` Two decisions inside this object quietly carry the whole project. The first is the split between ToolCall and ToolResult. A ToolCall is a request the model makes, the equivalent of a doctor writing an order on the chart for a blood test. A ToolResult is what actually came back once someone accountable carried that order out. The model only ever asks, and something else does the work and records the outcome. Hold on to that separation, because it becomes a security boundary later, and it is the reason the opening trace could show a request at step 2 and its consequence at step 3 as two separate, linkable facts. The second is one unremarkable line, extra="forbid". Any field this chart does not already have a place for is rejected the instant it appears, as a ValidationError. A typo dies at the boundary where it was born, rather than three steps later, disguised as a confident wrong answer with no obvious cause. One last point, because it matters further on. This chart is working memory, not a permanent record. It exists for a single run and clears when that run ends, the way a bedside chart belongs to one admission rather than to your whole medical history. Teaching an agent to remember across sessions is a different problem for a different article. What this object solves is the forgetting inside a single task, so that what the agent learned at step 2 is still in front of it at step 6. So the agent has its chart. Now the question is how you get reliable work out of a worker who remembers nothing. Factories answered this a century ago, with stations. Each station does one job on the piece and passes it on. The welder does not remember what the cutter did, and does not need to, because the piece carries its own history down the line. That is the runtime in one sentence. Receive the state, do one job, return the new state. A step is anything with a name and that one method. ``` php class Step(Protocol): name: str def run(self, state: AgentState) -> AgentState: ... ``` Steps never call each other. The prepare step builds the opening conversation, the model step talks to the LLM, and the tools step executes wishes. Each one is a few dozen lines, testable alone and replaceable alone. An assembly line is fixed, though, and an agent is not, which is the entire point. So between every two steps sits a router, a function that reads the state and decides what runs next. Not a config file. Not a hardcoded order. A decision, made fresh after every step, based on what the piece looks like right now. And here is the reveal that genuinely surprised me when I reached it. ReAct, the reason then act then observe pattern underneath basically every agent framework you have heard of, is not a framework feature. It is a routing policy. Here is all of it. ``` php def react_router(state: AgentState, current: str) -> str | None: if state.pending_tool_calls: # the model wished for tools? return "tools" # -> go execute them if current == "prepare": return "model" # after setup -> think if current == "tools": return "model" # after observing -> think again return None # nothing left to do -> stop ``` Six lines of logic. The famous model then tools then model loop from my opening trace is this function returning tools, then model, then, when the model finally answered instead of wishing, None. The path emerges from the state, and nobody scripted it. That is why my agent could decide on its own that one round of searching was not enough. Which raises the paranoid question straight away. If the path emerges at runtime, what stops it from looping forever, a model that keeps wishing for one more search, one more, one more, burning tokens in a cycle nobody scripted and nobody is watching. A budget stops it. The machine counts steps, and past the limit it raises a typed error, MaxStepsExceeded, loudly, with the limit in its hands. Not a timeout buried in an HTTP client. A named, catchable, traceable failure. The house rule of this codebase is that the loop continues until the model has enough, and enough must never mean forever. One more piece is bolted to this machine, and it is the reason the opening scene could happen at all. The machine does the bookkeeping. After every step it records what ran into the state’s history, and, as we will see, taps every one of those moments out to a file that survives anything. The steps do not know they are being recorded. That is what makes the recording honest. The runtime worked beautifully for exactly as long as the tools were imaginary. Then I connected a real API, and my first week read like a support ticket log. Monday brought a rate limit. My agent received an error payload, fed it straight into its next thought, and reasoned earnestly about an error message as if it were search results. Tuesday brought a silent API upgrade that renamed a field, and a temperature reading the agent expected simply vanished. Nothing crashed. The agent just quietly stopped knowing the temperature and started improvising. Wednesday brought a hallucinated tool, get_wether, a name that has never existed. The naive loop threw a KeyError and the whole run died with a traceback nobody asked for. Three failures, three completely different culprits, and the tutorial loop treats them identically, crash, or worse, reason over garbage. One corrupted observation early poisons every thought after it. The fix that finally stuck was not better error handling. It was a diagnosis rule. Before choosing the medicine, decide whose mistake it is. A hallucinated tool and missing arguments are the model’s mistake. Retrying the API changes nothing, because the same bad wish would just fail again. The right medicine is feedback, an error observation that names what is available and what is malformed, so the model’s next wish is different. Anyone who read Part two will know this move, **PromptProof****s **,retry-with-feedback, promoted from prompts to tools. Rate limits and mangled payloads are the world’s mistake. The model’s wish was fine, and reality flaked. The right medicine is a silent retry, and if the world stays broken after three attempts, a structured error the agent can actually reason about, worded plainly rather than handed over as a raw traceback. Structurally that is an airlock with a gate on each side. Nothing passes from the model to the world unvalidated, since arguments are checked against the tool’s schema before any API is touched, and nothing passes from the world to the model unvalidated, since responses are checked against an output schema before the agent ever sees them. ``` # tools/executor.py — the fault line, in two except-blockstry: args = self._registry.validate_call(call)except ToolFailure as exc: # The MODEL's mistake: no point retrying the transport. return ToolResult(call_id=call.id, name=call.name, output=f"ERROR: {exc.reason}", is_error=True)for _attempt in range(1, self._max_attempts + 1): try: raw = self._transport.execute(call, args) output = self._check_observation(tool, raw) # gate 2 return ToolResult(call_id=call.id, name=call.name, output=output) except (TransportError, GateFailure) as exc: # The WORLD's mistake: transient by assumption -- go again. last_reason = str(exc) ``` My favourite test in the whole suite demonstrates this as a magic trick. The scripted API rate-limits on attempt one and succeeds on attempt two. The assertion is that the agent’s trajectory is identical to the happy run, same path, same history, same answer. The transport was called twice, and the reasoning chain never knew. The failure was absorbed two layers beneath thought. That is what production-ready actually means for an agent, and it is worth saying as a sentence. An agent becomes production-grade not when it can call tools, but when a failing tool cannot corrupt it. A fair question arrives at this point. How do you test any of that? Retries, rate limits, schema drift, am I supposed to reproduce those against a live API, in CI, with secrets in the pipeline, hoping the world misbehaves on cue? You do not test the world. You test against a world you scripted. The runtime never imports anthropic. It never imports httpx. There are exactly two doors into the system, and both are small Python protocols. ``` python class ModelClient(Protocol): def complete(self, instructions, messages, tools=None) -> ModelResponse: ...class ToolTransport(Protocol): def execute(self, call: ToolCall, args: BaseModel) -> Any: ... ``` Claude walks through the first door in production, and a live search API through the second. In tests, scripted mocks walk through the same doors, a mock model that replays queued responses and a mock transport whose script can say rate-limit once, then malformed payload, then fine, in three lines. And here is the opinion I will defend in the comments. The mocks live in the package, not in the test folder. MockModelClient and MockTransport ship with the library, documented and first-class. The promise that the entire test suite runs with no API keys and no network is not a testing shortcut, it is a design requirement the architecture has to earn. Ninety-plus tests prove every failure mode from the previous section, offline, deterministically, in about one second. The punchline came when I built the demo agent from the opening. It is the same class the tests exercise. The entire difference between the tested agent and the deployed agent is which objects get passed to the constructor. Not a code path. An argument. Then my live run handed me a bill, 22,104 input tokens, for one question. Here is why. Remember the amnesia, every model call re-sends the conversation. And by round two, the conversation contained two full payloads of search results, riding along again and again on every subsequent call. Long conversations do not grow linearly in cost, they compound. The obvious fix is to trim the message list. Delete old messages, save tokens, done. And in this codebase, that obvious fix would have been quiet sabotage, because the state is not just the agent’s memory. It is the evidence. The flight recorder in the next section snapshots that state at every step, and the evaluation harness in the second half judges those snapshots. Trim the state, and you have shredded the flight recorder’s tape to save on ink. The resolution is one sentence I now apply everywhere. Memory is a view, not a mutation. The run knows everything, and state.messages only ever grows. A memory policy decides how much of it the model sees on each call. ``` php class SlidingWindow: def view(self, messages: list[Message]) -> list[Message]: system, rest = _split_system(messages) # identity always survives window = _drop_orphan_tools(rest[-self._max_turns:]) return system + window ``` Three policies, about ten lines each. Send everything, for maximum fidelity and a compounding bill. Send the last N turns, for a bounded bill that might drop something. Or summarise the old and keep the recent, for something compact that is only as good as the summary, which is itself produced through the same ModelClient door, so even the summariser is mockable. Two details in that little window function carry real scars. _split_system exists because the system prompt is identity, not history, and if the persona scrolls out of the window your careful instructions silently stop existing. And _drop_orphan_tools guards a trap with a name. A window must never start with a tool result whose requesting turn was cut off. An observation without its intent confuses the model, and some providers reject it outright, because results must link back to the wish that caused them. There is the ToolCall and ToolResult split from earlier, earning its keep again. The early 1950s brought a horror the aviation industry had never seen. De Havilland Comets, the world’s first commercial jets, started falling out of the sky, and nobody knew why. The industry’s answer was not to watch the planes harder. It was a box, record everything in something that survives the crash, and let investigators read the flight afterwards. That is the component this whole first half has been building toward, and its design rules each carry a reason. **Typed events, one JSON line each.** A trace is data, not logs. Four event kinds, run started, step completed, run finished, run failed, each a Pydantic schema, parseable back into typed objects years later. **Write and flush on every event.** The recorder never buffers. A run that dies mid-flight still left its story up to the final step, because a black box that only saves on landing is useless at a crash site. **A full state snapshot inside every step event.** Yes, it is redundant, and that is the point. It costs bytes and buys time travel, the exact contents of the agent’s working memory at any moment, without re-executing anything. **Replay rebuilds the run from the file alone.** No live process, no API, no luck. The house rule, stated as bluntly as I state it in the repo, is that if it is not in the trace, it did not happen. **Truncated is not corrupted.** A trace that simply stops, because the process was killed mid-run, loads fine and is marked truncated, since the crash is precisely when the box matters most. But a trace with pages torn from the middle, with sequence gaps or mixed run IDs, is refused with a ReplayError. A story that ends early is still true. A story with missing pages can no longer be trusted, and evidence you cannot trust is worse than no evidence. I got to watch that last rule defend itself, live, against my own bug. I re-ran my evaluation suite one afternoon, the same command, second time that day, and it crashed with a ReplayError saying the trace mixed events from more than one run. My first reaction was annoyance. My second was the slow realisation that the integrity check had just caught real corruption. The recorder opened its files in append mode, so the second run's events had spliced onto the first run's file. One file, two interleaved stories. The replay loader looked at it and refused, exactly as designed, rather than quietly grading my agent against Frankenstein evidence. The fix was one character, open with "w" and not "a", because one file is one run. That was always the contract, and the recorder just had not been enforcing it. The uncomfortable lesson is the one I keep retelling. Ninety green tests never found that bug. Five minutes of actually using the thing did. Every test wrote its traces into a fresh temporary directory, and no test had ever reused a path, because I had never imagined reusing one. Coverage follows imagination. Usage finds what imagination missed. Back to the opening trace, because you can now read the whole flight the way an investigator would. ``` === run fde1e6fce631 -- finished ===query: What did NVIDIA announce at its most recent GTC event?path: prepare -> model -> tools -> model -> tools -> modeltokens: 22104 in / 2363 out ``` Step 1, prepare, seeds the conversation with a system turn carrying four grounding rules, cite every claim with its URL, prefer trustworthy sources, say plainly when you cannot find reliable information, and keep it short. Step 2, model, is the first thought, ending in two tool wishes. Step 3, tools, executes both, gates the payloads, and links each observation to its wish. Steps 4 and 5 are the correction you have already seen. Step 6 is sixteen seconds of synthesis, and an answer where every claim ends in a bracketed source. Two things in this trace genuinely surprised me, and I want to name them honestly. The first is a capability I never designed. In step 2, the model issued two tool calls in a single turn, parallel searches. I never planned for that. But the tools step is just a humble loop that drains wishes from the state, one after another. Two wishes, two executions, two linked observations. A clean contract absorbed a case its author never imagined. The second is a behaviour I never programmed. Search, notice the results are stale, reformulate the query, search again, and only then answer. Retrieve, reflect, retry. I wrote four instruction rules and a six-line router, and the loop did the rest. That pattern has a name in the literature, agentic RAG, and it is supposed to be a later stage of this series. My agent walked into it early, uninvited, and left the evidence in its own flight recorder. And that is the real punchline of the demo. The researcher contains zero new machinery. One tool schema. One instruction block. Everything else is the runtime you just read about. When the runtime is right, an agent is configuration. Here is what the flight recorder bought me, concretely. Every claim in this post is a file on my disk. The self-correction, the parallel calls, the 22k-token bill, the append-mode bug, I did not reconstruct any of it from memory. I replayed it. But I want to turn now to what the recorder cannot do, because it is the whole reason there is a Part four. A trace tells you what happened. It cannot tell you whether what happened was any good. I have a run in my files where the agent finished cleanly, took a perfect path, stayed on budget, and gave a wrong answer. And I have a scorecard from one strange afternoon where a broken search key produced two nearly identical runs, and an automated judge passed one and failed the other. Correctly, both times. That afternoon is where Part four begins. Building the grader that made those two verdicts possible, one you can trust without trusting it blindly, is the whole of the next post. It takes the recordings this runtime now produces and turns them into a grade you can defend, four dimensions of quality, an LLM judge held under strict courtroom rules, and a CI gate that turns red the moment behaviour slips. This post gave the agent a perfect memory of what it did. Part four is where we decide whether what it did was any good. Everything built here is open and runnable at [github.com/HelloJahid/AgentProof](https://github.com/HelloJahid/AgentProof). Part four picks up exactly where this leaves off, and you can [read it here](https://pub.towardsai.net/PART_4_URL). *Part one and two built **PromptProof**. This is part three of the series, and Part four is **here**.* [An Agent You Cannot Watch Is an Agent You Cannot Trust.](https://pub.towardsai.net/building-an-agent-is-cheap-now-proving-it-works-is-the-skill-35bd2a5f62e9) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.