The Loop Was the Easy Part: Evals, Observability, and Rollbacks for Your DIY Claude Code

A developer built a DIY Claude Code agent using LangChain's Deep Agents library and then added evaluation harnesses, observability, and rollback capabilities to make it production-ready. The post details how to wrap the agent loop with flight recorders, token budget kill-switches, and checkpoint/snapshot systems for debugging and recovery. This work addresses the gap between a working prototype and a dependable coding agent that can be trusted with real tasks.

I wrote a blog post on building your own Claude Code using LangChain’s Deep Agents https://medium.com/towards-artificial-intelligence/build-your-own-claude-code-using-langchin-a-deepdive-into-langchains-deep-agents-9ef98d98a69a — opening up the harness that turns a language model into a coding agent and rebuilding it piece by piece, with tests proving each piece actually works. After it went out, someone suggested that the parts that really make Claude Code-like agents dependable are the ones It was a fair point, and it stuck with me. So I decided to write Part 2. This post stands on its own — you don’t need Part 1 to follow it. But you do need the sixty-second version of what we built there: Part 1 in sixty seconds We rebuilt the anatomy of Claude Code on LangChain’s deepagents library. One call — create deep agent model=..., tools=..., system prompt=... — gives you the agent loop ask the model what to do next; if it asks for a tool, run it and feed the result back; repeat until it answers in plain text , a built-in tool suite read/write/edit files, glob, grep, a shell, a to-do planner, subagent delegation , and automatic context management. Two pieces from Part 1 do heavy lifting in this post: The approval gate:interrupt on={"execute": True, "write file": True, "edit file": True} pauses the loopbeforeany destructive tool runs and waits for a human to approve or reject. The checkpointer:checkpointer=InMemorySaver saves the agent's full state at every step, which is what made conversations persistent. Everything else is introduced as we go. So: a working coding agent in under a hundred lines. Here’s the uncomfortable part — “working” was doing a lot of heavy lifting in that sentence. Hand that agent a real task and watch what happens. It churns for two minutes, makes forty tool calls, edits some files, and announces “Done ” Now answer three questions: Claude Code has an answer to all three, and none of them live in the model: Part 1 built the agent. Part 2 makes it operable. Same rules as before: for every capability we’ll look at how Claude Code does it, then build it on Deep Agents — and prove it works with tests. In this post Here’s the map. Nothing in it touches the loop from Part 1; it all wraps around it: the eval harness — Part 9, runs in CI, not in prod scripted scenarios ──▶ grade trajectory + outcome compare vs. baseline ──▶ block regressions before merge │ exercises a copy of... ▼┌───────────────────────────────────────────────────────────────────┐│ flight recorder + meter Part 8 ││ ┌─────────────────────────────────────────────────────────────┐ ││ │ the agent loop from Part 1 unchanged │ ││ │ │ ││ │ user ──▶ model decides ──▶ tool call? ──yes──▶ PAUSE │ ││ │ ▲ │ approval gate │ ││ │ └──── result fed back ─┤ + snapshot the │ ││ │ no workspace │ ││ │ ▼ │ ││ │ plain-text answer │ ││ └─────────────────────────────────────────────────────────────┘ ││ every model + tool call recorded · token budget kill-switch │└───────────────────────────────────────────────────────────────────┘┌───────────────────────────────────────────────────────────────────┐│ checkpoints + snapshots Part 10 : rewind the conversation, ││ restore the workspace — two undo buttons for two kinds of damage │└───────────────────────────────────────────────────────────────────┘ You cannot debug what you cannot see, and an agent gives you a lot not to see. A chatbot has one input and one output. An agent has one input, one output, and an arbitrarily long middle — the tool calls, the intermediate model decisions, the tokens each step consumed. When the answer is wrong, the bug is almost always in the middle. The temptation is to add print statements to your tools. Don't. The harness already emits an event for everything it does; you just have to listen. In LangChain, listening is a callback handler — an object you pass at invoke time that gets told about every model call with token usage attached and every tool call name, args, output . Crucially, this requires zero changes to the agent . Observability is a property of the harness, not the prompt. The whole tracer is one class. Record a span when something starts, close it when it ends, keep the results immutable: python from langchain core.callbacks import BaseCallbackHandlerclass AgentTracer BaseCallbackHandler : """Records a span for every model call and tool call in a run.""" def on chat model start self, serialized, messages, , run id, kw : self. starts run id = time.monotonic def on llm end self, response, , run id, kw : duration = time.monotonic - self. starts.pop run id message = response.generations 0 0 .message usage = message.usage metadata or {} input/output tokens self.model spans.append ModelSpan duration, usage.get "input tokens", 0 , usage.get "output tokens", 0 def on tool start self, serialized, input str, , run id, kw : ... def on tool end self, output, , run id, kw : ... def on tool error self, error, , run id, kw : ... Attach it when you invoke — this is the part people miss; it goes in the config , not the agent: tracer = AgentTracer model name="claude-sonnet-4-6" result = agent.invoke {"messages": {"role": "user", "content": "Fix the login tests."} }, {"callbacks": tracer }, ◀ the flight recorder rides along print format trace tracer php tool read file {'file path': 'auth/login.py'} - def login user, ...tool edit file {'file path': 'auth/login.py', ...} - Updated filetool run tests {'path': '.'} - 3 passed in 0.41sTOTAL 4 model calls | 3 tool calls | 48210 tokens | ~$0.2100 That last line is your /cost. Token usage arrives on every model response usage metadata , so summing it and multiplying by a price card is all cost tracking is: USD per million tokens input, output — Claude API pricing as of July 2026.PRICES PER MTOK = { "claude-sonnet-4-6": 3.00, 15.00 , "claude-opus-4-8": 5.00, 25.00 , "claude-haiku-4-5": 1.00, 5.00 ,} You don’t have to hand-roll any of this. The reason the tracer above is worth seeing once is that it demystifies the mechanism — but the mechanism is a standard slot, and full observability platforms plug straight into it. You can also go with Langsmith, which is paid by the way. But here, Langfuse https://langfuse.com/ is the one I’d reach for first: because it is an open-source LLM engineering platform you can self-host or use their cloud that gives you the grown-up version of everything AgentTracer does — nested traces of every model and tool call, token counts and cost broken down per trace, per user, per model, dashboards over time — plus prompt management and eval tooling that we'll meet again in comming part. The integration is one more callback handler, riding in the exact slot our tracer does: pip install langfuse v3 SDK; reads LANGFUSE PUBLIC KEY / SECRET KEY / HOST from langfuse.langchain import CallbackHandleragent.invoke {"messages": {"role": "user", "content": "Fix the login tests."} }, {"callbacks": CallbackHandler , tracer }, they compose - it's just a list That composability is the point: callbacks is a list, so ship Langfuse for the dashboards and keep your own in-process handlers for the things a hosted platform can't do — like the kill-switch below, which has to run inside the process to stop anything. LangSmith, LangChain's own hosted tracer, works the same way: set LANGSMITH TRACING=true and every span ships with zero code. A tracer that watches your agent burn $40 in a loop is a smoke detector with no sprinkler. Enforcement needs two brakes, and both live in the harness: A step budget is built into LangGraph. Every agent invoke accepts a recursion limit, and blowing it raises instead of silently spinning: agent.invoke inputs, {"recursion limit": 50} GraphRecursionError beyond this A token budget is the same callback trick pointed at enforcement — accumulate usage in on llm end and raise when it crosses the cap: class TokenBudget BaseCallbackHandler : raise error = True ◀ the load-bearing line def on llm end self, response, kw : self.spent += usage of response if self.spent self.max total tokens: raise BudgetExceededError f"spent {self.spent} of {self.max total tokens}" Here is the trap every agent builder falls into. You tweak the system prompt to fix one bad behavior. You try the one prompt that was failing. It works now Ship it. Three days later you discover the same tweak made the agent stop running tests after edits, and it’s been merging broken code all week. “I tried it and it seemed fine” is not a test suite. But you can’t unit-test an agent the way you unit-test a function, because an agent doesn’t have one output — it has two things worth grading: This is exactly why Anthropic evaluates Claude Code as a harness: the model didn’t change when your system prompt did, but the agent did. Every prompt edit, every new tool description is a behavior change, and behavior changes need regression tests. Graders are tiny frozen dataclasses that look at a transcript. Deterministic ones come first — they’re free, exact, and never drift: case = EvalCase name="fix-login", prompt="Fix the login bug.", graders= ToolWasCalled "write file" , trajectory: it edited ToolWasCalled "run tests" , trajectory: it verified ToolCallOrder "write file", "run tests" , trajectory: in that order ToolWasNotCalled "execute" , trajectory: no raw shell MaxToolCalls 5 , trajectory: under budget FinalAnswerContains "fixed" , outcome: says so FileContains "fix.py", "def login" , outcome: file is right , report = run evals agent factory, case fresh agent per case - no state leaks run evals invokes a fresh agent per case, extracts a transcript tool calls from the messages, final answer, plus the files the agent wrote , and runs every grader over it. The failure mode from the story above is precisely what a trajectory grader catches: the "lazy" agent that edits the file but skips run tests fails tool was called:run tests — and nothing else, so the report tells you exactly what regressed, not just that something did. For qualities you can’t regex — “does the answer explain the root cause?” — add an LLM judge. The judge model is injectable, which means even your judge is testable without an API key: LLMJudge rubric="The answer must explain the root cause.", judge model=init chat model "anthropic:claude-haiku-4-5" judge replies "PASS: ..." or "FAIL: ..." — anything else counts as FAIL, because an unparseable judge should never silently pass. This is also where Langfuse, from oberservability part above, earns a second mention: its eval tooling — datasets of cases, LLM-as-a-judge evaluators, human annotation queues — is the hosted version of exactly these ideas. The concepts map one-to-one, so nothing you build here is throwaway if you later adopt it. The piece that makes all of this pay rent is the regression gate . Check a baseline into the repo, and let CI enforce that prompt changes only ever move the score up: baseline = json.loads Path "evals/baseline.json" .read text {"fix-login": true, ...}report = run evals agent factory, CASES assert no regression report, baseline raises if a passing case now fails Cases that were already failing in the baseline are known issues, not regressions — the gate only screams on pass→fail transitions. That’s the same contract as a snapshot test: improvements update the baseline, regressions block the merge. An agent damages two different things, so it needs two different undo buttons. It damages the conversation. A bad turn — a hallucinated “fact,” a misunderstood instruction — poisons everything the model says after it, because it’s all context now. In Claude Code this is /rewind: jump back to before the bad turn and continue as if it never happened. It damages the workspace. Rewinding the conversation does not un-edit your files. Claude Code handles this with checkpointing — it snapshots files in a shadow git repo before edits, so rewinding can restore your code, not just your chat. We already built this in the previous blog post and didn’t gave the importance it deserved. The checkpointer — the thing we added purely for persistence — doesn’t just save the latest state. It stores a checkpoint at every step , and LangGraph exposes the whole history: checkpoints = list checkpoints agent, config newest first — the rewind menutarget = next c for c in checkpoints if c.is stable and c.num messages == 2 Fork the thread from that point, as if the turns after it never happened:forked = agent.invoke {"messages": {"role": "user", "content": "a different turn two"} }, fork from config, target.checkpoint id , Two properties matter. The fork’s history is rewritten — the abandoned turns are simply not in it, so the model never sees them again. And forking is non-destructive — the original branch stays in the checkpointer, so “undo” never destroys evidence. Here Deep Agents hands you something genuinely elegant. With the default backend, the agent’s “filesystem” is virtual — files live in graph state under the files key. And anything that lives in graph state is captured by every checkpoint. Which means: With the virtual filesystem, rewinding the conversation rewinds the files too. One mechanism, two undos. turn 1: agent writes notes.md = "version one" ◀ checkpoint captures it turn 2: agent edits notes.md to "version two" ◀ newer checkpointforked = agent.invoke ..., fork from config, v1 checkpoint id forked "files" "/notes.md" "content" == "version one". The edit unhappened. That elegance evaporates the moment you switch to LocalShellBackend — the backend from Part 1 that gives the agent a real shell — because now the agent is editing real files, and graph state knows nothing about your disk. For that you need real snapshots — the copy-based version of Claude Code's shadow-git checkpointing: snapshots = WorkspaceSnapshots root="my-project/" snap = snapshots.take "before the agent touches anything" ... agent modifies, deletes, creates files ...snapshots.restore snap modified files revert, deleted files return, created files are removed. Exactly as it was. And when should you take a snapshot? Part 1 already built the perfect moment: the approval gate. interrupt on pauses the loop right before anything destructive runs — the loop is stopped, the disk is quiet, and you know exactly what's about to happen. Snapshot there: result = agent.invoke {"messages": ... }, config if " interrupt " in result: paused at the gate Part 6 snap = snapshots.take f"before {pending actions result }" agent.invoke Command resume={"decisions": {"type": "approve"} } , config regret it later? snapshots.restore snap Safety composes: the gate that lets a human say “no” is the same place the machine quietly makes “yes” reversible. Part 1 ended by assembling every capability into one agent. Part 2 ends by wrapping that agent in everything this post built. The wrapper is thin — under 150 lines — because every hard problem is solved by a part above; this is just the wiring: python from langclaude.production agent import HardenedAgent, pending actionsagent = HardenedAgent "anthropic:claude-sonnet-4-6", root dir="my-project/", max total tokens=200 000, Part 8: the meter has a kill-switch recursion limit=100, Part 8: and a step cap result = agent.invoke "The login tests are failing. Fix them." for action in pending actions result : Part 1's gate, surfaced nicely print f"agent wants to run: {action.tool} {action.args} " result = agent.approve Part 10: snapshots FIRST, then runs ... or agent.reject - nothing runs, so nothing to snapshotprint agent.trace summary Part 8: /cost for your agentagent.rollback workspace Part 10: un-edit the filesagent.rewind checkpoint id, "try again, but" Part 10: un-say the turns Where are the evals? Deliberately not in this class. Evals run in CI against a factory that builds this same agent — they’re a property of your development loop, not your production loop. The eval suite plus the checked-in baseline is what lets you edit SYSTEM PROMPT on a Tuesday without wondering what you broke. What the harness gives you for free: the hooks. Callbacks for every event, token usage on every response, a checkpoint at every step, an interrupt before every gated tool. Everything in this post is arranging those hooks, not fighting them — the tracer, the budget, the eval transcript extractor, and the rewind menu are each well under a hundred lines. What’s still on you the hard 20% : Part 1’s takeaway was that the magic is the harness, not the model. Part 2’s takeaway is the operational sequel: the harness is also where trust comes from. Nothing in this post asked the model to behave better. We wrapped the same loop in a flight recorder so we can see what it did, a meter with a kill-switch so it can’t spend forever, an eval gate so prompt changes can’t silently regress, and two undo buttons so its mistakes are cheap. That’s the difference between a demo and a tool you’d let loose on a codebase you care about. And the pattern is the same one behind Part 1’s golden rule. There it was “a prompt is not a security boundary” — safety has to be enforced in the harness, not requested from the model. The operational sequel: a claim of reliability is not reliability either. Test the recorder, test the budget, test the rollback — don’t assert them. Every mechanism in this post exists in this repo with a test proving it fires, which is exactly how we found the swallowed-callback-exception footgun, the tool-error-kills-the-run behavior, and the write file overwrite guard. The best way to trust an agent is to build the harness that makes trust unnecessary — and watch the tests run. Thank you for reading this post. If you want to tinker with this more, here is the code https://github.com/the-Sreejith/langclaude/tree/part2-evals-observability-rollbacks . And comment if I missed anything worth mentioning. The Loop Was the Easy Part: Evals, Observability, and Rollbacks for Your DIY Claude Code https://pub.towardsai.net/the-loop-was-the-easy-part-evals-observability-and-rollbacks-for-your-diy-claude-code-45f2a9f1c99c was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.