{"slug": "the-loop-was-the-easy-part-evals-observability-and-rollbacks-for-your-diy-claude", "title": "The Loop Was the Easy Part: Evals, Observability, and Rollbacks for Your DIY Claude Code", "summary": "A developer built a DIY Claude Code agent using LangChain's Deep Agents library and then added evaluation harnesses, observability, and rollback capabilities to make it production-ready. The post details how to wrap the agent loop with flight recorders, token budget kill-switches, and checkpoint/snapshot systems for debugging and recovery. This work addresses the gap between a working prototype and a dependable coding agent that can be trusted with real tasks.", "body_md": "I wrote a blog post on *building your own *[ Claude Code using LangChain’s Deep Agents](https://medium.com/towards-artificial-intelligence/build-your-own-claude-code-using-langchin-a-deepdive-into-langchains-deep-agents-9ef98d98a69a) — opening up the harness that turns a language model into a coding agent and rebuilding it piece by piece, with tests proving each piece actually works. After it went out, someone suggested that the parts that really make Claude Code-like agents dependable are the ones\n\nIt was a fair point, and it stuck with me. So I decided to write Part 2.\n\nThis post stands on its own — you don’t need Part 1 to follow it. But you do need the sixty-second version of what we built there:\n\nPart 1 in sixty seconds\n\nWe rebuilt the anatomy of Claude Code on LangChain’s[deepagents]library. One call — create_deep_agent(model=..., tools=..., system_prompt=...) — gives you the agent loop (ask the model what to do next; if it asks for a tool, run it and feed the result back; repeat until it answers in plain text), a built-in tool suite (read/write/edit files, glob, grep, a shell, a to-do planner, subagent delegation), and automatic context management. Two pieces from Part 1 do heavy lifting in this post:\n\nThe approval gate:interrupt_on={\"execute\": True, \"write_file\": True, \"edit_file\": True} pauses the loopbeforeany destructive tool runs and waits for a human to approve or reject.\n\nThe checkpointer:checkpointer=InMemorySaver() saves the agent's full state at every step, which is what made conversations persistent.\n\nEverything else is introduced as we go.\n\nSo: a working coding agent in under a hundred lines. Here’s the uncomfortable part — *“working” was doing a lot of heavy lifting in that sentence.*\n\nHand that agent a real task and watch what happens. It churns for two minutes, makes forty tool calls, edits some files, and announces “Done!” Now answer three questions:\n\nClaude Code has an answer to all three, and none of them live in the model:\n\nPart 1 built the agent. Part 2 makes it operable. Same rules as before: for every capability we’ll look at how Claude Code does it, then build it on Deep Agents — and prove it works with tests.\n\nIn this post\n\nHere’s the map. Nothing in it touches the loop from Part 1; it all wraps around it:\n\n```\n[ the eval harness — Part 9, runs in CI, not in prod ]         scripted scenarios ──▶ grade trajectory + outcome         compare vs. baseline ──▶ block regressions before merge                                 │                       exercises a copy of...                                 ▼┌───────────────────────────────────────────────────────────────────┐│ flight recorder + meter (Part 8)                                  ││  ┌─────────────────────────────────────────────────────────────┐  ││  │           the agent loop from Part 1 (unchanged)            │  ││  │                                                             │  ││  │  user ──▶ [ model decides ] ──▶ tool call? ──yes──▶ PAUSE   │  ││  │              ▲                      │        (approval gate │  ││  │              └──── result fed back ─┤       + snapshot the  │  ││  │                                     no        workspace)    │  ││  │                                     ▼                       │  ││  │                             plain-text answer               │  ││  └─────────────────────────────────────────────────────────────┘  ││  every model + tool call recorded · token budget kill-switch      │└───────────────────────────────────────────────────────────────────┘┌───────────────────────────────────────────────────────────────────┐│ checkpoints + snapshots (Part 10): rewind the conversation,       ││ restore the workspace — two undo buttons for two kinds of damage  │└───────────────────────────────────────────────────────────────────┘\n```\n\nYou cannot debug what you cannot see, and an agent gives you a lot not to see. A chatbot has one input and one output. An agent has one input, one output, and an arbitrarily long *middle* — the tool calls, the intermediate model decisions, the tokens each step consumed. When the answer is wrong, the bug is almost always in the middle.\n\nThe temptation is to add print() statements to your tools. Don't. The harness already emits an event for everything it does; you just have to listen. In LangChain, listening is a **callback handler** — an object you pass at invoke time that gets told about every model call (with token usage attached) and every tool call (name, args, output). Crucially, this requires *zero changes to the agent*. Observability is a property of the harness, not the prompt.\n\nThe whole tracer is one class. Record a span when something starts, close it when it ends, keep the results immutable:\n\n``` python\nfrom langchain_core.callbacks import BaseCallbackHandlerclass AgentTracer(BaseCallbackHandler):    \"\"\"Records a span for every model call and tool call in a run.\"\"\"    def on_chat_model_start(self, serialized, messages, *, run_id, **kw):        self._starts[run_id] = time.monotonic()    def on_llm_end(self, response, *, run_id, **kw):        duration = time.monotonic() - self._starts.pop(run_id)        message = response.generations[0][0].message        usage = message.usage_metadata or {}          # input/output tokens        self.model_spans.append(ModelSpan(duration, usage.get(\"input_tokens\", 0),                                          usage.get(\"output_tokens\", 0)))    def on_tool_start(self, serialized, input_str, *, run_id, **kw): ...    def on_tool_end(self, output, *, run_id, **kw): ...    def on_tool_error(self, error, *, run_id, **kw): ...\n```\n\nAttach it when you invoke — this is the part people miss; it goes in the *config*, not the agent:\n\n```\ntracer = AgentTracer(model_name=\"claude-sonnet-4-6\")result = agent.invoke(    {\"messages\": [{\"role\": \"user\", \"content\": \"Fix the login tests.\"}]},    {\"callbacks\": [tracer]},          # ◀ the flight recorder rides along)print(format_trace(tracer))\nphp\ntool  read_file({'file_path': 'auth/login.py'})  ->  def login(user, ...tool  edit_file({'file_path': 'auth/login.py', ...})  ->  Updated filetool  run_tests({'path': '.'})  ->  3 passed in 0.41sTOTAL  4 model calls | 3 tool calls | 48210 tokens | ~$0.2100\n```\n\nThat last line is your /cost. Token usage arrives on every model response (usage_metadata), so summing it and multiplying by a price card is all cost tracking is:\n\n```\n# USD per million tokens (input, output) — Claude API pricing as of July 2026.PRICES_PER_MTOK = {    \"claude-sonnet-4-6\": (3.00, 15.00),    \"claude-opus-4-8\": (5.00, 25.00),    \"claude-haiku-4-5\": (1.00, 5.00),}\n```\n\nYou don’t have to hand-roll any of this. The reason the tracer above is worth seeing once is that it demystifies the mechanism — but the mechanism is a standard slot, and full observability platforms plug straight into it.\n\nYou can also go with Langsmith, which is paid by the way. But here, [Langfuse](https://langfuse.com/) is the one I’d reach for first: because it is an open-source LLM engineering platform you can self-host (or use their cloud) that gives you the grown-up version of everything AgentTracer does — nested traces of every model and tool call, token counts and cost broken down per trace, per user, per model, dashboards over time — plus prompt management and eval tooling that we'll meet again in comming part. The integration is one more callback handler, riding in the exact slot our tracer does:\n\n```\n# pip install langfuse   (v3 SDK; reads LANGFUSE_PUBLIC_KEY / _SECRET_KEY / _HOST)from langfuse.langchain import CallbackHandleragent.invoke(    {\"messages\": [{\"role\": \"user\", \"content\": \"Fix the login tests.\"}]},    {\"callbacks\": [CallbackHandler(), tracer]},   # they compose - it's just a list)\n```\n\nThat composability is the point: callbacks is a list, so ship Langfuse for the dashboards and keep your own in-process handlers for the things a hosted platform can't do — like the kill-switch below, which has to run *inside* the process to stop anything. (LangSmith, LangChain's own hosted tracer, works the same way: set LANGSMITH_TRACING=true and every span ships with zero code.)\n\nA tracer that watches your agent burn $40 in a loop is a smoke detector with no sprinkler. Enforcement needs two brakes, and both live in the harness:\n\n**A step budget** is built into LangGraph. Every agent invoke accepts a recursion_limit, and blowing it raises instead of silently spinning:\n\n```\nagent.invoke(inputs, {\"recursion_limit\": 50})   # GraphRecursionError beyond this\n```\n\n**A token budget** is the same callback trick pointed at enforcement — accumulate usage in on_llm_end and raise when it crosses the cap:\n\n```\nclass TokenBudget(BaseCallbackHandler):    raise_error = True          # ◀ the load-bearing line    def on_llm_end(self, response, **kw):        self.spent += usage_of(response)        if self.spent > self.max_total_tokens:            raise BudgetExceededError(f\"spent {self.spent} of {self.max_total_tokens}\")\n```\n\nHere is the trap every agent builder falls into. You tweak the system prompt to fix one bad behavior. You try the one prompt that was failing. It works now! Ship it. Three days later you discover the same tweak made the agent stop running tests after edits, and it’s been merging broken code all week.\n\n“I tried it and it seemed fine” is not a test suite. But you can’t unit-test an agent the way you unit-test a function, because an agent doesn’t have one output — it has two things worth grading:\n\nThis is exactly why Anthropic evaluates Claude Code as a harness: the model didn’t change when your system prompt did, but the *agent* did. Every prompt edit, every new tool description is a behavior change, and behavior changes need regression tests.\n\nGraders are tiny frozen dataclasses that look at a transcript. Deterministic ones come first — they’re free, exact, and never drift:\n\n```\ncase = EvalCase(    name=\"fix-login\",    prompt=\"Fix the login bug.\",    graders=(        ToolWasCalled(\"write_file\"),                  # trajectory: it edited        ToolWasCalled(\"run_tests\"),                   # trajectory: it verified        ToolCallOrder((\"write_file\", \"run_tests\")),   # trajectory: in that order        ToolWasNotCalled(\"execute\"),                  # trajectory: no raw shell        MaxToolCalls(5),                              # trajectory: under budget        FinalAnswerContains(\"fixed\"),                 # outcome: says so        FileContains(\"fix.py\", \"def login\"),          # outcome: file is right    ),)report = run_evals(agent_factory, [case])   # fresh agent per case - no state leaks\n```\n\nrun_evals invokes a *fresh* agent per case, extracts a transcript (tool calls from the messages, final answer, plus the files the agent wrote), and runs every grader over it. The failure mode from the story above is precisely what a trajectory grader catches: the \"lazy\" agent that edits the file but skips run_tests fails tool_was_called:run_tests — and nothing else, so the report tells you exactly *what* regressed, not just that something did.\n\nFor qualities you can’t regex — “does the answer explain the root cause?” — add an LLM judge. The judge model is injectable, which means even your judge is testable without an API key:\n\n```\nLLMJudge(rubric=\"The answer must explain the root cause.\",         judge_model=init_chat_model(\"anthropic:claude-haiku-4-5\"))# judge replies \"PASS: ...\" or \"FAIL: ...\" — anything else counts as FAIL,# because an unparseable judge should never silently pass.\n```\n\n(This is also where Langfuse, from oberservability part above, earns a second mention: its eval tooling — datasets of cases, LLM-as-a-judge evaluators, human annotation queues — is the hosted version of exactly these ideas. The concepts map one-to-one, so nothing you build here is throwaway if you later adopt it.)\n\nThe piece that makes all of this pay rent is the **regression gate**. Check a baseline into the repo, and let CI enforce that prompt changes only ever move the score up:\n\n```\nbaseline = json.loads(Path(\"evals/baseline.json\").read_text())  # {\"fix-login\": true, ...}report = run_evals(agent_factory, CASES)assert_no_regression(report, baseline)   # raises if a passing case now fails\n```\n\nCases that were already failing in the baseline are known issues, not regressions — the gate only screams on pass→fail transitions. That’s the same contract as a snapshot test: improvements update the baseline, regressions block the merge.\n\nAn agent damages two different things, so it needs two different undo buttons.\n\n**It damages the conversation.** A bad turn — a hallucinated “fact,” a misunderstood instruction — poisons everything the model says after it, because it’s all context now. In Claude Code this is /rewind: jump back to before the bad turn and continue as if it never happened.\n\n**It damages the workspace.** Rewinding the conversation does *not* un-edit your files. Claude Code handles this with checkpointing — it snapshots files (in a shadow git repo) before edits, so rewinding can restore your code, not just your chat.\n\nWe already built this in the previous blog post and didn’t gave the importance it deserved. The checkpointer — the thing we added purely for *persistence* — doesn’t just save the latest state. It stores a checkpoint at *every step*, and LangGraph exposes the whole history:\n\n```\ncheckpoints = list_checkpoints(agent, config)   # newest first — the rewind menutarget = next(c for c in checkpoints if c.is_stable and c.num_messages == 2)# Fork the thread from that point, as if the turns after it never happened:forked = agent.invoke(    {\"messages\": [{\"role\": \"user\", \"content\": \"a different turn two\"}]},    fork_from(config, target.checkpoint_id),)\n```\n\nTwo properties matter. The fork’s history is *rewritten* — the abandoned turns are simply not in it, so the model never sees them again. And forking is *non-destructive* — the original branch stays in the checkpointer, so “undo” never destroys evidence.\n\nHere Deep Agents hands you something genuinely elegant. With the default backend, the agent’s “filesystem” is *virtual* — files live in graph state under the files key. And anything that lives in graph state is captured by every checkpoint. Which means:\n\nWith the virtual filesystem, rewinding the conversation rewinds the files too. One mechanism, two undos.\n\n```\n# turn 1: agent writes notes.md = \"version one\"    ◀ checkpoint captures it# turn 2: agent edits notes.md to \"version two\"    ◀ newer checkpointforked = agent.invoke(..., fork_from(config, v1_checkpoint_id))forked[\"files\"][\"/notes.md\"][\"content\"]   # ==> \"version one\". The edit unhappened.\n```\n\nThat elegance evaporates the moment you switch to LocalShellBackend — the backend from Part 1 that gives the agent a real shell — because now the agent is editing *real* files, and graph state knows nothing about your disk. For that you need real snapshots — the copy-based version of Claude Code's shadow-git checkpointing:\n\n```\nsnapshots = WorkspaceSnapshots(root=\"my-project/\")snap = snapshots.take(\"before the agent touches anything\")# ... agent modifies, deletes, creates files ...snapshots.restore(snap)   # modified files revert, deleted files return,                          # created files are removed. Exactly as it was.\n```\n\nAnd *when* should you take a snapshot? Part 1 already built the perfect moment: the approval gate. interrupt_on pauses the loop right before anything destructive runs — the loop is stopped, the disk is quiet, and you know exactly what's about to happen. Snapshot there:\n\n```\nresult = agent.invoke({\"messages\": [...]}, config)if \"__interrupt__\" in result:                       # paused at the gate (Part 6)    snap = snapshots.take(f\"before {pending_actions(result)}\")    agent.invoke(Command(resume={\"decisions\": [{\"type\": \"approve\"}]}), config)    # regret it later?    snapshots.restore(snap)\n```\n\nSafety composes: the gate that lets a human say “no” is the same place the machine quietly makes “yes” reversible.\n\nPart 1 ended by assembling every capability into one agent. Part 2 ends by wrapping that agent in everything this post built. The wrapper is thin — under 150 lines — because every hard problem is solved by a part above; this is just the wiring:\n\n``` python\nfrom langclaude.production_agent import HardenedAgent, pending_actionsagent = HardenedAgent(    \"anthropic:claude-sonnet-4-6\",    root_dir=\"my-project/\",    max_total_tokens=200_000,     # Part 8: the meter has a kill-switch    recursion_limit=100,          # Part 8: and a step cap)result = agent.invoke(\"The login tests are failing. Fix them.\")for action in pending_actions(result):        # Part 1's gate, surfaced nicely    print(f\"agent wants to run: {action.tool}({action.args})\")result = agent.approve()                      # Part 10: snapshots FIRST, then runs# ... or agent.reject() - nothing runs, so nothing to snapshotprint(agent.trace_summary())                  # Part 8: /cost for your agentagent.rollback_workspace()                    # Part 10: un-edit the filesagent.rewind(checkpoint_id, \"try again, but\") # Part 10: un-say the turns\n```\n\nWhere are the evals? Deliberately *not* in this class. Evals run in CI against a factory that builds this same agent — they’re a property of your development loop, not your production loop. The eval suite plus the checked-in baseline is what lets you edit SYSTEM_PROMPT on a Tuesday without wondering what you broke.\n\n**What the harness gives you for free:** the hooks. Callbacks for every event, token usage on every response, a checkpoint at every step, an interrupt before every gated tool. Everything in this post is arranging those hooks, not fighting them — the tracer, the budget, the eval transcript extractor, and the rewind menu are each well under a hundred lines.\n\n**What’s still on you (the hard 20%):**\n\nPart 1’s takeaway was that the magic is the harness, not the model. Part 2’s takeaway is the operational sequel: **the harness is also where trust comes from.** Nothing in this post asked the model to behave better. We wrapped the same loop in a flight recorder so we can see what it did, a meter with a kill-switch so it can’t spend forever, an eval gate so prompt changes can’t silently regress, and two undo buttons so its mistakes are cheap.\n\nThat’s the difference between a demo and a tool you’d let loose on a codebase you care about. And the pattern is the same one behind Part 1’s golden rule. There it was “a prompt is not a security boundary” — safety has to be enforced in the harness, not requested from the model. The operational sequel: a *claim* of reliability is not reliability either. Test the recorder, test the budget, test the rollback — don’t assert them. Every mechanism in this post exists in this repo with a test proving it fires, which is exactly how we found the swallowed-callback-exception footgun, the tool-error-kills-the-run behavior, and the write_file overwrite guard.\n\nThe best way to trust an agent is to build the harness that makes trust unnecessary — and watch the tests run.\n\nThank you for reading this post. If you want to tinker with this more, [here is the code](https://github.com/the-Sreejith/langclaude/tree/part2-evals-observability-rollbacks). And comment if I missed anything worth mentioning.\n\n[The Loop Was the Easy Part: Evals, Observability, and Rollbacks for Your DIY Claude Code](https://pub.towardsai.net/the-loop-was-the-easy-part-evals-observability-and-rollbacks-for-your-diy-claude-code-45f2a9f1c99c) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/the-loop-was-the-easy-part-evals-observability-and-rollbacks-for-your-diy-claude", "canonical_source": "https://pub.towardsai.net/the-loop-was-the-easy-part-evals-observability-and-rollbacks-for-your-diy-claude-code-45f2a9f1c99c?source=rss----98111c9905da---4", "published_at": "2026-07-18 05:46:19+00:00", "updated_at": "2026-07-18 05:56:21.042591+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "developer-tools", "ai-agents", "large-language-models"], "entities": ["Claude Code", "LangChain", "Deep Agents", "InMemorySaver"], "alternates": {"html": "https://wpnews.pro/news/the-loop-was-the-easy-part-evals-observability-and-rollbacks-for-your-diy-claude", "markdown": "https://wpnews.pro/news/the-loop-was-the-easy-part-evals-observability-and-rollbacks-for-your-diy-claude.md", "text": "https://wpnews.pro/news/the-loop-was-the-easy-part-evals-observability-and-rollbacks-for-your-diy-claude.txt", "jsonld": "https://wpnews.pro/news/the-loop-was-the-easy-part-evals-observability-and-rollbacks-for-your-diy-claude.jsonld"}}