# Every Agent Session Is a Test Run > Source: > Published: 2026-09-16 17:53:05+00:00 ## The short version Every morning at 05:00, a scheduled job opens the transcripts of everything my AI coding agent did in the last 24 hours. It leaves me a list of proposed edits to the skills it used. Not a summary of what I did — a list, each item citing the exact line of the transcript and the exact line of the file that caused the problem. I built it on top of [xskills](https://github.com/lleqsnoom/xskills), my collection of agent skills. It is called `x-skills-daily-reflection`. It turned my skill set from “a folder of instructions I wrote once” into “a tool that gets measurably less wrong every week.” ## The problem: skills rot silently In [the last post](https://mielony.com/blog/build-your-own-agent-skillset) I argued for building your own agent skillset. That argument has a hidden bill, and I named it there: maintenance is on you, forever. Here is what maintenance looks like. A skill is a markdown file of instructions. When it is wrong, nothing crashes: no stack trace, no failed test, no red build. What happens instead is worse: the agent reads a stale instruction, stumbles, improvises, and *succeeds anyway*. The work gets done. The only trace is a slightly longer session, a repeated tool call, a correction you typed mid-flow and forgot about two minutes later. That trace lives in a place nobody looks: the session transcript. Every conversation your agent has is a test run of the skills it used, and every transcript is a test report that gets thrown away. People solved this for code decades ago. CI runs your tests on every push. What is missing is the equivalent for *instructions*: a job that treats your own agent sessions as the test suite. Your skill files are the code under test. ## The loop in one picture ``` php flowchart TD A["yesterday's sessions"] --> B["scan for friction"] B --> C["verified proposals, never edits"] C --> D{"your five minutes"} D -- "accept" --> E["the skill improves"] E -. "tomorrow" .-> A D -- "defer" --> C ``` Three pieces make that loop run, and none of them are clever: ``` automation/daily-reflection/ ├── collect-sessions.mjs # gather last 24h of sessions, scan for friction ├── precheck.sh # fail closed: skip the run when there is nothing honest to do └── runbook.md # the agent's instructions for the headless run ``` **The collector** walks every project directory my agent CLI (Crush) has touched recently. It exports the sessions modified in the last 24 hours and runs the `x-autoreflection` scanner over each one. The scanner extracts mechanical signals from each session: failed commands, repeated tool calls, moments I corrected the agent, skills loaded but never used, questions asked in prose instead of a structured prompt. Each signal keeps a severity, a suspect skill, and quoted evidence. **The precheck** runs before the agent is invoked and fails closed. It skips the day when a binary is missing, when `skills/` has uncommitted edits, or when no session in the window touched a skill. That git check is the whole philosophy in four lines: every proposal cites a skill by `file:line`, and uncommitted edits move those lines. A reflection that runs against a moving target produces plausible garbage. Plausible garbage is the one output you cannot afford from a system whose entire job is to be trustworthy. **The runbook** is what the headless agent follows. It is plain markdown, but written for an agent that cannot ask questions: - **At most 4 sessions, at most 10 proposals.** Caps, not targets. Without a human in the loop, an agent optimizing for “good retro” will find*something* to say about everything. A quiet day produces a three-line digest. The runbook says: do not invent findings to fill it. - **Verify every signal against the real file.** A signal is a lead, not a finding. The agent opens the actual`SKILL.md` and decides.*Keep* : the file is wrong.*Re-grade* : real friction, but not that skill’s fault.*Drop* : the transcript misled the scanner. A`grep` exiting 1 because it found nothing is the answer, not a failure. Dropping is recorded, so tomorrow’s run does not re-litigate it. - **Blame the instruction, not the agent.** “The`SKILL.md` did not say” is a gap. “I forgot” is not. - **Every proposal has Signal, Target, Change, and Check.** The Check must be a command that runs: a test, a lint rule, a script that exits 0 once the fix is in. A proposal you cannot check is a wish. And then the rule that defines the whole system, printed twice in the runbook: **Never edit anything under `skills/`.** Propose, then stop. The review decides. ## Why the loop does not close itself The obvious design is a fully closed loop: cron fires, agent reflects, agent edits the skill, agent commits. It would take an afternoon to build, and it would be the wrong system. Not because agents edit files badly — they are fine at it. Because a self-editing instruction set has no audit trail and no stopping condition. If today’s run misreads a signal and “fixes” a skill into uselessness, tomorrow’s run reflects on sessions shaped by that bad edit. The drift compounds. The precheck already knows this, which is why it refuses to run against a dirty tree. Skip the review, and you are not editing a file; you are editing the audit trail. So the output is a `DIGEST.md` with proposals, and the output of my morning coffee is five minutes of checkboxes: accept, defer, drop. Accepted proposals route by size. A one-line edit goes in directly, a cluster goes through `x-fix`, and a design decision goes to `x-plan` with the reflection attached. The automation read 40 sessions. It handed me three verified, checkable changes instead of a vague feeling that something was off. This is the same shape as a deploy pipeline. The pipeline builds, tests, and stages everything automatically — but something with judgement presses the button. ## What it found, honestly The loop is honest about its limits. Mechanical scanning only sees friction that leaves a trace. A wrong-but-successful instruction produces no failed command to flag, which is why the runbook lets the agent log `manual` findings it noticed by reading. And it costs one small agent run per day — on a quiet day, the precheck skips even that. Most days are the three-line digest. But the pattern of what surfaces is what you cannot see from inside: - **The same gap twice is a defect.** A missing flag in one session is a hypothesis. The same command failing in three sessions is a bug with a patch. - **Loaded but unused skills are a smell.** If the agent keeps loading a skill and abandoning it, the description promises the wrong thing or the body answers the wrong question. - **Documentation drift is catchable.** A documented command that no longer runs leaves one trace: the agent tries it, fails, and improvises. Hours later, that trace is in a transcript. Without the loop, you find it months later, when you run the command yourself. ## Build the minimal version You do not need Orca, a skill framework, or any of my code. You need a place where your agent’s conversations are stored, a scheduler, and the agent’s headless mode. ``` # crontab: 5 5 * * * ~/reflection/run.sh agent sessions export --since 24h --out /tmp/reflection/sessions/ # collect agent run --headless --runbook ~/reflection/runbook.md \ --context /tmp/reflection/sessions/ # reflect ``` And a first runbook can be five sentences: 1. Read every transcript in the folder. 2. List every moment the user corrected you, retried a command, or worked around a missing capability. 3. For each, check whether an instruction file could have prevented it. 4. For each that could, write one proposal: which file, what change, what command proves it. 5. Output the list to `DIGEST.md` . Edit nothing. The first week it will mostly produce noise, and the noise is informative — it shows you which signals to filter. Mine learned to stop flagging `grep`’s exit codes after I made “drop the signal, record why” part of the procedure. ## The part people get wrong Two failure modes, both of which I built first. **The loop that must find something.** You schedule a daily reflection. For three days it says “nothing to report.” The temptation is to loosen the criteria until something appears. Do not. A reflection that never comes back empty is not reflecting. It is generating content. **Closing the loop to save five minutes.** The human review feels like the weakest link, so it is the part everyone automates away first. It is the load-bearing part. Five minutes of checkboxes a day is the price of an instruction set that improves in one direction. ## The short version, again Your agent’s conversations are already a test suite for your workflows — the only one that runs on real work instead of synthetic benchmarks. A cron job, a collector script, and a runbook are enough to stop throwing that test suite away. The system does not improve itself. It improves *your* half-hour a week into a targeted, evidenced, checkable five minutes. Every accepted proposal makes the next week a little smoother, and the next digest a little shorter. Most mornings it is three lines. That is the system telling you it worked.