{"slug": "every-agent-session-is-a-test-run", "title": "Every Agent Session Is a Test Run", "summary": "A developer built x-skills-daily-reflection, a scheduled job that runs at 05:00 daily on top of the xskills agent-skill collection to scan the last 24 hours of AI coding agent sessions for friction and propose edits to the skills used. The system uses three components — collect-sessions.mjs, precheck.sh, and runbook.md — and caps each run at 4 sessions and 10 proposals, with every proposal citing a skill by file:line and verified against the actual SKILL.md before being surfaced. The precheck fails closed, skipping the run when a binary is missing, skills/ has uncommitted edits, or no session touched a skill, because uncommitted edits shift line numbers and produce what the author calls plausible garbage.", "body_md": "## The short version\n\nEvery morning at 05:00, a scheduled job opens the transcripts of everything my AI coding agent did in the last 24 hours. It leaves me a list of proposed edits to the skills it used. Not a summary of what I did — a list, each item citing the exact line of the transcript and the exact line of the file that caused the problem.\n\nI built it on top of [xskills](https://github.com/lleqsnoom/xskills), my collection of agent skills. It is called `x-skills-daily-reflection`. It turned my skill set from “a folder of instructions I wrote once” into “a tool that gets measurably less wrong every week.”\n\n## The problem: skills rot silently\n\nIn [the last post](https://mielony.com/blog/build-your-own-agent-skillset) I argued for building your own agent skillset. That argument has a hidden bill, and I named it there: maintenance is on you, forever.\n\nHere is what maintenance looks like. A skill is a markdown file of instructions. When it is wrong, nothing crashes: no stack trace, no failed test, no red build. What happens instead is worse: the agent reads a stale instruction, stumbles, improvises, and *succeeds anyway*. The work gets done. The only trace is a slightly longer session, a repeated tool call, a correction you typed mid-flow and forgot about two minutes later.\n\nThat trace lives in a place nobody looks: the session transcript. Every conversation your agent has is a test run of the skills it used, and every transcript is a test report that gets thrown away.\n\nPeople solved this for code decades ago. CI runs your tests on every push. What is missing is the equivalent for *instructions*: a job that treats your own agent sessions as the test suite. Your skill files are the code under test.\n\n## The loop in one picture\n\n``` php\nflowchart TD\n    A[\"yesterday's sessions\"] --> B[\"scan for friction\"]\n    B --> C[\"verified proposals, never edits\"]\n    C --> D{\"your five minutes\"}\n    D -- \"accept\" --> E[\"the skill improves\"]\n    E -. \"tomorrow\" .-> A\n    D -- \"defer\" --> C\n```\n\nThree pieces make that loop run, and none of them are clever:\n\n```\nautomation/daily-reflection/\n├── collect-sessions.mjs   # gather last 24h of sessions, scan for friction\n├── precheck.sh            # fail closed: skip the run when there is nothing honest to do\n└── runbook.md             # the agent's instructions for the headless run\n```\n\n**The collector** walks every project directory my agent CLI (Crush) has touched recently. It exports the sessions modified in the last 24 hours and runs the `x-autoreflection` scanner over each one. The scanner extracts mechanical signals from each session: failed commands, repeated tool calls, moments I corrected the agent, skills loaded but never used, questions asked in prose instead of a structured prompt. Each signal keeps a severity, a suspect skill, and quoted evidence.\n\n**The precheck** runs before the agent is invoked and fails closed. It skips the day when a binary is missing, when `skills/` has uncommitted edits, or when no session in the window touched a skill. That git check is the whole philosophy in four lines: every proposal cites a skill by `file:line`, and uncommitted edits move those lines. A reflection that runs against a moving target produces plausible garbage. Plausible garbage is the one output you cannot afford from a system whose entire job is to be trustworthy.\n\n**The runbook** is what the headless agent follows. It is plain markdown, but written for an agent that cannot ask questions:\n\n- **At most 4 sessions, at most 10 proposals.** Caps, not targets. Without a human in the loop, an agent optimizing for “good retro” will find*something* to say about everything. A quiet day produces a three-line digest. The runbook says: do not invent findings to fill it.\n- **Verify every signal against the real file.** A signal is a lead, not a finding. The agent opens the actual`SKILL.md` and decides.*Keep* : the file is wrong.*Re-grade* : real friction, but not that skill’s fault.*Drop* : the transcript misled the scanner. A`grep` exiting 1 because it found nothing is the answer, not a failure. Dropping is recorded, so tomorrow’s run does not re-litigate it.\n- **Blame the instruction, not the agent.** “The`SKILL.md` did not say” is a gap. “I forgot” is not.\n- **Every proposal has Signal, Target, Change, and Check.** The Check must be a command that runs: a test, a lint rule, a script that exits 0 once the fix is in. A proposal you cannot check is a wish.\n\nAnd then the rule that defines the whole system, printed twice in the runbook:\n\n**Never edit anything under `skills/`.** Propose, then stop. The review decides.\n\n## Why the loop does not close itself\n\nThe obvious design is a fully closed loop: cron fires, agent reflects, agent edits the skill, agent commits. It would take an afternoon to build, and it would be the wrong system.\n\nNot because agents edit files badly — they are fine at it. Because a self-editing instruction set has no audit trail and no stopping condition. If today’s run misreads a signal and “fixes” a skill into uselessness, tomorrow’s run reflects on sessions shaped by that bad edit. The drift compounds. The precheck already knows this, which is why it refuses to run against a dirty tree. Skip the review, and you are not editing a file; you are editing the audit trail.\n\nSo the output is a `DIGEST.md` with proposals, and the output of my morning coffee is five minutes of checkboxes: accept, defer, drop. Accepted proposals route by size. A one-line edit goes in directly, a cluster goes through `x-fix`, and a design decision goes to `x-plan` with the reflection attached. The automation read 40 sessions. It handed me three verified, checkable changes instead of a vague feeling that something was off.\n\nThis is the same shape as a deploy pipeline. The pipeline builds, tests, and stages everything automatically — but something with judgement presses the button.\n\n## What it found, honestly\n\nThe loop is honest about its limits. Mechanical scanning only sees friction that leaves a trace. A wrong-but-successful instruction produces no failed command to flag, which is why the runbook lets the agent log `manual` findings it noticed by reading. And it costs one small agent run per day — on a quiet day, the precheck skips even that.\n\nMost days are the three-line digest. But the pattern of what surfaces is what you cannot see from inside:\n\n- **The same gap twice is a defect.** A missing flag in one session is a hypothesis. The same command failing in three sessions is a bug with a patch.\n- **Loaded but unused skills are a smell.** If the agent keeps loading a skill and abandoning it, the description promises the wrong thing or the body answers the wrong question.\n- **Documentation drift is catchable.** A documented command that no longer runs leaves one trace: the agent tries it, fails, and improvises. Hours later, that trace is in a transcript. Without the loop, you find it months later, when you run the command yourself.\n\n## Build the minimal version\n\nYou do not need Orca, a skill framework, or any of my code. You need a place where your agent’s conversations are stored, a scheduler, and the agent’s headless mode.\n\n```\n# crontab: 5 5 * * *  ~/reflection/run.sh\n\nagent sessions export --since 24h --out /tmp/reflection/sessions/   # collect\nagent run --headless --runbook ~/reflection/runbook.md \\\n  --context /tmp/reflection/sessions/                               # reflect\n```\n\nAnd a first runbook can be five sentences:\n\n1. Read every transcript in the folder.\n2. List every moment the user corrected you, retried a command, or worked around a missing capability.\n3. For each, check whether an instruction file could have prevented it.\n4. For each that could, write one proposal: which file, what change, what command proves it.\n5. Output the list to `DIGEST.md` . Edit nothing.\n\nThe first week it will mostly produce noise, and the noise is informative — it shows you which signals to filter. Mine learned to stop flagging `grep`’s exit codes after I made “drop the signal, record why” part of the procedure.\n\n## The part people get wrong\n\nTwo failure modes, both of which I built first.\n\n**The loop that must find something.** You schedule a daily reflection. For three days it says “nothing to report.” The temptation is to loosen the criteria until something appears. Do not. A reflection that never comes back empty is not reflecting. It is generating content.\n\n**Closing the loop to save five minutes.** The human review feels like the weakest link, so it is the part everyone automates away first. It is the load-bearing part. Five minutes of checkboxes a day is the price of an instruction set that improves in one direction.\n\n## The short version, again\n\nYour agent’s conversations are already a test suite for your workflows — the only one that runs on real work instead of synthetic benchmarks. A cron job, a collector script, and a runbook are enough to stop throwing that test suite away.\n\nThe system does not improve itself. It improves *your* half-hour a week into a targeted, evidenced, checkable five minutes. Every accepted proposal makes the next week a little smoother, and the next digest a little shorter. Most mornings it is three lines. That is the system telling you it worked.", "url": "https://wpnews.pro/news/every-agent-session-is-a-test-run", "canonical_source": "https://mielony.com/blog/self-improvement-by-cron/", "published_at": "2026-09-16 17:53:05+00:00", "updated_at": "2026-09-16 18:15:48.929823+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-products"], "entities": ["xskills", "x-skills-daily-reflection", "Crush", "x-autoreflection", "collect-sessions.mjs"], "alternates": {"html": "https://wpnews.pro/news/every-agent-session-is-a-test-run", "markdown": "https://wpnews.pro/news/every-agent-session-is-a-test-run.md", "text": "https://wpnews.pro/news/every-agent-session-is-a-test-run.txt", "jsonld": "https://wpnews.pro/news/every-agent-session-is-a-test-run.jsonld"}}