I didn't set out to build a developer tool. For a long time now I've been working with AI on everything in my life — daily conversations about my daughters, planning projects, ideas for ones that don't exist yet. The goal was always the same: ease my life, get more done, and break the barrier between human and AI — stop treating it as a search box, start treating it as a partner.
Somewhere along the way, the partnership got serious. The workspace where my projects live grew an instruction system for AI coding agents — the files everyone is writing now: AGENTS.md
, CLAUDE.md
, a skills directory, rules for how agents should plan, log, and verify their work.
Then I asked an uncomfortable question: is any of it actually followed?
Not "do the agents seem to follow it." Could anyone tell, from the repository alone, whether an instruction was followed? For most of my rules, the answer was no. My own audit found that the two checks my instructions said must run before every commit were invoked by nothing — no CI, no hook, no scheduled task. The rule had been enforced, for its entire life, by whoever remembered. Replaying my last 200 commits, the index-freshness rule alone would have failed on 29 of 61 eligible commits — roughly half. My instructions were not rules. They were hopes with formatting.
So I wondered whether everyone else's are too. I wrote a tool and measured.
I analysed eight public agent-instruction collections — 1,332 instruction units, 17,611 individual instructions — each at a pinned commit SHA, with the raw per-repo JSON published alongside the tool.
An instruction counts as CHECKABLE if a reviewer could tell from the repo whether it happened: it's a tick-box, or contains a runnable command, or names a concrete file artifact, or refers to an exit code, a diff, an assertion. Everything else is CLAIMABLE — the only evidence it happened is the agent saying so.
Two limits, before any number, because they bound everything below:
First: checkability is not quality. Nobody — including me — has demonstrated that low-checkability instructions produce worse outcomes. The demonstrated finding is narrower and, I think, more important: nobody can tell either way. That is the weaker claim, the more defensible one, and the only one worth acting on.
Second: the metric is deliberately biased upward. Every ambiguous instruction counts as checkable — a bare filename mention counts, a bare command word counts. Every figure below is a ceiling. The true numbers are lower. And the metric reads only documents: one surveyed project ships a separate eval harness that drives real agents and grades them — real enforceability this method cannot see. Document analysis has a floor it cannot measure below.
Across the eight collections, the median proportion of procedural instructions that are checkable is 11.1%. The range: 2.2% to 22.9%. The best collection in the sample still leaves three-quarters of its procedural instructions unverifiable; the weakest leaves 98%.
Two findings surprised me more than the median.
Five of the eight collections mandate zero output artifacts. Not "promised and missing" — never promised anything at all. No file their instructions require to exist. Nothing that survives the session. Nothing a reviewer could ever audit.
The other three do promise artifacts — and of the 103 distinct artifacts they mandate, 44 have never existed in any commit of their repositories. Forty-three percent. I call these phantom artifacts: an instruction no model has ever been caught skipping, because nothing was ever looking. (The check walks full git history, not just the current tree — an artifact that existed once and was deleted is not a phantom.)
So the field splits into two failure modes: collections that make no verifiable claims at all, and collections whose claims fail at roughly the rate of a coin flip weighted for optimism.
One genre note, because scoring everything on checkability would be the obvious way to get this wrong: instructions about how to think — "name the assumption before building" — produce judgment, not files. They score ~0% checkable, and that is the genre working, not a defect. The headline figures are procedure-only for exactly that reason.
The measurement tool is kibsu — open source, zero dependencies, Python 3.8+, and it writes nothing to the repositories it reads:
pip install kibsu
python -m kibsu report /path/to/any/repo
It ends every report with "Nothing was written to this repo — run git status to confirm", and that line is the design philosophy in one sentence. The survey's raw evidence, with pinned SHAs per repository, is in the repo:
The first repository it publicly diagnosed was its own, and the verdict was "0 of 5 ready." It launched by telling the truth about itself. I intend to keep it that way.
My own workspace's instruction system measures 28.65% procedure-only checkable — about 2.6× the public median, and still failing seven of ten instructions. The plan is to improve it using the tool's own findings and publish the before/after.
That plan has an obvious attack: "you optimised for your own metric." It's partly correct and can't be answered with numbers — so I've pre-registered the terms instead. The baseline is committed, pinned to a SHA, recorded before a single skill was improved. The prediction: procedure-genre scores will rise; doctrine-genre scores are expected to stay ≈0 — because doctrine produces judgment, not artifacts, and a rising doctrine score would mean I gamed the metric. Every rewrite publishes as a text diff, so you judge each change rather than trusting a percentage.
The tool is Act 1 of three: diagnose. Act 2 builds what's missing — the maps, indexes, logs, and commit gates that let any agent walk into a repository cold and continue from the last task. Act 3 measures whether it worked, across commits, over time. Almost no developer tool measures its own effect. That, more than any number above, is the point.
One more reason all of this is public. Everything I know about working with AI, I learned from people who shared what they found — I take from community learning all the time. When I saw what this way of working did for my own life, I wished everyone could use it and see the difference. This is my part, given back.
kibsu is Akkadian, from kabāsu, "to tread": the track left behind, the course of conduct, the way of calculation. The dictionary entry turned out to be the architecture.
Built with AI coding assistants under my direction, gating, and review — the commit trailers record which ones, because a project about verifiable claims shouldn't have an unverifiable author line.