You Cannot Distill What You Do Not Record

Enterprise AI programs often mistake acceptance rate for a measure of review integrity, but it only measures approvals, not attention, creating a dangerous gap in observability. Observability is a prerequisite for every stage of AI maturity, from detecting shadow usage to distilling skills, yet most organizations measure only easy metrics like seats and tokens while ignoring defect escape rates. This dashboard theater leaves organizations blind to the actual performance of their AI systems.

Picture the AI transformation dashboard most executive teams are looking at right now: seats licensed, tokens consumed, acceptance rate, all trending the right direction, all green. Now ask the room a second question. What's your defect escape rate? What percentage of your gates actually catch a planted defect when you test them? Watch how fast the room goes quiet. Acceptance rate is a sentiment metric in a lab coat acceptance-rate-is-a-sentiment-metric-in-a-lab-coat That line showed up once already in this series, describing the review bottleneck /amm-3-diagonal-law that most enterprise AI programs are currently living inside without knowing it. It belongs here too, because it's really a claim about observability: acceptance rate looks like a hard number. It has decimal places. It updates in real time on a dashboard. And it measures approvals, not attention, which means it can climb every single week that the thing it's supposed to be a proxy for, actual review integrity, quietly falls apart underneath it. That gap is not a data problem you fix by adding another chart. It's a track this model treats as its own dimension, separate from where trust lives and separate from what capability an organization has deployed. Why observability is a track and not a level why-observability-is-a-track-and-not-a-level It would be tempting to make observability Level 5, the reward for finishing everything else. That's exactly backwards, and the model is built to resist it on purpose. Making observability a level lets an organization defer it to the end, which means deferring the one thing every other transition actually depends on. Every level transition in this model is an observability upgrade before it is a tooling upgrade: 0 → 1 requires seeing shadow usage: the gap between what the egress logs actually show and what the sanctioned tool list says should be there. You cannot govern what you deny exists, and an organization that hasn't built the visibility to see its own shadow fleet can't move past prohibition honestly, only on paper. 1 → 2 requires usage telemetry: which tools, which teams, what spend. Without it there's no way to know whether sanctioned tooling is actually being used, or just approved. 2 → 3 requires outcome telemetry, defect escape rates, an honest measurement of how deeply reviewers are actually reading. This is the one most dashboards skip, because the review bottleneck is invisible to usage telemetry by construction. Seats and acceptance rates look wonderful right up until you superimpose the escaped-defect curve on top of them, and the two lines tell completely different stories. 3 → 4 requires learning telemetry: which findings recur, which gates catch planted defects, which skills actually get hit in practice. Distillation and skill mining consume this telemetry as raw material. Without it, there's nothing to mine. Put plainly: You cannot verify what you cannot observe, and you cannot distill what you do not record. Each transition is gated by the specific kind of evidence it consumes, not by telemetry in general. Outcome telemetry doesn't help you distill. Learning telemetry doesn't help you see shadow usage. Bringing the wrong instrument to a transition is its own kind of theater. M5, Dashboard Theater observability track stuck at stage 2 m5-dashboard-theater-observability-track-stuck-at-stage-2 I've sat in too many meetings where the dashboard is all green and the one number that matters was never on it. This is the trap living directly underneath the pleasant-looking dashboard from the top of this post. Seats, tokens, and acceptance rate get reported as outcomes, full stop, with no defect-escape data, no gate-calibration data, and no accuracy analysis anywhere in the building. It's not usually dishonesty. It's an organization measuring what was easy to measure at the moment it started measuring, and then never asking whether that measurement still means what it used to. The uncomfortable part of this trap is that it can coexist with real progress elsewhere. A company can build genuine rails, charter real adversarial review, and still report its AI program's health entirely in usage terms, because that's the telemetry the organization invested in first and nobody circled back. Dashboard Theater isn't the absence of data. It's the presence of the wrong kind, reported with the confidence of the right kind. One caution the model is explicit about, because it would be easy to over-correct into a different mistake: telemetry is not virtuous by itself. An organization that instruments everything and acts on nothing has just built a more expensive version of the same theater. The point was never "collect more data." It's that each transition consumes a specific kind of evidence, and the discipline is matching the instrument to the boundary you're actually trying to cross. What this buys the next two tracks what-this-buys-the-next-two-tracks Observability isn't interesting on its own. It's interesting because of what it feeds. The verification track /amm-6-review-prosecution-calibration runs on outcome and learning telemetry: you cannot calibrate a gate's catch rate without recording what it caught and what it missed, and you cannot know a review process has degraded into rubber-stamping without measuring review depth directly rather than inferring it from approvals. The knowledge track /amm-4-rag-runtime-skills-compiled runs on the same raw material from the other direction: skill mining needs to know which patterns recur before it can compile them, and skill-rot detection needs a record of what a skill claimed to be true so it can be checked against what's actually true now. Neither of those loops closes without something recording, continuously, what actually happened. That's what makes observability load-bearing rather than optional: it isn't a level you eventually reach. It's the thing every other level was quietly standing on the whole time.