Your LLM Telemetry Table Does Not Have One Denominator

A developer auditing an LLM telemetry report found that its tables conflated several distinct studies, with core process metrics attributed to model epochs within threads while the completion proxy was computed only per main thread with model purity at or above 0.9. The report also separated main sessions from sidechains and split pre-relaunch and relaunch observations into distinct model epochs rather than pooling them under a single model name. The developer argues that each row's denominator must be defined by model_epoch, role, and analysis unit before any comparison is valid.

Originally published on hexisteme notes https://hexisteme.github.io/notes/stratified-comparison.html . I had an LLM telemetry report with the usual reassuring furniture: medians, interquartile ranges, sample sizes, and bootstrap intervals. The rows were grouped by model. It looked ready for comparison. It was actually several different studies sharing a table. The core process metrics were attributed to model epochs inside threads. The completion proxy existed only at thread level. Main sessions and sidechains had different structures. Mixed-model threads could contribute to one table but fail the purity rule for another. Historical routing policy was often unknown, and task family was not observed at all. The dangerous mistake was no longer simply calling an association causal. It was treating every n beside a model label as if it counted the same kind of thing. For readers auditing an agent harness, hexisteme/hard-gate-hooks https://github.com/hexisteme/hard-gate-hooks contains two MIT-licensed Stop-hook examples, their tests, and a read-only scanner. They are adjacent implementation examples, not the telemetry instrument described here. For the core metrics, attribution happened inside a thread. A multi-model thread could produce separate epoch rows because each assistant turn was assigned to the model epoch that produced it. Tool-error rate, re-edit rate, validation runs, recovery sequences, and output tokens therefore described epoch-attributed portions of work. The completion proxy had a different contract. It was computed once per thread, only for main threads with model purity at or above 0.9 , and censored threads were excluded. The same model label could therefore have one sample size in the core table and a smaller one in the proxy table without either count being wrong. That distinction changes what a sentence is allowed to say: Putting the values in neighboring sections does not make their denominators interchangeable. Before reading a difference, I now ask what one row represents: a turn-attributed epoch fragment, a whole thread, a delegation edge, or something else. The report kept main sessions and sidechains separate. That was necessary because they do not end the same way and do not carry the same amount of interaction. A main thread can contain repeated reads, edits, recovery, and user turns. A sidechain is often a bounded delegated run. A re-edit rate in those two populations mixes model behavior with session structure. Calling the gap “rework” would add another unsupported interpretation: the metric sees repeated edits to a file, but it cannot tell productive iteration from repair. The completion proxy made the role mismatch even sharper. One sidechain harness commonly ended on a tool result line. In one recorded cell, that happened in 92 of 99 threads. A last-line heuristic could read those endings as incomplete even when the delegated work had finished. The report therefore excluded sidechains from completion comparison instead of repairing the headline with a caveat. That is the right direction for an invalid metric: narrow its valid population. A warning below a cross-role chart does not undo a denominator that never meant the same thing across roles. A model name can survive a relaunch, configuration change, or routing change. The treatment does not. The report split pre-relaunch and relaunch observations into separate model epochs and refused to pool them. This was more than a naming preference. Sequence metrics were calculated within each epoch fragment, so an edit performed by one model and validated after a model switch was not credited as a within-epoch validation sequence for either side. That limitation is useful because it is visible. Pooling both epochs under the same display name would create a larger sample by erasing the boundary that explains why the sample is heterogeneous. The practical key for a row is therefore at least: model epoch × role × analysis unit Add policy and task family only when those fields are actually observed. A friendly model name is presentation. It is not a stable experimental identity. The report also grouped rows by a source-stamped dispatch-policy hash when that evidence existed. Many historical sessions had no known policy version. Those rows remained UNKNOWN ; they were not backfilled from the current router or guessed from which model happened to run. Task family was stricter still. Every populated row in that stratum was marked NOT OBSERVED . The correct output was an empty comparison, despite thousands of rows elsewhere in the report. This is the point where a telemetry system proves whether it is an instrument or a story generator. A missing assignment variable is not a neutral baseline. It is an unmeasured confounder. The table may still describe the sample under the routing policy that produced it, but it cannot claim to have held task family or policy constant. The interval calculation also had to follow the unit contract. Continuous process metrics used a difference in medians. Binary completion proxies used a difference in proportions. Both cells needed enough observations, and comparisons stayed within the same role. Several zero-heavy cells still produced a bootstrap interval of 0, 0 . That did not mean the effect was known exactly. When most resamples contain the same tied value, the percentile bootstrap can collapse because the statistic does not move. The interval is describing the resampling behavior of a nearly constant cell, not granting the measurement infinite precision. An interval inherits every limitation of the metric beneath it. It cannot repair a proxy that changes meaning by role, a pooled epoch, or a policy field that was never observed. Before I let a telemetry row influence routing, I record these fields with it: UNKNOWN ; That last field matters most. In routed production telemetry, an observed gap can reverse after task family, project, time, or policy is held fixed. The table is useful for monitoring drift and choosing where a controlled experiment would pay. It is not a model leaderboard. As of the 2026-09-14 source snapshot, the table remains association-only. The testable prediction is that at least some apparent model gaps will shrink, disappear, or reverse after recorded policy, role, task family, and analysis unit are held fixed. That prediction is invalidated if adequately overlapping matched cohorts preserve the same gaps in direction and practical magnitude. The trigger is the first source-stamped task-family cohort large enough for that comparison. Persistence would justify a randomized dispatch experiment; only randomized assignment could support a causal attribution to the model. When you look at your own LLM telemetry, does every n count the same kind of thing? Email list for these notes: hexisteme.beehiiv.com https://hexisteme.beehiiv.com/?modal=signup&utm source=devto&utm campaign=notes-engineering&utm content=stratified-comparison — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell. More notes at hexisteme.github.io/notes https://hexisteme.github.io/notes/ .