Measuring Without Breaking

Companies that track AI usage risk turning metrics into weapons, as seen with token leaderboards that spawned 'tokenmaxxing' behavior. The problem is not measurement itself but the culture that uses it, where metrics become targets and lose their value. Healthy cultures use measurement as a tool, not a boss, and acknowledge that the most important things cannot be counted.

Measuring Without Breaking A healthy culture can track AI use. An unhealthy one turns the same number into a weapon. Disclaimer: This final draft is AI generated, then edited by me. What does that mean? Well, after some fairly substantial writing, I was tempted to drop this article entirely. I’d taken the writing in a few directions, and it was becoming a sprawl that would take a long time to recover from. I felt there were interesting ideas, but wasn’t sure it was worth continuing. Before abandoning, I worked with Claude to rewrite around a new concept. The core ideas here are very much my own. Since Claude had access to my failed drafts, many words are mine too. But the direct product is not. If you’re substantially opposed to AI writing, and you dislike this, you can reaffirm your priors. If you can see through that to the ideas here, then maybe there’s something valuable. I leave this with you. I could spend time using this as an inspiration, rewriting parts, and eventually this disclaimer would no longer be necessary. Every company past a certain size runs into the same problem. It needs to know what’s happening inside itself, and the only way to know at scale is to measure. But measuring changes the thing you measure. Tie a number to someone’s standing and the number stops telling you about the work. It starts telling you how people react to being numbered. That’s not a flaw in any one metric. It’s the basic mechanics of management, and most of the recurring messes in corporate life are those mechanics showing up in new clothes. The newest example is AI usage tracking. Companies built “leaderboards” to measure how many tokens each employee burned, and “ tokenmaxxing https://en.wikipedia.org/wiki/Token maxxing “ grew up around them. The leaderboards are being torn down now, and the easy lesson is that they were a mistake. I think that misses the more useful story. The leaderboard was a measurement. Like any measurement, it could have been a way to understand the team, or a machine that wrecked it. Which one it became had less to do with tokens. It had to do with the culture it landed in. I want to flip the framing. Tokenmaxxing is the supporting story. The real one is about how a management culture takes the measurements it actually needs without those measurements rotting in its hands. The saying that starts the trouble Start with one of the most quoted lines in management, and one of the most quietly destructive: you can’t manage what you can’t measure. It sounds like rigor, but it’s really a hidden assumption — that everything important can be measured. It can’t. Whether your people trust each other. Whether they tell you the truth when the truth is inconvenient. Whether one quiet engineer is the reason three teams ship on time. Whether someone’s messing-around this quarter becomes a real tool next year. These decide whether a team is any good, and none of them hold still long enough to be counted. Take the saying seriously and it tells managers to ignore exactly these things, or to invent stand-ins for them and manage the stand-ins instead. You end up with a manager optimizing a dashboard who believes they’re doing sharp work. They’re doing bad work with better instruments. The first thing a healthy culture admits is that the most important things will never show up on a chart, and that managing them anyway — by judgment, by paying attention, by knowing your people — is the actual job. Measurement helps inside that job. It’s a useful helper and a terrible boss. Why measurements rot Say you accept that you still have to measure something. The question becomes mechanical: what turns a useful measurement into one that poisons the thing it tracks? Goodhart’s Law https://en.wikipedia.org/wiki/Goodhart%27s law names the result — when a measure becomes a target, it stops being a good measure. But it doesn’t tell you what lets it take hold. Three things do most of the damage. You tie the number to standing. The moment a number decides who gets rewarded and who’s at risk, everyone being measured cares more about the number than about the thing it was supposed to stand for. The dishonesty is terrible. But it’s hard to lay blame. They’re being rational. You built a game and they’re playing it. You roll the number up. In most companies a manager’s standing is built from their reports’ numbers, and that manager’s number feeds the layer above, and so on. This is the quietly fatal part. When a manager catches a report gaming a metric, calling it out lowers the manager’s own score too. You’ve asked people to police a number that pays them to look away. Stack ranking https://en.wikipedia.org/wiki/Vitality curve and the usual performance calibration run on this same wiring — managers set against each other, reports who game the system pulling their manager up with them. I’ve never seen a clean example of it working, and the reason is structural. It’s not about finding better people. You only manage downward. Management that runs in one direction — pulling data up, never answerable for the environment it creates — has no correction built in. Information flows up after it’s already been polished, and nobody whose incentives are intact is in a position to notice the data stopped being true. Put the three together and you get a machine that reliably turns measurement into theater. Notice that none of the three is a property of the metric. They’re all properties of the culture you drop it into. The same number, two outcomes This is where tokenmaxxing earns a bad name, by participating in bad management practices. There was a reasonable purpose at the start, but the theater arrived quickly. Technically, the word has two meanings, and it helps to engage with each. On the company’s side, tokenmaxxing was the choice to build a leaderboard and signal that more AI use was better. That choice had purpose. Most companies had spent a year discouraging AI with restrictive security policies, and they needed a push to break both that inertia and the ordinary human reluctance to change how work gets done. Later, tokenmaxxing described the unproductive response — and became the dominant meaning. Some people tried AI in good faith and kept what worked. Others focused on their usage numbers for no reason except that tokens were what got counted. In other words, doing work to look like they did work. A company running a leaderboard gets both the experimentation and waste because it can’t tell them apart at scale. Nobody can audit intent across a few thousand people. So the program is really a bet. Take a pile of aimless activity and some deliberate waste, in exchange for the slice that turns into something durable — a real skill, a useful tool, a project nobody had time to chase before. Said that way it’s an ordinary bet, the same shape as a research budget or a hiring class. And like those, it should always have been temporary. You retire it once the inertia is broken. Now watch the same leaderboard land in two different cultures. In a healthy one, a manager sees a report sitting at zero tokens and reads it as a question . What’s going on? Is the tool not helping? Is there a reason? The number becomes a reason to have a conversation. The manager has no rollup score to protect, so they can be curious instead of defensive, and the number stays roughly honest because nobody’s livelihood is riding on bending it. In an unhealthy one, the same zero reads as a verdict . The rollup punishes any manager who admits their team’s numbers are soft. Within a quarter the leaderboard measures one thing: each person’s willingness to game it. Same tool. Opposite outcome. The variable was never the token. That’s why I’d call the leaderboard a stress test, not a cause. Drop it onto a culture with a hidden crack and it doesn’t make the crack. It loads it until it shows. The worker backlash these programs set off was real and it did damage, but its source wasn’t the number. It was the accumulated, accurate sense that bad management rarely gets removed and often gets rewarded, and that any new tool would be bent to serve it like every tool before. The leaderboard just made that easy to see. Running a culture that can measure If the metric isn’t the variable, then “pick a better metric” isn’t the fix. Neither is the opposite reflex of refusing to measure anything. The fix is cultural, and it’s harder, because you have to keep it up rather than decide it once. A few things seem to separate the cultures that can hold a measurement from the ones that break it. Keep a gap between the number and the reward. The instant a metric is wired straight into pay and survival, the rot starts. The people you most need telling you the truth now have the strongest reason not to. Healthy cultures treat a measurement as one input a manager weighs against everything they can’t measure, not as the verdict itself. “Let’s understand why usage varies” survives. “Bottom decile is at risk” is already rotting. Give managers a real reason to want the truth. Mostly this means taking apart the rollup, or at least refusing to let a manager’s standing be a straight sum of their reports’ numbers. A manager whose rating doesn’t depend on their team’s metric looking good is finally free to do the thing you hired them for — notice when the number and reality have split, and say so. Point accountability inward, not outward. The reflex, when people game a measure, is to go after the visible gamers and stop there. The trouble with that framing is it quietly lets the managers off, and the managers are usually the ones who built the environment that produced the gaming. A leader at the top can be careful and dodge the rollup trap — refuse to let their own standing ride on their org’s numbers. That doesn’t stop a manager one level down from doing the opposite. They take the metric they were handed and roll it down onto their reports as a hard target, then turn a blind eye to how the reports game it. The pressure to make the number gets passed along even without explicit design. So when gaming shows up, the manager is rarely a bystander. Often they were part of the gaming — they wanted the number to look good and didn’t care how it got there. Where they weren’t actively in on it, they were just bad at the job, blind to the fact that their own pressure was manufacturing the behavior. I’m not sure which is worse, and for individuals the difference barely matters. Either way, punishing the report who followed those incentives, while leaving that manager alone, fixes nothing and adds a fresh unfairness. Pulling back unearned rewards so gaming doesn’t visibly pay is punishment enough. But the real correction is at the center — the people who shaped the environment — not the edges. Fix them first. They’re the ones who’ll do it again. Trust is the thing holding it all up, and it’s self-fulfilling. In a culture where people trust that a low number gets met with curiosity and that gaming gets caught instead of rewarded, you can introduce a measurement without panic, and the calm keeps it honest. In a culture where people expect the worst, they respond to the worst, and the response creates the very rot they feared. The belief and the outcome make each other. So trust isn’t a soft extra bolted onto a measurement program. It’s the load that everything else rests on. A measurement dropped into a low-trust team is closer to tossing in a grenade than running a diagnostic. I’ve argued before that money itself is just trust https://substack.norabble.com/p/money-is-trust — measurement inside a company is no different. It only works if people believe the other side is dealing straight. Stay humble about the whole thing. Management is a field of unintended consequences. Its job is to improve the value the people under it create, which in a perfect world would mean nothing, because you can’t improve on perfect. The mistake is for management to act as if it lives in that perfect world — clean numbers, frictionless incentives, its own presence bending nothing. Accepting that the world isn’t ideal, that every metric is a little flawed and gets more flawed as the work gets more complex, is the first of many steps toward doing it well. The cultures that can measure without breaking the measurement are just the ones that never forgot the tool was imperfect, held it loosely, and were willing to put it down once it had done its job. The lesson hiding in the leaderboard The leaderboards deserved retirement, but they weren’t all folly. They did an ordinary job that reached its expiration date. In cultures healthy enough to use them well, they were probably useful. Even in less effective cultures there may have been a golden period before the rot set in. Where they turned into a disaster, the disaster was already there, waiting — in the rollups, in the important things ignored for measurable stand-ins, in accountability that pointed outward instead of inward, in trust that had been spent long before anyone counted a token. That’s the lesson worth keeping once the leaderboards are gone. The next iteration is already coming. It’ll promise the same clean view into the same messy reality, and it’ll rot the same way, unless the culture holding it has done the slow, unglamorous work that lets a company measure itself without lying to itself. The hard problem was never the metric. It’s building a place where the truth can survive measurement.