{"slug": "tokens-are-nre-not-cogs", "title": "Tokens Are NRE, Not COGS", "summary": "Kelsey Hightower's PlatformCon talk 'Zero Token Architecture' argues that the AI industry is misallocating costs by treating tokens as recurring expenses (COGS) rather than one-time investments (NRE), and his core principle is 'infer once, export, run without inference.' The author, who works at a hardware-shipping company, applies the hardware distinction between non-recurring engineering and cost of goods sold to AI, urging developers to build assets like libraries and schemas that run on CPUs for near-zero cost instead of paying per inference indefinitely. The piece warns that agents that write code on the fly are a 'naive-solution trap' that perpetuates COGS spending.", "body_md": "Right now, as an industry, we are running almost entirely on COGS and calling it innovation. Kelsey Hightower gave a talk at PlatformCon called [Zero Token Architecture](https://www.youtube.com/watch?v=A7WFt2JQ5sg), and the whole thing compresses down to one sentence: **infer once, export, run without inference.** He’s right. I want to give you the vocabulary that makes it land in a budget meeting, because I work at a company that ships hardware, and we already have words for this.\n\nHe opens with a story about buying a very fancy track saw. He watched something like ten hours of YouTube picking one out and another fifty hours learning to use it, and by the end he *felt* like a carpenter. Then he plugged it in, moved too fast, and the blade jumped the track and came back at him at ten thousand RPM.\n\nThat’s his setup for the real point — asking Claude to deploy a Kubernetes cluster gets you a cluster, it does not make you a platform engineer — and it’s a good one. But I want to stay on the track saw a minute longer than he did, because I think the *tool itself* answers the question the talk raises.\n\nThe track costs you once. Every cut after that is free.\n\nA track saw is not a better circular saw. It’s a circular saw plus a track. And the track is a **jig**. You align it once — carefully, slowly, measuring twice, cursing a little — and then every cut after that is straight, repeatable, and takes almost no thought at all. The expensive part is the setup. The cuts are free.\n\nFreehand a circular saw and you pay the full cost of attention, judgment, and error on *every single cut, forever*. And eventually you will get one wrong.\n\nThat’s the whole argument. Everything below is bookkeeping.\n\n## NRE and COGS, For the Software People\n\nIf you’ve done hardware you already have this vocabulary in your bones. If you haven’t, here it is.\n\n**NRE** is non-recurring engineering. It’s the money you spend *once* to make a thing manufacturable. The board layout. The injection mold. The test fixture. The tooling. It’s real money, it’s often uncomfortably large, and you write the check exactly one time.\n\n**COGS** is cost of goods sold. It’s what each unit costs you *after* that. Parts, labor, freight.\n\nYou make an injection mold for plastic parts once - usually for a lot for actual money - and then every part that falls out of it costs you cents. Nobody 3D-prints a production run at forty dollars a part because 3D printing felt easier at the beginning. Well — some people do, briefly, and then someone in finance finds out.\n\nI’ve sat in the rooms where these two numbers get argued about. When you ship physical product, the difference between NRE and COGS isn’t a metaphor. It’s the difference between a business and a hobby.\n\nSo here’s the thing. Tokens are a budget line now. My [post from May on inference costs](https://blog.herlein.com/post/ai-inference-costs-reality-check/) laid out the hard numbers on where that’s heading, and nothing since has made me less worried. Tokens are a real, growing, variable line item — which means they get the same treatment as any other cost, and the question for any token you spend is simple:\n\n**Which column does it land in?**\n\nSpend in the NRE column and you bought an **asset**. A library. A schema. A code generator. A binary. A jig. It cost you a pile of inference once, and it will run for years on an ordinary CPU for approximately nothing.\n\nSpend in the COGS column and you bought **one cut**. Tomorrow the same request comes in and you buy it again, at the same price, forever.\n\nOne of these shapes is an asset. The other is a subscription you can't cancel.\n\nSo back to Kelsey’s point: right now, as an industry, we are running almost entirely on COGS and calling it innovation.\n\nThink about that for a minute.\n\n## Your Agent Will Get This Wrong, and It’s Not Its Fault\n\nHere’s the part I think is under-discussed, and it’s a naive-solution trap, not a malice problem.\n\nAsk an agent to integrate with a service and it will write a direct API call. Ask it again in another file and it will write *another* direct API call. It will hand-roll the auth, the retries, the pagination, the backoff, the error mapping, and the timeout, from scratch, each time, slightly differently. And it will do this cheerfully, instantly, and without complaint, because it can, and because nothing in your request told it not to.\n\nAnd each call site *works*. That’s what makes this dangerous. There’s no failing test. There’s no red build. Nothing is broken. You just quietly acquired forty implementations of the same idea.\n\nOh, and you are using Claude web or Claude Desktop? Yeah. Every time too.\n\nLook — the software industry did not build abstraction layers because typing was expensive. We built them because **understanding and maintenance** are expensive. DRY was never a keystroke optimization. It was about having exactly one place where you have to be right, and exactly one place to go when you turn out to have been wrong.\n\nAgents dropped the cost of keystrokes to roughly zero, and a lot of very smart people have quietly concluded that the *reason* for abstraction went with it.\n\nIt did not. It got worse. Forty hand-rolled call sites means forty places the retry logic is subtly different, forty things to review, forty things to patch when the vendor deprecates a header — and not one human being who has read all forty.\n\nThis is Kelsey’s ORM story running in reverse. My version of the story was the time we didn’t test a system built with an ORM at scale - and when we deployed it the sheer number of objects it created ran us out of memory hard and dropped the system. And of course, it did that on every startup. Days of outage. Almost lost us our largest customer ([Eric Luke](https://www.linkedin.com/in/ericjluke/) do you remember?).\n\nKelsey’s observation: ORMs hid the database, developers stopped learning how databases work, the queries got absurd, and the DBAs had to rewrite them by hand and hardcode them into the app. Knowledge *rotted*. The agentic version is worse, because agents skip the abstraction entirely and the knowledge never forms in the first place.\n\nThe library is not overhead. **The library IS the exported artifact.** It’s the durable form of an understanding you paid inference to reach. Throw it away and re-derive it at every call site and you paid for the understanding and then declined to take delivery. The **code* may in fact be fungible - but the design of the abstraction is most definitly NOT.\n\nBuild the Lego blocks. Spend real money on them — this is the one place where lavish token spend is exactly right. Get them reviewed by a human who actually understands what they do. And test the hell out of them. *Then* spend a much smaller amount of tokens wiring blocks together, and let the runtime burn a thin trickle of inference on the fifteen percent of cases your if/else tree never covered.\n\n## The Three Tiers of Spend\n\n| Tier | What it buys | How often | Where the tokens go |\n|---|---|---|---|\nBuild |\nLibraries, schemas, generators, binaries — the jig | Once, then on change | Heavy. Deep reasoning, iteration, human review |\nAssemble |\nWiring blocks into a working system | Occasionally | Moderate |\nRun |\nThe residual cases the rules don’t cover | Every request | A trickle |\n\nNow read that table upside down and you have a depressing number of production agent architectures in 2026.\n\nTeams are spending Tier 1 money at Tier 3 frequency and reporting it upward as a platform capability. Hoo boy.\n\n## The Part That Should Scare Your CFO\n\nSoftware ate the world for exactly one reason. Say it with me: **the marginal cost of another copy is zero.**\n\nBuild it once, ship it a billion times, and your unit economics get *better* with every customer. That’s the whole magic trick. That is why software companies get valued the way they do and cement companies do not.\n\nRuntime inference on every request puts a variable cost back on every transaction.\n\nSit with that one. Welcome to a cement company.\n\nA system that pays for a model call per operation has a cost curve that does not bend. Ten times the traffic is ten times the bill. You have taken a software business and reintroduced a **bill of materials**. You are un-inventing software, one agentic loop at a time, and the only reason nobody has stopped you is that the volume is still small enough that the invoice is survivable. Mostly because it was being subsidized.\n\nAnd the invoice, as Kelsey points out, is the one piece of technology in this entire stack that reliably tells the truth. Twenty bucks a month, nobody blinks. Two thousand, questions start. When the token burn matches somebody’s salary, we’ve gotten silly. Or fired.\n\n## Same Mistake, One Layer Down: RAG vs. CAG\n\nThis isn’t only an agent problem. The identical error shows up in retrieval, and it’s worth calling out because RAG-vs-CAG is having a moment on every architecture slide I see.\n\nIBM has a clean [explainer on RAG vs. CAG](https://www.youtube.com/watch?v=HdafI0t3sEY). Short version: RAG indexes your corpus into a vector database and pulls the top few relevant chunks per query. CAG — cache-augmented generation — front-loads the *entire* corpus into the context window, lets the model process it in one pass, and keeps the resulting KV cache around so later queries skip the reprocessing. CAG wins on latency and simplicity, loses on scale, and needs recomputation when the underlying data changes.\n\n(Their numbers are dated, by the way — the video talks about 32K to 100K context windows and we are well past that. Which is precisely *why* CAG got popular. Bigger windows made “just put everything in” look free.)\n\nOh yeah… the OpenClaw/Karpathy approach that’s so popular of just putting all the data in markdown in a big folder? Unless you do something to control how much gets read into the LLM you are just burning NRE.\n\nHere’s what I want to point out, and I think it’s the interesting bit: **CAG, done properly, IS Zero Token Architecture.** You pay an enormous cost once, capture the digested state, and reuse it. OK, so you periodically refresh it (using a very low cost, possibly even local embedding model). That’s firing the mold. That’s aligning the track. The instinct is exactly right.\n\nBut most of what gets *called* CAG in practice is not that. It’s stuffing the whole corpus into the prompt on every single call and hoping a cache discount covers it. Prompt caching is a **discount, not an elimination**. You are still paying to move and attend over the same material forever. Same corpus, same answer, new bill. That’s the freehand cut again, just with a bigger blade.\n\nAnd honestly? The whole framing is too narrow. “RAG or CAG” assumes the knowledge has to be consulted by a language model at all. Often it doesn’t. Escalate deliberately:\n\n**Precompute.** If a question has one stable answer, that’s not retrieval — that’s a*value*. Put it in a table.**Look up.** Deterministic key, deterministic answer. No embeddings required.**Retrieve.** Now you genuinely need semantic matching over something large or fresh. Fine: RAG.**Infer over context.** Now you need synthesis across a bounded, stable corpus. Fine: CAG.**Reason from scratch.** Genuinely novel. Spend freely — and then export whatever you learned.\n\nEach rung costs roughly an order of magnitude more than the one below it. Or so. Let’s not argue about the specifics since if we keep spending like that we’ll be unemployed.\n\nThe question was never RAG or CAG. It's how far down you can push each request.\n\nMost designs I see start at rung four or five and never look down. Take IBM’s own help desk example — a 200-page manual, updated a few times a year. CAG is a perfectly defensible answer there. But twenty questions are going to be most of your ticket volume, and those twenty have *fixed answers*. Precompute them, route the tail to the model, and you just moved eighty percent of your traffic off the meter entirely.\n\nThe interesting question was never RAG or CAG. It’s: **how far down this ladder can I push each class of request?** You want to know what happens to Engineering in the AI age? THAT.\n\n## Tokens Are Joules\n\nOne more thing that I don’t think gets said plainly enough.\n\nInference is not an abstraction. It is electricity through silicon in a building that needs water and a grid interconnect. **Tokens are joules.**\n\nMoney can be raised quickly. Substations cannot. Sooner or later, “we’ll just buy more capacity” stops being a sentence you can finish.\n\nWe’ve run this arc before, slower. Early cloud was “just spin up more instances,” because compared to a procurement cycle it felt free. Then the bills got big enough to be visible at the board level, and an entire discipline appeared to deal with it — rightsizing, reserved instances, spot, and eventually a wave of repatriation by companies who did the arithmetic and did not care for the answer. Nobody arrived at FinOps because it was intellectually exciting. They arrived because somebody in finance asked a question they could not answer.\n\nThat question is coming for inference. It’s the same question it always was: **what is our cost per transaction, and is it going down?**\n\nFOMO is what’s keeping that question from being asked out loud today. No executive wants to be the one who slowed down the AI initiative. That protection has a shelf life, and it expires the *instant* a competitor demonstrates the same capability at a tenth of the run rate.\n\nIt’s already starting. Kelsey described a company he visited in France that revoked unlimited token budgets and got something close to a revolt — teams genuinely could not articulate how they’d meet their previous throughput without it. He compares it to the “GitHub is down” reflex on social media, and that’s funny, but look at what it actually is.\n\nThat’s not a cost story. That’s an **operational risk finding**. If the tokens going away means the work stops, you have a dependency you never signed off on.\n\n## Yes, the Naysayers Have a Point\n\nLet me not get too high on my own supply. There are three objections here that deserve real answers, and I’ve made two of them myself.\n\n**“Inference costs are collapsing, so this is a temporary problem.”** Per token? Yes, dramatically. Per *task*? No. We spent every single efficiency gain immediately — on longer context, more reasoning, more loop iterations. Falling unit price met rising unit consumption and the bill went up anyway. Jevons has not been repealed. And volume growth outruns price decline: if your unit cost drops 10x while your traffic grows 100x, you have a bigger problem, not a smaller one.\n\n**“Premature abstraction is real.”** Absolutely it is, and building the *wrong* Lego blocks is worse than building none. But this isn’t an argument against exporting, it’s an argument about *when*. You use inference to explore, to find the shape, to be wrong a few times. Then you export. That’s “infer once” in full — the first pass is *supposed* to be expensive and exploratory. The failure mode is never coming back to harvest it.\n\n**“Some things genuinely can’t be enumerated.”** True, and important. Open-ended natural language, novel inputs, actual judgment calls. That’s the tail, and the tail is exactly where inference earns every penny. Spend there. The argument was never “no tokens.” It’s: know which of your requests are the tail, and **stop paying tail prices for the head.**\n\nBOTH can be true. The technology is genuinely astonishing AND most of us are currently deploying it with the unit economics of a cement company. The industry keeps trying to pick one of those. Baloney. It’s both.\n\n## Conclusion\n\nThe engineering skill that’s about to matter is not prompting. I’ll say that again because there’s an entire cottage industry betting otherwise: **it is not prompting.**\n\nIt’s deciding what deserves inference at all — and having the taste to recognize, mid-loop, that the expensive thing you just did could be done *once* and exported.\n\nAnd that’s not a new skill. It’s the oldest one we’ve got. It’s caching. It’s compiling. It’s why we have libraries instead of retyping the standard library into every program. It’s a machinist looking at a job and knowing the first hour goes into the fixture so the next four hundred parts fall out identical without anybody thinking about it.\n\nI started coding on a TRS-80 with 4K of RAM, where you could feel every byte. Then memory got cheap and we stopped thinking about it, and mostly that was fine. Tokens are having their expensive era right now. Some of the habits we’re forming in this era are going to look really dumb in five years, and some of them are going to look like the thing that saved the company.\n\nAlign the track. Then make the cut, and every cut after it, for free.\n\nSo: go look at your token bill and ask which column it’s in. If you find something in the COGS column that should have been NRE, I’d genuinely love to hear about it — that’s the interesting part of this whole thing, and I don’t think anybody has a good playbook yet.\n\nDrop me a note on LinkedIn: [linkedin.com/in/gherlein](https://www.linkedin.com/in/gherlein/).\n\n## Related Reading\n\n[ZTA: Zero Token Architecture](https://www.youtube.com/watch?v=A7WFt2JQ5sg)— Kelsey Hightower, PlatformCon 2026. Worth all 29 minutes; the Q&A is better than the talk. Also covered by[The Register](https://www.theregister.com/2026/04/08/automation_zerotoken_architecture_ai/).[RAG vs. CAG](https://www.youtube.com/watch?v=HdafI0t3sEY)— IBM Technology. Dated numbers, solid framing.[When to Replace an Agent With a Script](https://bmdpat.com/blog/when-to-replace-ai-agent-with-script-2026)— supplies the rule of thumb Kelsey leaves implicit: if an if/else tree handles 95% of cases, it’s code.[No shade, but that agent should have been a Python script](https://jeremiahdillon.com/writing/that-agent-should-have-been-a-python-script/)— Jeremiah Dillon. (And yes, I have[opinions about the Python part](https://blog.herlein.com/post/agentic-python-considered-harmful/).)[Improving token efficiency in GitHub Agentic Workflows](https://github.blog/ai-and-ml/github-copilot/improving-token-efficiency-in-github-agentic-workflows/)— the same idea from a vendor actually shipping it: move deterministic steps out of the reasoning loop entirely.[How AI assistance impacts the formation of coding skills](https://www.anthropic.com/research/AI-assistance-coding-skills)— Anthropic. Relevant to the knowledge-never-forms problem.", "url": "https://wpnews.pro/news/tokens-are-nre-not-cogs", "canonical_source": "https://blog.herlein.com/post/tokens-are-nre-not-cogs/", "published_at": "2026-07-31 08:00:01+00:00", "updated_at": "2026-07-31 23:54:11.436483+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-infrastructure", "ai-agents", "ai-policy"], "entities": ["Kelsey Hightower", "PlatformCon", "Claude"], "alternates": {"html": "https://wpnews.pro/news/tokens-are-nre-not-cogs", "markdown": "https://wpnews.pro/news/tokens-are-nre-not-cogs.md", "text": "https://wpnews.pro/news/tokens-are-nre-not-cogs.txt", "jsonld": "https://wpnews.pro/news/tokens-are-nre-not-cogs.jsonld"}}