{"slug": "million-dollars-in-tokens", "title": "Million Dollars in Tokens", "summary": "A billing engineering leader at a hyper-growth AI model broker met with the writer to discuss a token leak that cost the company roughly $1 million over a year, caused by an integration with a major agentic marketplace where usage from about 32,000 trial accounts was never logged in the credit ledger. The incident highlights how AI usage-based billing breaks traditional assumptions, since prices are often dynamic and unknown at the moment a usage event is sent, complicating metering, reconciliation, and token-bank integrity at scale.", "body_md": "Over the last weekend, OpenAI released Astra. It was a happy long weekend for developers. I stayed up the whole weekend trying one-shot prompts to create some cool stuff with it, and it’s super cool. Yet another step toward AGI.\n\nIt’s no secret that OpenAI did a great job with developer relations lately with Codex. It surpassed Claude Code, at least for me, over the last couple of months, and I see more and more engineers moving to Codex over Claude Code.\n\nA consistent discussion I saw over the weekend in the engineering groups I’m a member of was the feeling of unfairness that users with a private OpenAI subscription get more, while users with an organizational subscription get lower limits. With a personal subscription you get a subsidy, and usually you can just upgrade yourself to the 100 bucks plan. So people keep finding cheat codes: how to get personal accounts, how to turn experiments and trials into credits for the organizational account.\n\n**Developers are desperate for tokens these days.**\n\nNot accidentally, last week I also had a meeting with a billing (financial, to say) engineering leader from a hyper-growth AI startup that you all know. A hyper-growth AI model broker that a lot of developers use. The reason for the meeting was a token leak. They found a bug in their system that leaked a million dollars in tokens over a year, and they wanted to learn how we use metering and billing infrastructure for AI applications. They already use some solutions in the market, but now they need to do it at scale.\n\nIf you are building an AI application, you want to make sure your token bank works, does not leak tokens, and does not create situations where your users don’t pay for compute that you pay for. These are the learnings I took from the meeting.\n\n## **The bug**\n\nBefore we start, let’s understand the bug they had.\n\nThey had an integration with one of the biggest agentic marketplaces out there. When a user opened an account via this marketplace, usage somehow didn't get logged in their credit ledger. Probably agents, maybe real users, maybe someone else, found that. The company found out they had 32,000 accounts created as trials via this marketplace, consuming tokens in a way that made no sense for them. It cost them around a million dollars a year until they discovered it.\n\nOf course, it is not a bug in their infrastructure per se. It’s a bug in the way they connected everything. But that’s the point. A standard bank thinks it has one source of truth for a transaction and the way it counts it. With agentic applications and AI usage, it is way more complex. There are way more connections, and way more points where you make this connection.\n\nSo let’s go into what we learned from the discussion.\n\n## **1. The price is not known when the usage event is sent**\n\nCurrent usage-based systems assume you know the price before sending the usage event. Meaning you already have a catalog. A catalog item can be, for example, a model API, and when you send a usage event for this model, by token count or just by API call, you already know the price.\n\nIn reality, with AI systems at high scale, especially when they all play together, prices are dynamic. Many times there is no static price line when you send the usage event. It can be dynamic token pricing, where you kind of hide the credit price from the user. It can be that you decouple inference metering from the contract, so when you send the usage event you don’t know yet the price in the contract. Or you have cascading prices in the contract, and you send the usage event without knowing which one applies.\n\nThis changes the economy. The price is set on the fly, sometimes not even by a human, without any correlation to what is happening now in the usage pricing. It also needs to touch the billing system, and all of that has to happen at scale, with zero latency and reliable reconciliation.\n\nSo when we build usage-based systems for AI, we need to take that assumption out and build horizontally, so we don’t make the vertical mistakes that later force corrections for pricing errors.\n\n## **2. Coupling usage management to the billing provider**\n\nMany startups, because it is comfortable, start their usage management coupled to their billing provider. They need a fast way for people to swipe a credit card or create a contract, and then they use the provider’s APIs and SDKs to create the catalogs. And like the Telegraph Road song, then come the layers, then come the rules, and then your billing system cannot be the source of truth anymore for all the complexity of usage management you have.\n\nIf you are already caching inferences and getting direct access to frontier model inference, having your usage management, catalogs and contracts depend on a billing-as-a-service provider does not make sense.\n\nIt doesn’t mean you need to start with the whole usage infrastructure. But you need to keep a copy of the contract system, the catalog and the dynamic pricing as your source of truth, step by step. Then, when you need to scale your usage runtime, you can connect it to that source of truth, instead of running everything out of a not-that-sophisticated, not-built-to-scale billing system.\n\n## **3. Upfront validation**\n\nMany times people just want to move fast, and they don’t validate entitlements against usage limits in real time. That means that at scale you start getting more and more correction automations. The engineer I talked with currently has 10 continuous automations in place that correct contracts according to the usage and metering they built.\n\nHe said he wished someone had told him to invest in upfront validation and enforcement of entitlements before they started. It is not that much effort at the beginning, relative to the time it takes for the corrections. Sometimes these contract corrections even create a bad relationship with customers. Instead, they built a system that measures after the fact, and even when they do ledger the contract and the pricing, it comes after, not beforehand.\n\n## **4. There is no standard sequence for the pipeline**\n\nThis one personally interests me, because it’s related to what I mentioned at the beginning about the OpenAI subsidy for personal accounts. It’s a conclusion we came to in our discussion, and something I kind of heard from OpenAI too, given the complexity of their metering: it’s super hard to predict and build the right sequence of modern usage-based pricing.\n\nThink about it like a pipeline. It includes a pricing line, the catalog, meter, model, usage, organization, contract, personal account inside that contract, the economy unit of a token, compute cost, caching, inference prices. And more, this is not the whole list. There is no way to standardize the sequence of this pipeline. If we had one, we would predict better, model better, build better hierarchical contracts, and subsidize personal accounts inside enterprise contracts better. The fact that an enterprise needs a contract creates a complexity that almost every AI company does differently.\n\nAt Stigg, when we help AI companies manage their usage runtime, we never try to decide the sequence for them. We try to have the best building block for each part of the pipeline.\n\nIn a personal account the sequence is very hierarchical, and you can easily track from compute cost to user cost. You don’t have pricing lines, that many hierarchies, or economy units. This is why it’s way easier for AI companies to subsidize personal accounts. By the way, it’s also what makes them easier on the beginner account, which is exactly where the token leak happened.\n\n## **5. Agents will find the hole**\n\nI think this is one of the most interesting points because it tells a lot about how software is built in a world of AI. The leak happened because the integration with the marketplace did not log the initial usage, and at some point an agent, or someone, discovered that. They don’t know yet if it was intentional or just happened.\n\nWhen we build software today, the baseline is an agent using our software. We need to make sure there is nowhere in our pipeline, our usage management, our ledger, that is not measurable. If an agent desperately needs tokens and finds a way to get them, like the OpenAI agent that hacked the sandbox, it will take tokens it is not paying for.\n\nSo a usage runtime for AI applications needs to take into account, first, that we will have tons of integrations, and all these external integrations need to be aware of and work with our usage management architecture. And second, we need to be ready to measure the spikes that come from that. I also saw today that Vercel announced a flat-rate CDN. I think it’s another pointer that when agents use our product, it is not predictable how. Your usage runtime needs to handle spikes, agentic use, and the things agents find.\n\n## **6. The whole is greater than the sum of its parts**\n\nWhen we manage the usage runtime for an AI application, we need observability at any zoom level in our system.\n\nIn their case, the only visibility they had was on the billing infrastructure they maintain. Because the usage was never logged, they could not see that 32,000 accounts were opened. When ledgers and meters are tied to billing systems and to the personal account, you lose the option to observe a large leak before it happens.\n\n## **7. You can’t take just the part you need**\n\nThis is one of the ways they got complex. Many usage management providers today don’t let you take just the credit pool, just the ledger, just the meter, just the part you need. They force you to manage the catalog with them, including the pricing. But the catalog is something you already have in your billing system, your ERP, your internal systems. This redundant integration point creates data that doesn’t match between points, drives you to data reconciliation, and forces you to correlate everything in your system on the fly. It creates complexity in the usage event itself.\n\nAt Stigg, we try to keep everything decoupled: the metering, the credit ledgers, each component. You don’t need to manage the whole usage runtime with us. You can connect it to your catalog, to your contracts in your ERP, to your billing provider integration. This is super important when you consider a usage runtime.\n\n## **8. Discounts and subsidies**\n\nAnother point about traditional usage management that struck me in the discussion is discounts and subsidies. Traditionally, discounts were constrained by the contract period. You give someone 20% on the whole pricing line for the contract. With AI that creates complexity. You want to test a new model, or give someone their money back for some time, so you give them a discount. Sometimes the discount is not even visible to the user, for example when you have failed runs and you return the credits so the account is balanced.\n\nAll these discount intervals are now way more flexible than what common discount systems allow. When you serve hundreds of models together with dynamic pricing, you need a way simpler discount architecture that you can rely on and that connects well to your contracts and hierarchical contracts.\n\n## **Bonus: developer platforms are dead**\n\nBeside all the learnings from someone developing a high-scale, hyper-growth AI system, one bonus. I saw a tweet a couple of weeks ago, I can’t remember by who, that you either create a harness or tooling for a harness. We are not creating software for users anymore. Either you are the harness and serve models directly to the user, or you build a tool for the harness that serves the model.\n\nI’ve been building developer tools for a decade now. In the past, we wanted to build a developer platform: a platform for auth, a platform for observability. I think developer platforms are dead. Now we have developer infrastructure, or more correctly, agentic infrastructure. You need to build and invest in the critical building blocks that require your research, your expertise, the way you see customers, and the way you serve, always on the edge, giving developers the advantage.\n\nWe shouldn’t build developer tools anymore as a platform that provides everything to the user. In this case, we heard someone say: I can build many of these things myself, but I need you for the critical path. When I need decentralized metering. When I need zero-latency entitlement enforcement. This is where I need you.\n\nThat is the learning I took from this discussion, and it is actually the way we work on Stigg’s AI Usage Management Runtime: a collection of components that build the critical path, the building blocks of usage runtimes for AI. We deliver the SDKs and all the connectors so you, or your agents, can easily connect it to your software. But a platform isn't always what you need.", "url": "https://wpnews.pro/news/million-dollars-in-tokens", "canonical_source": "https://thefinancialengineer.substack.com/p/milion-dollars-in-tokens", "published_at": "2026-09-10 15:32:56+00:00", "updated_at": "2026-09-10 15:47:17.513238+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-agents", "ai-products", "developer-tools"], "entities": ["OpenAI", "Astra", "Codex", "Claude Code"], "alternates": {"html": "https://wpnews.pro/news/million-dollars-in-tokens", "markdown": "https://wpnews.pro/news/million-dollars-in-tokens.md", "text": "https://wpnews.pro/news/million-dollars-in-tokens.txt", "jsonld": "https://wpnews.pro/news/million-dollars-in-tokens.jsonld"}}