41 billion tokens later: dogfooding local inference routing

Spectro Cloud reported that 85 of its engineers processed 41 billion tokens in a one-month pilot of its PaletteAI Inference Launchpad, with 40 billion tokens handled locally on a single server with eight NVIDIA B200 GPUs. Engineers in the pilot had direct Claude bills about 46% lower than colleagues outside the pilot, according to the company, which said it spends well over $100,000 on Claude per month. Spectro Cloud deployed Launchpad at the end of July, started with GLM 5.2 and moved to GLM 5.3 Flash in late August, routing Sonnet and Haiku requests locally while sending complex Opus requests to Claude Opus 5 under a per-engineer daily dollar limit it cut from $200 to $50.

41 billion tokens later: our dogfooding experience with PaletteAI Inference Launchpad Here at Spectro Cloud, our software engineers and many other power-user roles too are heavy users of AI coding tools, and our Claude bill has been growing dramatically every month, to our finance team's alarm. Familiar story? This is exactly the situation we developed PaletteAI Inference Launchpad https://www.spectrocloud.com/platform/inference-launchpad for. Inference Launchpad is our software appliance for running open models locally, on your own GPU hardware. It sits between coding tools and models, serving inference requests locally where policy requires, routing selected work to a hosted service such as Claude. Engineers keep using their familiar coding assistants, while the platform team controls where requests go and how much they can spend. Like any good software company, we believe in drinking our own champagne or eating our own dog food, you pick . We made ourselves 'customer zero' for Inference Launchpad, seeing whether it would really work for demanding engineers: not just saving money, but hitting our quality expectations, too. The headline: in a one month pilot, 85 of our engineers processed 41 billion tokens, 40 billion of which were handled locally using Inference Launchpad. Engineers trying Inference Launchpad had direct Claude bills about 46% lower than colleagues outside the pilot. That gave us an encouraging first set of results, and enough everyday engineering work to examine how the local model performed. Up to 85 engineers on one eight-GPU server We're pretty open about how much we spend on frontier models. Although we're a relatively small organization, we spend well over $100,000 on Claude per month, most through Claude Code. Moving just some of that work to a shared local model would give us a way to reduce reliance on per-token API spending as our use of AI coding continues to grow. At the end of July, our team deployed Launchpad on a single server with eight NVIDIA B200 GPUs. We started with GLM 5.2, moved to GLM 5.3 Flash in late August, and rolled access out to engineering teams in waves. Engineers kept their existing coding tools Keeping tools like Claude Code was important. It meant we could onboard engineers into the pilot without asking them to learn a different coding assistant first. For Claude Code, we pointed the client at the Launchpad endpoint and replaced the Anthropic key with a per-user Launchpad token. The example below shows the configuration used in our pilot; the Claude Code setup guide https://docs.spectrocloud.com/paletteai-inference-launchpad/how-to-guides/use-claude-code/ covers connecting to your own deployment. export ANTHROPIC BASE URL=https://<your-launchpad-endpoint export ANTHROPIC AUTH TOKEN='<per-user-token ' export ANTHROPIC MODEL=claude-opus-5 export ANTHROPIC DEFAULT OPUS MODEL=claude-opus-5 export ANTHROPIC DEFAULT SONNET MODEL=claude-sonnet-4-5 export ANTHROPIC DEFAULT HAIKU MODEL=claude-haiku-4-5 export CLAUDE CODE EFFORT LEVEL=auto export CLAUDE CODE MAX OUTPUT TOKENS=64000 The model name requested by the coding tool doesn't have to be the model that serves it. Our centrally managed routing map https://docs.spectrocloud.com/paletteai-inference-launchpad/how-to-guides/configure-semantic-routing/ decides where each request goes: - Requests for Sonnet or Haiku go to local GLM 5.3 Flash, with reasoning effort set to high or low respectively. - Opus requests are classified locally. Complex requests go to Claude Opus 5 through a company-owned Anthropic key, subject to a daily dollar limit per engineer. We started at $200, then cut it to $50 — most users didn't notice a difference. Other requests stay local. - In our pilot, reaching the daily limit sends subsequent requests to GLM until the next day; if we hit our total monthly frontier budget, all user requests fall back to GLM until the next month. This all happens transparently. That arrangement puts a real, hard limit on external spending through Launchpad while keeping the local model available. Engineers can continue working after reaching their allowance, and the team can adjust routing centrally as it learns which tasks suit each model. 40 billion local tokens and lower Claude bills The 30-day report covers August 15 to September 14, and involved a growing number of users. From an initial team, we expanded the pilot eventually up to 85 engineers using the local service during that period, in total processing 41 billion tokens between them — only 1 billion of which ended up being routed to the frontier. That's 97.5% of work happening on the box. Almost all our user requests over 99% were for Opus — this is human nature, and it shows that our engineers haven't yet set up a harness to break up tasks across multiple models. That means that we offloaded over 40 billion tokens from Claude Opus 5: - 39.5B input tokens @ $0.76/M blended rate = $30k - 830M output tokens @ $25/M = $20k Even accounting for the cost of our B200 box, saving $50k of Claude tokens in one month is a huge deal. For Ray Krueger, one of our esteemed Principal Architects, there was another benefit. He used local GLM to write an AWS Local Zone Terraform module and described why that mattered: "The model was never the interesting part. The interesting part is that my Terraform state, my AWS keys, and my Palette token never left the trust boundary they were issued in." Ray has also recorded a Launchpad demo https://www.youtube.com/watch?v=YGonL0-Ccvo using Claude Code with local models. Watch it to see how the workflow looks from a developer's side. Engineers who had Inference Launchpad access for the full reporting period averaged $714 in direct Claude spending, compared with $1,317 for colleagues who hadn't started using it: about 46% less. That's an encouraging early comparison between groups. The next step is to track spending for the same engineers over time as adoption grows. Cached-token pricing matters to the cost comparison Coding agents reuse context across requests, including instructions and information about the codebase. With prompt caching https://platform.claude.com/docs/en/build-with-claude/prompt-caching , reading that reused input costs less than processing it afresh. A local-versus-hosted cost comparison needs to reflect the price you actually pay for those cache hits. In our Opus traffic, more than 95% of input tokens came from Anthropic's prompt cache. The team's calculation put the saving from caching at about 83% against uncached pricing. That made cache accounting a significant part of evaluating the pilot's economics. The pilot helped us refine Launchpad's savings estimates. The dashboard's initial comparison used uncached external-model prices, which overstated the benefit for this workload. We adjusted the assumed input price to $0.75 per million tokens based on our Anthropic statement while work continued on accounting for reads from the local prefix cache. For your own evaluation, use your provider invoice as the starting point and include cached-token pricing in the comparison. Our AI inference TCO calculator https://www.spectrocloud.com/ai-tco can help you explore how local infrastructure costs compare with hosted usage. Can local models do the work our engineers need? When we demonstrate Launchpad to customers and at events, one question comes up regularly: can an open model running locally deliver the quality people expect from Claude or OpenAI's models? We checked it against our own work. The team responsible for the core platform backend evaluated GLM 5.3 Flash through Inference Launchpad alongside direct Claude Opus 5, using tickets from its backlog. GLM handled a security-hardening batch from the findings in git through to fixes and Jira tickets, without falling back to an external model. The engineer described it as a "practical daily driver," with better speed and use of context than GLM 5.2. The local model could follow a complete engineering task through the code and the tools around it. For another task, GLM drafted an implementation plan and Opus checked it against the codebase. Opus identified a missing operationId, and GLM then carried out the plan with no rework. This is a useful way to combine models within one task: do the planning and implementation locally, and use Claude for an additional review. We also tested a more demanding use case: an unattended agent that reads the results of nightly tests and matches failures to Jira tickets. It needs to distinguish a familiar failure from a new regression that deserves investigation. With no developer reviewing each decision, the consequences of a missed regression are different from a mistake in an interactive coding session. Across 27 runs, overall classification accuracy was similar: 89% for Claude direct, 94% for Claude through Launchpad and 92% for GLM through Inference Launchpad. No invented ticket was reported in any of those runs. Heavy daily use is helping us improve the product Using Inference Launchpad ourselves gives us specific cases to improve. Prompt-injection checks matter when coding agents work with content from other tools. Our filter sometimes flagged legitimate content, including a permission prompt and a login URL. Those cases are helping us tune the checks so they can remain cautious about suspicious input while causing fewer interruptions. We also learned to check how each coding tool picks up its configuration. Some IDE extensions weren't reading the environment variables set in the terminal and continued connecting directly to Anthropic through personal logins. Those requests never reached Launchpad. The onboarding check is to verify the endpoint used by the running IDE as well as the terminal, so both use the intended service. The initial deployment with GLM 5.2 on one eight-B200 server supported roughly 15-25 concurrent users, which can reasonably serve a group of 50 people in different timezones. After switching to GLM 5.3 Flash we are supporting around 50 concurrent users, which allows a group of 100 people across different timezones to use the machine. These are heavy power users, and model tuning is helping us get more out of the same hardware. The range is a starting point for capacity planning; your workload and serving configuration will affect how many people a node can support. With more than 100 engineers to onboard, we expanded access in waves while tuning the deployment. Work on the inference engine and model configuration is feeding back into the product defaults, so the lessons from this pilot can benefit the next deployment. Try Inference Launchpad with the work your team does The pilot has given our engineers a working way to run substantial coding workloads locally and keep Claude available for the tasks that benefit from it. The early spending figures are very encouraging, and the completed engineering tasks prove that local models can be useful today. We're extending access to more teams and carrying what we learn into the product. If you're curious about what this could look like for your team, talk to us about Inference Launchpad https://www.spectrocloud.com/get-started . Bring a representative coding task and your current usage figures so we can explore the model quality and economics with you. We have a sandbox ready to go.