{"slug": "don-t-put-all-your-tokens-in-one-basket", "title": "Don't Put All Your Tokens in One Basket", "summary": "AI service disruptions, rate limits, and silent model downgrades are becoming common, with providers like Anthropic and OpenAI tightening access and increasing costs. Developers are urged to adopt multi-provider strategies to mitigate risks from outages, pricing changes, and regulatory shifts.", "body_md": "[AI](https://www.theminimalistdeveloper.com/tags/ai/)/\n\n[Resiliency](https://www.theminimalistdeveloper.com/tags/resiliency/)\n\n# Don't Put All Your Tokens in One Basket\n\nAI is present everywhere these days, whether you like it or not, especially in our industry.\n\nThe degree of reliance goes to extremes, to the point where a service disruption means no work gets done at all.\n\nI am not here to debate the pros and cons of AI in development, but to explore alternative strategies to reduce risks and tame the costs of money-hungry AI machines.\n\n## What we are trying to avoid\n\nOutages continue to be a regular occurrence. [Anthropic's status page](https://status.anthropic.com) paints a good picture of the situation, with incidents practically every day. [GitHub Copilot](https://blog.incidenthub.cloud/github-reliability-outage-history-2025-2026) struggles to keep up with demand, just to name a few.\n\nRate limits have tightened as providers shift from heavily subsidized to trying to (unsuccessfully) monetise.\n\nIn early 2025, OpenAI lowered default tokens-per-minute caps for new accounts, and Anthropic introduced stricter tier limits that require [escalated payment plans to bypass](https://finance.biggo.com/news/aHDqzJ0BDPbb-ItTijCV)\n\nRegional access remains unpredictable. Anthropic updated its list of supported countries in early 2025, [dropping service in several markets](https://www.anthropic.com/news/updating-restrictions-of-sales-to-unsupported-regions) without advance notice, while OpenAI's terms continue to shift in response to evolving data sovereignty rules. A global product can find its core feature legally inaccessible in a key jurisdiction overnight not because the code changed, but because a compliance checkbox was unticked.\n\nNot to mention the increasing involvement of governmental entities actively in \"regulation\" with less-than-opaque reasoning, like the debacle around Anthropic's [Fable 5 (their latest model family)](https://www.anthropic.com/news/fable-mythos-access).\n\nIn a multi-jurisdiction world, trusting a single provider's availability map is an operational gamble.\n\nThose are the headlines, broadcasted everywhere. There are subtle and more prevalent issues that can be just as damaging for teams that heavily rely on AI.\n\n### Silent Downgrades\n\nThe API stays identical, but the model behind it doesn't. Your prompts stop working as expected, and nobody sends a changelog. We see this particular example with the Fable 5 now back in the picture, which [will fall back to Opus 4.8 for various tasks](https://platform.claude.com/cookbook/fable-5-fallback-billing-guide).\n\nThis could be due to a myriad of reasons:\n\n- A/B testing\n- Cost optimization\n- Safety or policy updates.\n\nThe reason is not really important for us, LLM's are not known for their determinism when you add all those extra variables on top it doesn't help.\n\n### Creeping billing\n\nUnlike the early days, the [open buffet of tokens is closed](https://www.forbes.com/sites/jemmagreen/2026/07/02/ai-costs-more-than-the-people-it-replaced/). We are seeing the landscape shift from attracting users to desperately showing the world that the AI money-burning-machine can be a sustainable business model.\n\nProviders employ many tactics to make sure their IPO is within reach: they deprecate old model IDs, or quietly remove the most cost-efficient variants from their catalogues. Your prompts still work, but the underlying model has been swapped to a more expensive one without a notice period.\n\nIn 2025, OpenAI began enforcing usage tiers that unlock cheaper rates only after a monthly spend threshold, meaning small teams and startups pay a premium simply for being small. The economies of scale don't get passed down; they get priced out.\n\n[Token pricing](https://developers.openai.com/api/docs/pricing) has also been updated to a more complex and difficult-to-predict model: input, output, and even per-route within the same model family have different prices.\n\n## Strategies\n\nThe following strategies are not exclusive; they work best when combined into a layered approach that matches your team's risk tolerance and budget.\n\n### Standardise on the OpenAI API format\n\nOpenAI API design has become the de facto standard for LLM interaction. Most providers and local runtimes (Ollama, llama.cpp servers, vLLM, etc.) support this interface, assuming your stack can target it.\n\nWrite your code against the interface, not against a specific vendor. Swapping providers becomes a simple configuration change, not a code rewrite. This decoupling protects you from vendor lock-in and enables flexible experimentation.\n\n### Choose the right tool for the job\n\nNot everything requires a frontier bazooka. Most people would be surprised how far smaller models have come. Running locally (or self-hosted) won't beat frontier models on raw capability, but they don't disappear, don't raise prices, and don't relay information home. For a lot of tasks—classification, summarisation, structured extraction—they're already good enough.\n\nConsider a tiered approach for models:\n\nBaseline models, where you can balance speed and performance for most of everyday tasks: Claude Sonnet 4.6, GPT-4o, DeepSeek v3.2\n\nSmaller, quicker models for completion or simple tasks: Qwen2.5-Coder 1.5B, DeepSeek-Coder 3B\n\nFrontier models, for tasks that require deep, nuanced understanding, like planning a difficult technical solution: Claude Opus 4.8, DeepSeek v4-pro, GPT-5.5\n\nIt is prudent to also maintain a chain of fallbacks with different providers mechanism ensuring that your core functionality remains robust and independent.\n\n### Abstract the provider behind your own layer\n\nDon't scatter SDK calls across your codebase. Route everything through one thin layer you control, ensuring a single point of management for all external interactions.\n\nWith this approach, you can easily switch models per task or per cost budget, add retries, timeouts, and a local fallback in one place, and log and evaluate outputs to catch silent downgrades.\n\n#### LiteLLM\n\nYou don't have to build this layer yourself. There are a few robust projects like [LiteLLM](https://github.com/BerriAI/litellm) which acts as a proxy and SDK that exposes a single OpenAI-compatible interface in front of 100+ providers—OpenAI, Anthropic, Bedrock, local Ollama, you name it.\n\n- One interface, many backends: swap providers with a config change, not a rewrite.\n- Built-in retries, fallbacks, and per-model routing covering the abstraction concerns above, off the shelf.\n- Cost tracking and rate limiting in one place, so you can catch a \"premium tier\" surprise before it hits the invoice.\n\nThe point isn't LiteLLM specifically, it's that your code should talk to *your* boundary, and let something like this sit behind it.\n\n### Own your data and prompts\n\nKeep prompts, evals, and any fine-tuning data versioned in your own repo. If you can rebuild your behaviour on a different model, you're portable.\n\n## Conclusion\n\nRedundancy isn't paranoia, it's the same engineering discipline you'd apply to any critical dependency. You wouldn't build on a single database with no backups and no migration path. AI providers deserve exactly that skepticism.", "url": "https://wpnews.pro/news/don-t-put-all-your-tokens-in-one-basket", "canonical_source": "https://www.theminimalistdeveloper.com/dont-put-all-your-tokens-in-one-basket/", "published_at": "2026-07-13 07:19:24+00:00", "updated_at": "2026-07-13 07:35:17.228134+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-policy", "ai-infrastructure", "ai-tools"], "entities": ["Anthropic", "OpenAI", "GitHub Copilot", "Fable 5", "Opus 4.8"], "alternates": {"html": "https://wpnews.pro/news/don-t-put-all-your-tokens-in-one-basket", "markdown": "https://wpnews.pro/news/don-t-put-all-your-tokens-in-one-basket.md", "text": "https://wpnews.pro/news/don-t-put-all-your-tokens-in-one-basket.txt", "jsonld": "https://wpnews.pro/news/don-t-put-all-your-tokens-in-one-basket.jsonld"}}