Don't Put All Your Tokens in One Basket

AI service disruptions, rate limits, and silent model downgrades are becoming common, with providers like Anthropic and OpenAI tightening access and increasing costs. Developers are urged to adopt multi-provider strategies to mitigate risks from outages, pricing changes, and regulatory shifts.

AI https://www.theminimalistdeveloper.com/tags/ai/ / Resiliency https://www.theminimalistdeveloper.com/tags/resiliency/ Don't Put All Your Tokens in One Basket AI is present everywhere these days, whether you like it or not, especially in our industry. The degree of reliance goes to extremes, to the point where a service disruption means no work gets done at all. I am not here to debate the pros and cons of AI in development, but to explore alternative strategies to reduce risks and tame the costs of money-hungry AI machines. What we are trying to avoid Outages continue to be a regular occurrence. Anthropic's status page https://status.anthropic.com paints a good picture of the situation, with incidents practically every day. GitHub Copilot https://blog.incidenthub.cloud/github-reliability-outage-history-2025-2026 struggles to keep up with demand, just to name a few. Rate limits have tightened as providers shift from heavily subsidized to trying to unsuccessfully monetise. In early 2025, OpenAI lowered default tokens-per-minute caps for new accounts, and Anthropic introduced stricter tier limits that require escalated payment plans to bypass https://finance.biggo.com/news/aHDqzJ0BDPbb-ItTijCV Regional access remains unpredictable. Anthropic updated its list of supported countries in early 2025, dropping service in several markets https://www.anthropic.com/news/updating-restrictions-of-sales-to-unsupported-regions without advance notice, while OpenAI's terms continue to shift in response to evolving data sovereignty rules. A global product can find its core feature legally inaccessible in a key jurisdiction overnight not because the code changed, but because a compliance checkbox was unticked. Not to mention the increasing involvement of governmental entities actively in "regulation" with less-than-opaque reasoning, like the debacle around Anthropic's Fable 5 their latest model family https://www.anthropic.com/news/fable-mythos-access . In a multi-jurisdiction world, trusting a single provider's availability map is an operational gamble. Those are the headlines, broadcasted everywhere. There are subtle and more prevalent issues that can be just as damaging for teams that heavily rely on AI. Silent Downgrades The API stays identical, but the model behind it doesn't. Your prompts stop working as expected, and nobody sends a changelog. We see this particular example with the Fable 5 now back in the picture, which will fall back to Opus 4.8 for various tasks https://platform.claude.com/cookbook/fable-5-fallback-billing-guide . This could be due to a myriad of reasons: - A/B testing - Cost optimization - Safety or policy updates. The reason is not really important for us, LLM's are not known for their determinism when you add all those extra variables on top it doesn't help. Creeping billing Unlike the early days, the open buffet of tokens is closed https://www.forbes.com/sites/jemmagreen/2026/07/02/ai-costs-more-than-the-people-it-replaced/ . We are seeing the landscape shift from attracting users to desperately showing the world that the AI money-burning-machine can be a sustainable business model. Providers employ many tactics to make sure their IPO is within reach: they deprecate old model IDs, or quietly remove the most cost-efficient variants from their catalogues. Your prompts still work, but the underlying model has been swapped to a more expensive one without a notice period. In 2025, OpenAI began enforcing usage tiers that unlock cheaper rates only after a monthly spend threshold, meaning small teams and startups pay a premium simply for being small. The economies of scale don't get passed down; they get priced out. Token pricing https://developers.openai.com/api/docs/pricing has also been updated to a more complex and difficult-to-predict model: input, output, and even per-route within the same model family have different prices. Strategies The following strategies are not exclusive; they work best when combined into a layered approach that matches your team's risk tolerance and budget. Standardise on the OpenAI API format OpenAI API design has become the de facto standard for LLM interaction. Most providers and local runtimes Ollama, llama.cpp servers, vLLM, etc. support this interface, assuming your stack can target it. Write your code against the interface, not against a specific vendor. Swapping providers becomes a simple configuration change, not a code rewrite. This decoupling protects you from vendor lock-in and enables flexible experimentation. Choose the right tool for the job Not everything requires a frontier bazooka. Most people would be surprised how far smaller models have come. Running locally or self-hosted won't beat frontier models on raw capability, but they don't disappear, don't raise prices, and don't relay information home. For a lot of tasks—classification, summarisation, structured extraction—they're already good enough. Consider a tiered approach for models: Baseline models, where you can balance speed and performance for most of everyday tasks: Claude Sonnet 4.6, GPT-4o, DeepSeek v3.2 Smaller, quicker models for completion or simple tasks: Qwen2.5-Coder 1.5B, DeepSeek-Coder 3B Frontier models, for tasks that require deep, nuanced understanding, like planning a difficult technical solution: Claude Opus 4.8, DeepSeek v4-pro, GPT-5.5 It is prudent to also maintain a chain of fallbacks with different providers mechanism ensuring that your core functionality remains robust and independent. Abstract the provider behind your own layer Don't scatter SDK calls across your codebase. Route everything through one thin layer you control, ensuring a single point of management for all external interactions. With this approach, you can easily switch models per task or per cost budget, add retries, timeouts, and a local fallback in one place, and log and evaluate outputs to catch silent downgrades. LiteLLM You don't have to build this layer yourself. There are a few robust projects like LiteLLM https://github.com/BerriAI/litellm which acts as a proxy and SDK that exposes a single OpenAI-compatible interface in front of 100+ providers—OpenAI, Anthropic, Bedrock, local Ollama, you name it. - One interface, many backends: swap providers with a config change, not a rewrite. - Built-in retries, fallbacks, and per-model routing covering the abstraction concerns above, off the shelf. - Cost tracking and rate limiting in one place, so you can catch a "premium tier" surprise before it hits the invoice. The point isn't LiteLLM specifically, it's that your code should talk to your boundary, and let something like this sit behind it. Own your data and prompts Keep prompts, evals, and any fine-tuning data versioned in your own repo. If you can rebuild your behaviour on a different model, you're portable. Conclusion Redundancy isn't paranoia, it's the same engineering discipline you'd apply to any critical dependency. You wouldn't build on a single database with no backups and no migration path. AI providers deserve exactly that skepticism.