If you've built anything serious with LLM APIs, you've probably hit this pattern:
Not fun.
The cleanest version of this setup is simple: keep your app speaking the OpenAI API format, but route requests to different models behind the scenes.
That's the idea behind an OpenAI-compatible AI gateway.
Disclosure: I work on TokenBay, an AI model API gateway. This post is based on the setup I usually recommend when developers want access to multiple model families without rewriting their app every time they test a new provider.
Most AI apps start with one model provider.
That works fine until you need to optimize for cost, latency, quality, availability, or task type.
For example:
The painful part is not calling one API.
The painful part is maintaining five slightly different API integrations.
Instead of wiring every provider directly into your app, you can use an OpenAI-compatible gateway.
Your application keeps using the familiar chat.completions.create()
interface.
The only things you usually change are:
baseURL
apiKey
model
That means your app can switch between GPT, Claude, Gemini, Qwen, and other models while keeping most of your code unchanged.
Install the OpenAI SDK:
npm install openai
Create a file called index.js
:
import OpenAI from "openai";
const client = new OpenAI({
apiKey: process.env.TOKENBAY_API_KEY,
baseURL: "https://api.tokenbay.com/v1"
});
async function main() {
const response = await client.chat.completions.create({
model: "gpt-4o-mini",
messages: [
{
role: "system",
content: "You are a concise technical assistant."
},
{
role: "user",
content: "Explain model routing in one paragraph."
}
]
});
console.log(response.choices[0].message.content);
}
main().catch(console.error);
Run it:
TOKENBAY_API_KEY=your_api_key_here node index.js
If your app already uses the OpenAI SDK, the main migration is usually just the client config:
const client = new OpenAI({
apiKey: process.env.TOKENBAY_API_KEY,
baseURL: "https://api.tokenbay.com/v1"
});
That's the whole point.
You should not need a different SDK for every model family just to run a basic chat completion.
Once your app can talk to multiple models through the same interface, you can make routing decisions in code.
Here's a simple example:
function selectModel(taskType) {
switch (taskType) {
case "reasoning":
return "gpt-4o";
case "cheap_summary":
return "qwen-plus";
case "long_context":
return "claude-3-5-sonnet";
default:
return "gpt-4o-mini";
}
}
Then call the selected model:
async function runTask(taskType, userInput) {
const model = selectModel(taskType);
const response = await client.chat.completions.create({
model,
messages: [
{
role: "user",
content: userInput
}
]
});
return response.choices[0].message.content;
}
This is intentionally boring code.
Boring is good here.
You want model routing to be understandable, testable, and easy to change.
Provider outages happen. Rate limits happen. Weird transient API failures happen.
A basic fallback wrapper can save you from a lot of production pain:
async function completeWithFallback(messages) {
const models = [
"gpt-4o-mini",
"qwen-plus",
"gemini-1.5-flash"
];
let lastError;
for (const model of models) {
try {
const response = await client.chat.completions.create({
model,
messages
});
return {
model,
content: response.choices[0].message.content
};
} catch (error) {
lastError = error;
console.warn(`Model ${model} failed, trying next option...`);
}
}
throw lastError;
}
Example usage:
const result = await completeWithFallback([
{
role: "user",
content: "Summarize this customer support ticket in 3 bullet points."
}
]);
console.log(`Used model: ${result.model}`);
console.log(result.content);
This is not a full production-grade retry system, but it gives you the shape:
An OpenAI-compatible gateway is useful when you have multiple LLM workloads inside the same product.
Some common examples:
Not every task needs the most expensive model.
A lot of production AI work is routing the right request to the right model at the right cost.
This is where gateways become more than a convenience layer.
Once all requests pass through one API layer, you can start thinking about:
TokenBay is built around this workflow: one API key for major model families, OpenAI-compatible requests, and a usage dashboard so you can see what your app is actually spending.
In many cases, routing routine tasks to lower-cost models can reduce API spend without hurting the user experience.
The exact savings depend on your workload and model choices, so I would treat any blanket savings claim with suspicion. Test it against your own traffic.
When I look at an AI feature, I usually split calls into three buckets:
| Task | Model Strategy |
|---|---|
| User-facing reasoning | Use a stronger model |
| Summarization / tagging / extraction | Try cheaper models first |
| Background automation | Optimize heavily for cost and latency |
You do not need a complicated ML routing system on day one.
A plain function like selectModel(taskType)
is enough to start.
The best AI API architecture is usually the one that gives you room to change your mind.
Models change. Prices change. Latency changes. Your product changes.
If your code is tightly coupled to one provider, every model experiment becomes annoying.
If your app talks to one OpenAI-compatible interface, you can test and route across GPT, Claude, Gemini, Qwen, and others with much less friction.
That is the real win: not just saving money, but making model choice a runtime decision instead of a rewrite.
For this post, I used TokenBay as the OpenAI-compatible gateway example, but the same routing pattern applies to any gateway that supports the OpenAI API format.
I'd also be curious how other teams are handling model routing today. Are you using a gateway, building your own abstraction layer, or still calling each provider directly?