# Vercel AI Gateway Virtual Models: Swap Providers Safely

> Source: <https://dev.to/mudassirworks/vercel-ai-gateway-virtual-models-swap-providers-safely-33e9>
> Published: 2026-11-19 18:41:30+00:00

**Recommendation:** Route every model call through one wrapper that logs requested versus response model IDs, and encode a fallback policy that retries only timeouts, 429s and 5xx errors.
More in the playbook "Running AI in production": <https://wpnews.pro/playbooks/production-ops>

Vercel just shipped virtual model IDs on AI Gateway, so you can change the provider, the fallback chain and the model behind a request centrally, without deploying a single line of application code. That's a great escape hatch. It's also a new way to change production behavior with no pull request, no diff and no test run. This post is the guardrail kit I'd wire up before the first person flips a route in the dashboard.

**TL;DR**

Here's the change in one sentence: your code asks for a stable name, and the gateway decides what that name means today. If you've read my guide to [LLM model routing for cost and quality](https://mudassirkhan.me/blog/llm-model-routing), this is the same idea moved out of your codebase and into the platform.

The first rule is boring and it matters most: the model ID lives in exactly one place. Not in five route handlers, not in a prompt file, not in a feature flag someone forgot about.

```
// src/ai/models.ts
// One source of truth for which model name the app asks for.
// AI_MODEL_ID holds the virtual model ID you created in the AI Gateway dashboard.
// AI_MODEL_PINNED holds a concrete provider/model ID you trust, used for evals and rollback.

function requireEnv(name: string): string {
  const value = process.env[name];
  if (!value) throw new Error(`Missing env var ${name}`);
  return value;
}

export const MODELS = {
  chat: requireEnv("AI_MODEL_ID"),
  pinned: requireEnv("AI_MODEL_PINNED"),
} as const;

export type ModelRole = keyof typeof MODELS;
```

Then every call goes through one function:

``` js
// src/ai/generate.ts
import { generateText } from "ai";
import { MODELS, type ModelRole } from "./models";

export async function ask(prompt: string, role: ModelRole = "chat") {
  const requested = MODELS[role];
  const result = await generateText({ model: requested, prompt });

  // Log both sides: what you asked for and what the response says answered.
  console.log(
    JSON.stringify({
      requested_model: requested,
      response_model: result.response.modelId,
      input_tokens: result.usage.inputTokens,
      output_tokens: result.usage.outputTokens,
    })
  );
  return result.text;
}
```

The AI SDK sends a plain string model ID through AI Gateway when `AI_GATEWAY_API_KEY` is set. Check in your own logs what `response.modelId` reports for a virtual ID before you build dashboards on it. I'd rather you see it with your own eyes than trust my guess about it.

Fallbacks sound like free reliability. They aren't. A fallback is a second model answering a question you tuned for the first one, so you need a contract that says when that's allowed.

Cloudflare AI Gateway documents the same pattern from the other side: a multistep fallback chain that moves to the next model after a request fails or times out, plus a `cf-aig-step` response header that tells you which step answered (step 0 is the primary, then 1, then 2). Whichever gateway you run, that's the shape to copy: a fallback happens for a reason you named, and the response tells you it happened.

My rule of thumb:

| Error | Fall back? | Why | 
|---|---|---|
| Timeout | Yes | The provider is slow, your prompt is fine | 
| 429 rate limit | Yes | Capacity problem, not a content problem | 
| 5xx from the provider | Yes | Their outage, not your bug | 
| 400 bad request | No, fail fast | A different model won't fix a malformed request | 
| Policy refusal | No, fail fast and log | Silently retrying elsewhere hides a real decision | 

If part of your stack still does its own retries (a second gateway, an older code path), make that rule explicit in code so the two layers don't fight:

```
// src/ai/fallback-policy.ts
export type FailureKind = "timeout" | "rate_limit" | "provider_down" | "bad_request" | "refused";

export function classify(status: number | undefined, err: unknown): FailureKind {
  if (err instanceof DOMException && err.name === "TimeoutError") return "timeout";
  if (status === 429) return "rate_limit";
  if (status !== undefined && status >= 500) return "provider_down";
  if (status === 400 || status === 422) return "bad_request";
  return "refused";
}

const MAY_FALL_BACK: ReadonlySet<FailureKind> = new Set(["timeout", "rate_limit", "provider_down"]);

export function mayFallBack(kind: FailureKind): boolean {
  return MAY_FALL_BACK.has(kind);
}

// Usage with a hard timeout on the request:
// const res = await fetch(url, { signal: AbortSignal.timeout(15_000), ... });
```

Pick your fallback models on purpose, too. A fallback should be a model you've already evaluated for that job, not whatever was cheapest that afternoon. My notes on [choosing between Anthropic, OpenAI and Google models for agents](https://mudassirkhan.me/blog/llm-comparison-agentic-ai) are a decent starting list for that comparison.

Once routing lives in a dashboard, your traces are the only diff you get. OpenTelemetry publishes GenAI semantic conventions (version 1.44.0 at the time of writing) for exactly this: standard attribute names for model calls, so any backend can read them.

``` js
// src/ai/traced.ts
import { trace, SpanStatusCode } from "@opentelemetry/api";
import { generateText } from "ai";
import { MODELS, type ModelRole } from "./models";

const tracer = trace.getTracer("ai-gateway-client");

export async function tracedAsk(prompt: string, role: ModelRole = "chat") {
  const requested = MODELS[role];
  return tracer.startActiveSpan(`chat ${requested}`, async (span) => {
    span.setAttribute("gen_ai.operation.name", "chat");
    span.setAttribute("gen_ai.request.model", requested);
    try {
      const result = await generateText({ model: requested, prompt });
      span.setAttribute("gen_ai.response.model", result.response.modelId);
      span.setAttribute("gen_ai.usage.input_tokens", result.usage.inputTokens ?? 0);
      span.setAttribute("gen_ai.usage.output_tokens", result.usage.outputTokens ?? 0);
      return result.text;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}
```

The query that matters: group by `gen_ai.request.model` and `gen_ai.response.model`. When someone edits a route, the pairs change, and you see it without anyone posting in Slack.

Rolling back a virtual route is a dashboard action. Knowing whether you need to is the hard part. Keep a pinned model in your config and a small fixed eval set, and run both IDs side by side whenever a route changes:

``` js
// scripts/route-eval.ts  (run with: npx tsx scripts/route-eval.ts)
import { generateText } from "ai";

const virtualId = process.env.AI_MODEL_ID!;
const pinnedId = process.env.AI_MODEL_PINNED!;

const cases = [
  { prompt: 'Return JSON {"intent": string} for: "cancel my order 4417"', mustInclude: ["intent"] },
  { prompt: 'Return JSON {"refund": boolean} for: "I want my money back"', mustInclude: ["refund"] },
];

function passes(text: string, keys: string[]): boolean {
  try {
    const obj = JSON.parse(text.replace(/```

json|

```/g, "").trim());
    return keys.every((k) => k in obj);
  } catch {
    return false;
  }
}

async function score(model: string) {
  let ok = 0;
  for (const c of cases) {
    const { text } = await generateText({ model, prompt: c.prompt });
    if (passes(text, c.mustInclude)) ok++;
  }
  return ok / cases.length;
}

const [v, p] = await Promise.all([score(virtualId), score(pinnedId)]);
console.log({ virtual: v, pinned: p });
if (v < p) {
  console.error("Virtual route scores below the pinned model. Roll the route back.");
  process.exit(1);
}
```

Two cases is a toy. Twenty real ones from your logs is a gate. Wire it into CI on a schedule, and run it by hand right after any route edit.

Here's my practitioner opinion: the teams that get burned by central routing aren't the ones without fallbacks. They're the ones who can't answer "which model answered this ticket last Tuesday?" Fix that first.

`gen_ai.response.model` logged on every call
That last line is where routing pays for itself or doesn't. If you're still sizing what an agent feature will cost to run, the [agentic AI MVP cost estimator](https://mudassirkhan.me/tools/agentic-ai-mvp-cost-estimator) gives you a first number to compare your traces against.

**How does Vercel AI Gateway virtual models work?**

Your code asks for one stable virtual model ID. The gateway maps that ID to a provider, a model and a fallback chain that you manage centrally, so changing what answers the request doesn't need an application deploy.

**What changes for teams adopting Vercel AI Gateway virtual models?**

The change control moves. Model switches stop showing up as code diffs, so you need traces that record which model answered, a named list of errors that may fall back, and an eval you run around every route change.

**Do I still need retries in my own code?**

Only where something else calls providers directly. If the gateway owns fallbacks, keep your own layer to timeouts and logging so you don't retry twice.

Related reading: [why model routing needs its own evaluation scorecard](https://medium.com/@mudassir00seven/copilot-fusion-is-live-why-model-routing-needs-its-own-evaluation-scorecard-f74a77bc590f) and [how routing cut my inference bill without a quality drop](https://medium.com/p/9e1c3cec80d8).

*How are you handling route changes today: dashboard only, or a gate in CI? Drop your setup in the comments, I'm curious what's working in production.*
