Writing a Contract Test Suite for Your Own LLM Gateway

If you run a proxy in front of one or more model providers, the interesting failures are not in the model. They are in your layer: a header you forwarded that you should have stripped, an upstream 429 you turned into a 500, a stream you accidentally buffered.

A gateway sits between your applications and one or more providers, and it promises two things at once. Downstream, it promises to speak the OpenAI-compatible contract so that ordinary SDKs work against it. Upstream, it promises to translate faithfully into whatever each provider actually wants. Almost every bug that costs you an incident lives in one of those two promises rather than in the inference.

That means the suite you want is not the one from evaluating a candidate provider. That suite asks whether a remote service is shaped correctly. This one asks whether your code preserves shape, and it should run without touching a provider at all: fast, deterministic, and on every pull request. The distinction matters because a suite that needs a real model is a suite that runs nightly at best, and a proxy bug found nightly has already shipped.

List the promises explicitly before writing a line. A typical set: the response envelope is preserved; the model alias resolves to the documented concrete model; the upstream key never appears downstream and the downstream key never appears upstream; usage is reported and equals what the upstream reported; an upstream error becomes a documented downstream error rather than a stack trace; and a streamed request produces a stream, not a buffered response delivered at the end.

The direction of the stub is the design decision. If you mock your own gateway you are testing nothing. What you want is a fake provider: a local HTTP server that returns whatever fixture the test needs, including malformed and hostile fixtures a real provider will not produce on demand. Mock Service Worker’s Node interceptor is the usual choice in TypeScript, and it works at the request level so your gateway’s real HTTP client, real timeouts and real retry logic all execute.

// gateway/contract/upstream.ts
import { setupServer } from "msw/node";
import { http, HttpResponse } from "msw";

export const upstream = setupServer();

const FIXTURE_ID = "chatcmpl-fixture-1";

export const completionFixture = (over = {}) => ({
  id: FIXTURE_ID,
  object: "chat.completion",
  created: 1754870400,
  model: "provider-model-v2",
  choices: [{
    index: 0,
    message: { role: "assistant", content: "ok" },
    finish_reason: "stop",
  }],
  usage: { prompt_tokens: 11, completion_tokens: 2, total_tokens: 13 },
  ...over,
});

export const respondsWith = (status: number, body: unknown) =>
  http.post("https://upstream.test/v1/chat/completions", () =>
    HttpResponse.json(body as never, { status }));

Two properties make this worth the setup. The fixture is a value, so a test can mutate one field and assert that your gateway notices. And the interceptor records the outbound request, so you can assert on what your gateway sent, which is half the contract and the half that is otherwise invisible.

The single most valuable assertion in a gateway suite is that unknown fields survive. A gateway that parses an upstream response into a typed struct and re-serialises it will silently drop every field its struct does not know about — and providers add fields constantly. Your users lose logprobs

, or a reasoning block, or a cache-hit count, and nobody notices until one of them opens a ticket.

it("preserves fields the gateway does not model", async () => {
  upstream.use(respondsWith(200, completionFixture({
    system_fingerprint: "fp_abc123",
    provider_specific_metadata: { cache_hit: true },
  })));

  const res = await fetch(gatewayURL + "/v1/chat/completions", {
    method: "POST",
    headers: { "content-type": "application/json", authorization: "Bearer tenant-key" },
    body: JSON.stringify({ model: "alias-fast", messages: [{ role: "user", content: "hi" }] }),
  });
  const body = await res.json();

  expect(body.system_fingerprint).toBe("fp_abc123");
  expect(body.provider_specific_metadata).toEqual({ cache_hit: true });
  expect(body.usage.total_tokens).toBe(13);
});

The companion assertion runs in the other direction: capture the request your gateway made upstream and check that the client’s parameters arrived intact. A gateway that rebuilds the request body from a known-parameter list drops the one parameter a client added last week, and the client sees a model that ignores their response_format

with no error anywhere.

Error mapping is a contract in both directions and it is the part teams write once and never test. Assert each upstream status your gateway can receive against the downstream status it should produce: a 429 must stay a 429 or your clients’ backoff never engages; a Retry-After

header must survive, since it carries the only number worth obeying; a 400 from the provider about an invalid parameter must not become a 502, because a 502 tells the client to retry a request that will fail identically forever.

Retry-After: 3

Header hygiene deserves its own explicit test because the failure is a security failure rather than an availability one. Assert that the downstream Authorization

value never appears in the captured upstream request, and that no upstream provider header that identifies your account is echoed downstream. Both are one-line assertions on captured objects and neither is caught by any other kind of test.

A gateway can pass every buffered test and still be broken for streaming, because the two are usually different code paths and the streaming one has a failure mode the buffered one cannot have: it can be correct in content and wrong in time. A proxy that reads the whole upstream stream, assembles it, and writes it out as SSE at the end produces byte-identical output and destroys the only reason anyone streams.

So assert on timing, not only on bytes. With a stub upstream you control the delay between chunks, which makes this deterministic rather than flaky: emit chunk one, wait, emit chunk two, and assert that your gateway delivered the first chunk before the second was sent. Record the wall-clock offset of the first downstream chunk and require it to be below the upstream’s inter-chunk delay. The detail of the wire format — the data:

prefix, the blank line separator, the terminating sentinel — is covered in contract tests for streaming chunk format, and a gateway suite should reuse the same parser its clients use rather than writing a second one.

Most of the work in operating a gateway is exactly this suite: the error mapping, the header hygiene, the pass-through fidelity, and the streaming path, maintained across every provider you add. If that layer is not itself the product you are building, Multigrid is that layer as a service — one API and one key, with the provider differences absorbed behind it.