One endpoint is convenient. One endpoint that quietly rewrites your model request is an outage waiting for a release you did not control.
An agent works perfectly in a direct provider sandbox. It streams, calls tools, keeps a long context warm, and returns a strict object your application can trust. Then the team moves it behind an LLM gateway to centralize billing and policy. A new model setting disappears. A usage field no longer arrives. A streaming tool loop stalls. Nothing crashes loudly enough to make the cause obvious.
This is the hidden trade-off in multi-provider AI: a gateway can make model choice easier while making capability loss harder to see. The problem matters more now that OpenAI has announced that frontier models and Codex workflows are coming to Google Cloud Model Garden. More deployment routes are useful. They also add another place where an application, a gateway, and an upstream model can disagree about what a request means.
The practical answer is not to avoid gateways. It is to treat a gateway as a compatibility boundary and test it like one. This guide explains LLM gateway passthrough vs unified API design, the contract tests that catch silent loss, and a safe way to evaluate a new provider route before it serves real traffic.
The key idea: normalize the controls that should be common — identity, budgets, audit logs, and routing policy. Preserve or explicitly reject the features whose meaning belongs to a specific model provider.
For years, “pick a model provider” often meant “pick a cloud, SDK, billing account, and operational lane.” That assumption is weakening. OpenAI’s October announcement says its frontier models and Codex workflows will be available through Google Cloud Model Garden, alongside Google Cloud governance and Marketplace purchasing. The announcement is an access and procurement change, not proof that every direct API feature, region, rate limit, or client workflow is identical on day one.
That distinction is easy to miss. A model name may be familiar while the route is new. Authentication, quotas, billing attribution, tool support, streaming behavior, retention controls, and logs can be different at the brokered route. A gateway in front adds another translation layer.
Current practitioner discussions describe the same concern in blunter terms: a universal request shape is pleasant until a gateway silently strips a provider-specific option. Teams also report that per-request switching can erase valuable prompt-cache reuse in long agent sessions. These are not arguments against model choice. They are reasons to define what must survive the choice.
A passthrough API accepts the native request shape for a provider and forwards the body, headers, and streaming response with minimal change. It is the safer option when the application relies on a provider’s newest tool format, a reasoning setting, cache control, a response continuation ID, or an unusual media feature. The gateway can still authenticate the caller, apply a budget, record an audit event, and choose an allowed upstream route around that request.
A unified API presents one common schema across providers. It is excellent for workloads that genuinely need interchangeable, basic operations: short text classification, one-shot extraction, embeddings, or a clean fallback for a non-critical chat feature. A unified surface can reduce integration code and make a controlled experiment much easier.
The dangerous belief is that a unified schema automatically creates semantic equivalence. It does not. A field can be accepted but ignored. A gateway can map it to a weaker alternative. It can buffer a stream, reshape a tool result, omit usage detail, or fail over an in-progress session to a model that cannot continue it.
Use a two-surface design instead. Offer a unified route for portable workloads, and native passthrough routes for capability-sensitive workloads. Then give callers an explicit promise about which route they are using. “Compatible” should never mean “probably close enough.”
A model catalog tells a team what it can call. A capability contract tells the team what it may safely depend on. Write the contract per workload, rather than declaring that an entire provider is interchangeable.
For example, a support-answer workload may require only text, citations, a maximum latency, and a JSON response. A coding agent may require streamed events, parallel tool calls, a durable continuation mechanism, repository-sized context, token-usage events, and a stable way to cancel a run. Those are different contracts even if both call the same underlying model.
Write down five things for each route:
The most useful part is the forbidden list. A coding agent with provider-bound state should not be quietly moved to another provider halfway through a task merely because a router found a cheaper path. A financial extraction job should not fall back from strict structured output to prose and hope a repair parser can rescue it.
Do not test a gateway with “hello world.” Build a small probe suite from the exact features your workload needs. The test should run the same fixture directly against the intended provider route and through the gateway. It should compare observable behavior, not hidden model reasoning.
Here is a deliberately small Python pattern. In production, use your own client adapters and redact sensitive fixtures. The important part is the assertion: an unsupported feature should produce an explicit, typed rejection — not a successful response with the feature discarded.
from dataclasses import dataclass
@dataclassclass Probe: name: str request: dict must_observe: set[str]
PROBES = [ Probe("strict-json", { "input": "Return an object with ok=true", "response_format": {"type": "json_schema", "json_schema": { "name": "verdict", "schema": { "type": "object", "properties": {"ok": {"type": "boolean"}}, "required": ["ok"], "additionalProperties": False}}}, {"valid_json", "schema_match"}), Probe("tool-stream", { "input": "Use the weather tool for Pune, then summarize.", "tools": [{"type": "function", "name": "weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}], "stream": True}, {"tool_call", "stream_complete", "usage"}),]
python
def assert_contract(result, probe): seen = set(result["observations"]) missing = probe.must_observe - seen if missing: raise AssertionError(f"{probe.name}: missing {sorted(missing)}") if result.get("dropped_fields"): raise AssertionError(f"{probe.name}: gateway dropped fields")
for probe in PROBES: direct = run_probe("direct-provider", probe.request) routed = run_probe("gateway-native-route", probe.request) assert_contract(direct, probe) assert_contract(routed, probe)
Add probes only when they represent a real product promise. Good candidates include tool calls, parallel tool results, a strict schema, cached-token accounting, cancellation, timeouts, continuation after a tool result, and a provider-specific setting your agent relies on. For each test, capture the effective upstream, gateway version, model version, response status, error class, and a safe hash of the fixture.
Test the negative path too. If the unified route cannot support a provider feature, it should return something like unsupported_capability before work begins. A clear 4xx response is cheaper than a quiet behavior change that surfaces in an incident review.
Streaming is where friendly demos turn into difficult integrations. A gateway may send all the visible text while losing the final usage event. It may reorder tool-call arguments, buffer events long enough to break a UI, or restart a request without explaining that a partially emitted result was abandoned.
Your probe should record the event sequence. Assert that a tool call includes a stable ID, that arguments are complete before execution, that the terminal event arrives, and that usage appears where your meter expects it. If the upstream sends usage only in a final event, test a cancelled stream and a delayed stream as well. “The text looked fine” is not proof that your accounting or agent loop is correct.
When a native route uses provider-bound reasoning artifacts or signatures, treat a change of provider as a state migration, not an HTTP retry. The safe default is session stickiness: choose a route when a stateful session starts, keep it for that session, and re-route only at an explicit, replayable checkpoint. Reserve per-request routing for independent, short tasks where cache loss and conversation continuity do not matter.
As of the announcement, general availability is planned rather than a universal promise today. That makes this an ideal time to prepare a test packet, not to assume a production configuration. Ask the provider for the model and region matrix attached to your account, supported APIs and authentication flow, quota behavior, pricing units, required networking controls, audit-log fields, and the exact path for local Codex or agent integrations.
Then run a narrow, reversible evaluation. Keep your existing direct route as the baseline. Choose one read-only workload with scrubbed fixtures. Send paired requests through both routes, but do not require the raw model text to match. Compare the contract: acceptance of requests, schema validity, tool-event behavior, latency distribution, errors, usage fields, effective total cost, and whether the log receipt can identify the route.
Evaluate data controls as implemented configuration, not launch-language. Identity and access, data perimeter, retention, log destinations, and egress controls should each have an owner and an evidence artifact. If a protection is described as forthcoming, record it as unavailable for the initial decision. The safe procurement question is not “can we buy it with our cloud commitment?” It is “can this exact workload meet its contract under the controls we actually have?”
Every consequential call should emit a small, machine-readable route receipt. It does not need to store prompts or chain-of-thought. It needs enough data to answer a practical question later: what did we ask for, what actually handled it, and why?
{ "request_id": "req_7f3d", "workload": "contract_extract_v3", "requested_route": "portable-json", "effective_route": "gcp-openai-native", "model": "approved-model-id", "capability_contract": "extract-v3@4", "contract_verdict": "pass", "fallback": null, "usage_available": true, "policy_version": "routing-2026-10-22", "outcome": "schema_valid"}
The receipt makes investigations faster and routing safer. If a cost spike follows a release, you can see whether the cause was longer inputs, cold cache, a fallback, or a routing-policy change. If a structured-output failure rises, you can isolate a route before changing every caller. If a gateway upgrade drops a field, the probe suite and receipts identify the affected workloads.
Token price is a component of cost, not the outcome. A cheaper model or route may produce more retries, lose cache reuse, require manual repair, or violate a contract that forces an expensive fallback. Measure cost per accepted outcome alongside latency and contract failures.
That is especially important for agent traffic. A long session often has a large stable prefix: instructions, tools, policy, context, and history. Moving the next turn to a new route can turn inexpensive cache reads into fresh input. Use model routing at session start or at deliberately isolated subtask boundaries. Do not constantly reselect the “best” provider from the latest user sentence while ignoring the state you are throwing away.
Likewise, an automatic fallback should be a policy decision. Define whether it may retry the same model through another healthy path, wait for the primary provider, use a weaker compatible model, or stop and ask a human. The policy should vary by workload. A background summary can degrade gracefully; an authorization explanation should not silently change its evidence standard.
A compatibility test is most valuable when it changes a release decision. Run the probe suite whenever you add a model, change an upstream region, alter a gateway policy, upgrade a gateway, or enable a new provider feature. Keep a small approved fixture set in source control and version the capability contract beside it. That gives a release reviewer something much stronger than a promise that the gateway is “OpenAI-compatible.”
Set a simple promotion rule. A portable route can move forward only when every required probe passes and the receipt is complete. A native route can move forward when it preserves its documented provider behavior and the organization controls around it are verified. A route that fails a noncritical probe may still serve a limited workload, but the contract must name that limitation. This turns capability drift from an accidental production experiment into an explicit engineering choice.
Keep the fixtures boring and safe: a JSON extraction, a deterministic tool stub, a short stream, a cancellation, and a controlled error. The goal is repeatability, not model benchmarking theater. If a provider or gateway update changes a result, first classify the change: requested behavior, unsupported capability, adapter bug, upstream regression, or a test that was too vague. Only then decide whether to adjust code, policy, or the route itself.
Multi-cloud model access is a real advantage when it lets teams choose the best environment without rewriting every application. The discipline is to keep “choice” from becoming “mystery.” Make the gateway promise explicit, test the promise against each route, and fail visibly when the promise cannot be kept.
A passthrough API preserves a provider’s native request and response shape as much as possible. A unified API translates requests into one common shape across providers. Passthrough is safer for provider-specific capabilities; unified APIs are useful for genuinely portable workloads.
No. Compatibility with a basic endpoint does not guarantee support for every newer parameter, streaming event, tool contract, cache control, or continuation mechanism. Test the exact features your workload needs.
Send capability probes to a direct route and the gateway route, then assert observable behavior. Also use an echo upstream in a test environment to compare the received request against the original one. Unsupported fields should fail explicitly.
Only at a defined replayable checkpoint. Stateful sessions can depend on provider-bound context, cache, reasoning artifacts, and tool-call history. Prefer session stickiness and explicit recovery over a silent mid-session switch.
Do not assume so. The announced offering changes access and purchasing routes, but teams should validate the API surface, quotas, region support, logs, controls, pricing, and workflow behavior available to their account.
Record the workload, requested and effective route, approved model identifier, contract version and verdict, fallback reason, policy version, safe usage fields, latency, and terminal outcome. Avoid retaining sensitive prompt content unless it is explicitly approved.
This is an implementation framework, not a claim that any provider route has identical behavior or controls. Confirm the current contract with the provider and test it with your own approved fixtures.
LLM Gateway Passthrough vs Unified API: The Contract Test That Stops Silent Feature Loss was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.