A reproducible gateway evaluation covering protocol conformance, failure drills, latency, usage and cost reconciliation, security, operations, and exit risk.
Evaluate an AI API gateway by running your real protocol contract through it, forcing the failure modes you care about, and checking the request-level evidence it produces. A feature list or successful “hello world” request cannot prove streaming correctness, tool compatibility, fallback safety, cost accuracy, security boundaries, or a usable exit path.
The most reliable process uses two kinds of criteria: non-negotiable gates that cause a candidate to fail, and scored operational qualities that distinguish candidates after every gate passes.
Do not begin with a vendor comparison table. Begin with one representative workload and write down its invariant contract:
One gateway can pass for a text-only internal assistant and fail for a streaming coding agent. “OpenAI-compatible” is not a sufficient workload definition because compatibility can differ by endpoint, event type, tool, schema keyword, and provider route.
If the team is still deciding whether it needs a proxy or a model-aware control plane, start with LLM Proxy vs AI Gateway. This checklist assumes the gateway category is already justified and tests whether a specific implementation can own the required responsibilities.
The first review should remove candidates that cannot satisfy a mandatory boundary. Ask for demonstrable behavior, not a roadmap statement.
| Gate | Immediate failure condition | Evidence to request |
|---|---|---|
| Protocol | A required request field, output item, or stream event is dropped or rewritten incorrectly | Captured redacted request/response pair and parser result |
| Model identity | The gateway silently changes the requested model | Attempt record with requested and actual model |
| Streaming | It buffers the full response, loses cancellation, or corrupts Tool Argument fragments | Timestamped event sequence and cancellation trace |
| Authentication | Browser or workload clients receive provider credentials | Credential-flow diagram and a live key-rotation test |
| Tenant isolation | One project can use or inspect another project’s keys, usage, or logs | Authorization tests using real isolated accounts |
| Cost evidence | A final charge cannot be connected to model, route, price basis, and usage | One reconciled request ledger |
| Failure safety | A partial stream is transparently replayed or a cancellation starts another attempt | Forced partial-stream and cancel traces |
| Export and exit | Configuration and request contract cannot be recovered without rewriting the application | Export sample and provider-native rollback drill |
A candidate that fails a mandatory gate should not receive a high aggregate score to compensate. Security isolation cannot be offset by a good dashboard, and an incorrect Tool Contract cannot be offset by a lower price.
Use deterministic, non-sensitive inputs and keep the expected wire behavior under version control. The corpus should call the real gateway endpoint; it should not mock the provider or reimplement the gateway’s conversion logic as the test oracle.
| Case | Request | Required observation |
|---|---|---|
| Basic non-streaming text | Pinned model and fixed prompt | Correct status, model identity, text location, usage, request ID |
| Streaming text | Same prompt with streaming enabled | Ordered events, first effective output, final event, cancellation behavior |
| Structured output | Strict schema with required fields and additionalProperties: false |
Valid output or an explicit unsupported error; no silent downgrade |
| Function Calling | One read-only function and one returned result | Function name, JSON arguments, Call ID correlation, final answer |
| No-tool path | Same tools declared but no tool required | Normal text without a fabricated Tool Call |
| Invalid field | Deliberately unsupported or malformed request | Stable client error; no provider fallback that hides the defect |
| Long input boundary | Input just below and above the approved limit | Documented acceptance or explicit rejection, not truncation |
| Usage detail | Request that exercises cached or reasoning usage when supported | Fields survive the route and reconcile with the billing record |
| Cancellation | Client cancels after connection and after first output | Upstream work stops and no new fallback attempt starts |
| Partial stream | Connection fails after effective output | One explicit partial failure; no invisible second answer |
Run every required case against every route that may serve the workload. A primary route passing does not qualify its fallback. The Structured Outputs guide and Function Calling comparison provide field-level test cases for those two capabilities.
Record the gateway version, route configuration version, model ID, provider, region, timestamp, and sanitized result hash. Volatile model and provider behavior should be rechecked before rollout and after a material route change.
Reliability claims matter only when the failure policy is visible. Force each condition before production:
Retry-After;
For each case, capture the attempt order, selected route, status, timing, whether output had started, terminal reason, and any usage or cost. Confirm that the gateway preserves the requested model and protocol unless a separate model-substitution policy is explicitly enabled.
Measure attempt amplification across the SDK, application, gateway, and provider. One layer should own immediate same-contract fallback, while the application owns whether the entire user action can be repeated. The AI API Fallback Strategy provides a phase-aware failure matrix and retry-budget model.
Latency needs the same precision. Compare time to upstream headers, first SSE event, first effective output, first visible text, completion, and visible output speed at realistic concurrency. Do not accept one unexplained “latency” average. See AI API Latency Metrics for definitions and comparison controls.
Choose several completed requests and follow each one through the full accounting chain:
application request ID
→ gateway attempt sequence
→ selected model and route
→ provider or normalized usage
→ applicable price basis
→ final recorded charge
The evaluation should answer:
Test a normal completion, a same-contract fallback, a cancelled request, and an upstream error. A dashboard total is not enough; the gateway must produce a defensible per-request record. AI API Cost Tracking separates provider usage, platform pricing, customer charge, and supplier cost.
Do not compare vendor savings unless the same model, workload, cache behavior, output length, failure rate, and provider price basis are held constant. A lower apparent cost can be caused by missing usage or silent model substitution.
Draw the actual data flow from the client to the gateway and each provider. For every hop, identify who can read credentials, request content, response content, metadata, and administrative configuration.
At minimum, verify:
Ask what happens during key rotation, an operator departure, a compromised application key, and a provider-key leak. Run a rotation and revocation exercise with real isolated test credentials. Do not copy a production secret into an evaluation environment.
The gateway cannot make unsafe application tools safe. Tool authorization, transactionality, approval, and idempotency remain application responsibilities even when model access is centralized. Use AI API Key Security and Cost Controls to separate credentials and workload limits.
The data plane may work while the control plane creates operational risk. Review how operators make and recover from changes:
| Area | Questions to answer |
|---|---|
| Versioning | Is each route, price, policy, and key change versioned or attributable? |
| Validation | Can an invalid route or incompatible model be rejected before activation? |
| Rollout | Can a change target a small workload or percentage first? |
| Rollback | Can operators restore the last known-good configuration quickly? |
| Availability | What happens to existing and new requests if the control plane is unavailable? |
| Health | Is channel health based on current evidence, and can automatic disablement be inspected? |
| Incidents | Can one request be reconstructed without searching several unrelated systems? |
| Limits | Are rate and quota decisions atomic enough to remain correct under concurrency? |
| Change ownership | Are emergency edits separated from normal product configuration? |
Run one configuration rollback and one unhealthy-route removal. Measure the operator steps and verify the resulting data-plane behavior. A screenshot of a rollback button is not the same as a completed rollback drill.
Gateway adoption can create dependencies on model aliases, custom headers, proprietary route names, log APIs, normalized error shapes, or hosted Prompt and Tool configuration. List each dependency and decide whether it is an intentional benefit or accidental lock-in.
A practical exit drill should:
An exit path does not require the gateway to be interchangeable with every provider. It requires the team to know what it owns, what the gateway owns, and how to recover the underlying protocol contract.
Use a worksheet with pass/fail for hard boundaries and a small evidence score for operational quality:
| Score | Meaning |
|---|---|
| 0 | Unsupported or contradicted by the test |
| 1 | Claimed or manually demonstrated once, with weak evidence |
| 2 | Repeatably demonstrated with request-level evidence |
| 3 | Repeatably demonstrated, monitored, and recoverable through a tested control |
Suggested scored areas are protocol coverage, route reliability, attempt evidence, latency diagnostics, usage accuracy, cost reconciliation, key isolation, auditability, configuration rollback, supportability, and exit effort. Weight them according to the workload, but keep the raw evidence beside every score.
Avoid a false-precision total such as 87.4/100 when several rows are subjective. Record the decision as:
The right question is not whether an internal gateway has a license fee. It is which responsibilities the team can own continuously.
| Responsibility | If built internally | If purchased or managed |
|---|---|---|
| Protocol updates | Track provider schemas and regressions | Verify vendor updates and route compatibility |
| Routing and retry | Design state machine and failure evidence | Configure policy and audit actual attempts |
| Usage and billing | Normalize usage and maintain pricing logic | Reconcile vendor records with internal finance truth |
| Security | Store secrets, enforce tenancy, audit access | Validate vendor boundary and configure least privilege |
| Reliability | Operate data plane, control plane, and on-call | Monitor vendor plus your integration and retain an exit path |
| Product support | Diagnose every application/provider interaction | Triage gateway versus provider versus application failures |
Do not insert generic salary or “engineering time saved” figures. Estimate from your own on-call load, protocol-change history, incident frequency, finance requirements, and compliance work. A managed product still requires an accountable internal owner.
For Modelflare, the current evaluation boundary should be explicit. It provides workload API Keys, requested-model routing through eligible groups and channels, ordered group fallback for regular keys, strategy-based group selection for Smart API Keys, pre-upstream group RPM admission, and request-level usage, cost, status, and timing records.
GPT, Codex, and OpenAI traffic is the fully adapted compatibility target. Other OpenAI-compatible model families should be evaluated as raw Chat Completions pass-through unless a capability has been separately verified. A shared base URL is not proof that Responses, hosted tools, Structured Outputs, or Function Calling behave identically on every route.
Modelflare’s fallback should search for an eligible path for the requested model rather than silently selecting another model. Channel failover stops after downstream output starts. Those claims can be evaluated with the protocol corpus and failure drills above instead of accepted as marketing statements.
Use Models & Pricing to identify the current model and group surface and Modelflare Docs to configure an isolated test key. Keep the evaluation requests non-sensitive, pin the exact model, and retain the Request IDs needed to inspect each attempt.
The final decision should be reproducible: another engineer should be able to run the same corpus, inspect the same categories of evidence, and understand why the candidate passed. That process is slower than reading a comparison page, but it is much faster than discovering an incompatible Tool Contract, untraceable bill, or unsafe fallback after the gateway owns production traffic.
Originally published at modelflare.dev.
Disclosure: I'm part of the Modelflare team, which builds the AI API gateway mentioned in this article.