That 200 OK From Your LLM Gateway Probably Means Nothing

A developer warns that HTTP 200 responses from LLM gateways do not guarantee correct output, as gateways like LiteLLM, Portkey, and OpenRouter only check transport-level success. The developer proposes adding contract validation to verify response structure and content, citing negligible overhead of 45µs P50. This approach catches failures like wrong pricing or outdated data that pass standard checks.

Every AI gateway on the market — LiteLLM, Portkey, OpenRouter, Olla — checks the same things: HTTP status code, response time, token usage. If the backup provider returns HTTP 200 with valid JSON, the gateway declares success. But HTTP 200 only tells you the request completed. It says nothing about whether the response is correct . In production monitoring across multi-provider setups, a consistent pattern emerges during failover events: The gateway logs show "failover successful." Monitoring shows no errors. But the output is wrong. All major LLM gateways operate at the transport level : python Every gateway does this: def handle failover request, providers : for provider in providers: try: response = provider.complete request if response.status code == 200: return response "Success " except Exception as e: log f"Provider failed: {e}" continue Try next Transport-level checks validate: What they don't validate: Instead of accepting any 200 OK, add a contract validation step after failover: python from dataclasses import dataclass from typing import List, Optional import json @dataclass class ResponseContract: """Define what a valid response looks like.""" required fields: List str forbidden patterns: List str max tokens: int require json: bool = True field constraints: dict = None def validate response response: dict, contract: ResponseContract - dict: """Validate response against contract. Returns validation result.""" issues = 1. Structural checks ~45µs P50 for field in contract.required fields: if field not in response: issues.append f"Missing required field: {field}" 2. Field type validation if contract.field constraints: for field, expected type in contract.field constraints.items : if field in response: if not isinstance response field , expected type : issues.append f"Field {field}: expected {expected type. name }, got {type response field . name }" 3. Content pattern checks if isinstance response.get "content", "" , str : content = response "content" for pattern in contract.forbidden patterns: if pattern.lower in content.lower : issues.append f"Forbidden pattern found: {pattern}" return { "valid": len issues == 0, "issues": issues, "issue count": len issues , } Usage example def validated failover request, providers, contract : """Failover with response validation.""" for provider in providers: try: response = provider.complete request result = validate response response, contract if result "valid" : return response else: log f"Provider {provider.name}: contract validation failed - {result 'issues' }" Option: retry with next provider, or surface degradation except Exception as e: log f"Provider {provider.name} error: {e}" raise AllProvidersFailed "All providers failed or produced invalid responses" This pattern adds 45µs P50 overhead diagnostic engine microbenchmark, 70,000 fault injections across 7 failure types — negligible compared to the 700-900ms of a typical LLM API call. Based on the arXiv:2606.14589 taxonomy from a production LLM agent runtime: The response is structurally perfect — all fields present, correct types, valid JSON. The content is just wrong. Example : You ask for pricing of GPT-4o. The backup model returns valid JSON with a plausible price that happens to be outdated or from a different model. Detection : Field-level constraints and cross-field validation e.g., "model name + price must match known pricing table" . In multi-step agent workflows, each individual response looks fine — but the combination produces contradictions. Example : Step 1 says "user is in California." Step 3 says "applying NY state tax." Each response is independently valid. Detection : Stateful validation across the conversation context, checking for logical consistency between steps. The response is coherent, well-structured, and cites sources — but the citations don't exist or don't support the claim. Example : An analysis that cites specific research papers, but the papers don't contain the claimed findings. Detection : Structured predicates that verify assertions against known reference data. The validation layer belongs in the proxy , not the application: ┌─────────────┐ ┌──────────────┐ ┌─────────────┐ │ Application │────▶│ Gateway │────▶│ Provider 1 │ │ Agent/App │ │ + Validation│ ├─────────────┤ └─────────────┘ │ Layer │ │ Provider 2 │ │ │ ├─────────────┤ │ After every │ │ Provider 3 │ │ response: │ └─────────────┘ │ 1. Validate │ │ 2. If fail → │ │ retry or │ │ flag │ └──────────────┘ Benefits of proxy-level validation: When evaluating a gateway, add one more row to your comparison spreadsheet: | Capability | Any current gateway | Should be standard | |---|---|---| | Provider routing | ✅ | ✅ | | Failover | ✅ | ✅ | | Circuit breakers | ✅ | ✅ | | Rate limiting | ✅ | ✅ | | Cost tracking | ✅ | ✅ | Response validation | ❌ | ✅ Required | Semantic correctness | ❌ | ✅ Required | The microsecond-level overhead 45µs P50, 102µs P99 makes this a no-brainer addition to the proxy layer. The validation approach shown above is simplified for illustration. A production-grade implementation — with configurable contracts, multi-provider support, and MCP integration — is what we're building at Correctover https://correctover.com . But the pattern itself is framework-agnostic. You can add response validation to any gateway today with < 100 lines of Python. References: