That 200 OK From Your LLM Gateway Probably Means Nothing

Every AI gateway on the market — LiteLLM, Portkey, OpenRouter, Olla — checks the same things: HTTP status code, response time, token usage. If the backup provider returns HTTP 200 with valid JSON, the gateway declares success.

But HTTP 200 only tells you the request completed. It says nothing about whether the response is correct.

In production monitoring across multi-provider setups, a consistent pattern emerges during failover events:

The gateway logs show "failover successful." Monitoring shows no errors. But the output is wrong.

All major LLM gateways operate at the transport level:

def handle_failover(request, providers):
    for provider in providers:
        try:
            response = provider.complete(request)
            if response.status_code == 200:
                return response  # "Success!"
        except Exception as e:
            log(f"Provider failed: {e}")
            continue  # Try next

Transport-level checks validate:

What they don't validate:

Instead of accepting any 200 OK, add a contract validation step after failover:

from dataclasses import dataclass
from typing import List, Optional
import json

@dataclass
class ResponseContract:
    """Define what a valid response looks like."""
    required_fields: List[str]
    forbidden_patterns: List[str]
    max_tokens: int
    require_json: bool = True
    field_constraints: dict = None

def validate_response(response: dict, contract: ResponseContract) -> dict:
    """Validate response against contract. Returns validation result."""
    issues = []

    for field in contract.required_fields:
        if field not in response:
            issues.append(f"Missing required field: {field}")

    if contract.field_constraints:
        for field, expected_type in contract.field_constraints.items():
            if field in response:
                if not isinstance(response[field], expected_type):
                    issues.append(f"Field {field}: expected {expected_type.__name__}, got {type(response[field]).__name__}")

    if isinstance(response.get("content", ""), str):
        content = response["content"]
        for pattern in contract.forbidden_patterns:
            if pattern.lower() in content.lower():
                issues.append(f"Forbidden pattern found: {pattern}")

    return {
        "valid": len(issues) == 0,
        "issues": issues,
        "issue_count": len(issues),
    }

def validated_failover(request, providers, contract):
    """Failover with response validation."""
    for provider in providers:
        try:
            response = provider.complete(request)
            result = validate_response(response, contract)
            if result["valid"]:
                return response
            else:
                log(f"Provider {provider.name}: contract validation failed - {result['issues']}")
        except Exception as e:
            log(f"Provider {provider.name} error: {e}")
    raise AllProvidersFailed("All providers failed or produced invalid responses")

This pattern adds 45µs P50 overhead (diagnostic engine microbenchmark, 70,000 fault injections across 7 failure types) — negligible compared to the 700-900ms of a typical LLM API call.

Based on the arXiv:2606.14589 taxonomy from a production LLM agent runtime:

The response is structurally perfect — all fields present, correct types, valid JSON. The content is just wrong.

Example: You ask for pricing of GPT-4o. The backup model returns valid JSON with a plausible price that happens to be outdated or from a different model.

Detection: Field-level constraints and cross-field validation (e.g., "model_name + price must match known pricing table").

In multi-step agent workflows, each individual response looks fine — but the combination produces contradictions.

Example: Step 1 says "user is in California." Step 3 says "applying NY state tax." Each response is independently valid.

Detection: Stateful validation across the conversation context, checking for logical consistency between steps.

The response is coherent, well-structured, and cites sources — but the citations don't exist or don't support the claim.

Example: An analysis that cites specific research papers, but the papers don't contain the claimed findings.

Detection: Structured predicates that verify assertions against known reference data.

The validation layer belongs in the proxy, not the application:

┌─────────────┐     ┌──────────────┐     ┌─────────────┐
│  Application │────▶│    Gateway   │────▶│  Provider 1 │
│  (Agent/App)  │     │  + Validation│     ├─────────────┤
└─────────────┘     │     Layer    │     │  Provider 2 │
                    │              │     ├─────────────┤
                    │  After every  │     │  Provider 3 │
                    │  response:    │     └─────────────┘
                    │  1. Validate  │
                    │  2. If fail → │
                    │     retry or  │
                    │     flag      │
                    └──────────────┘

Benefits of proxy-level validation:

When evaluating a gateway, add one more row to your comparison spreadsheet:

The microsecond-level overhead (45µs P50, 102µs P99) makes this a no-brainer addition to the proxy layer.

The validation approach shown above is simplified for illustration. A production-grade implementation — with configurable contracts, multi-provider support, and MCP integration — is what we're building at Correctover.

But the pattern itself is framework-agnostic. You can add response validation to any gateway today with < 100 lines of Python.

References: