{"slug": "that-200-ok-from-your-llm-gateway-probably-means-nothing", "title": "That 200 OK From Your LLM Gateway Probably Means Nothing", "summary": "A developer warns that HTTP 200 responses from LLM gateways do not guarantee correct output, as gateways like LiteLLM, Portkey, and OpenRouter only check transport-level success. The developer proposes adding contract validation to verify response structure and content, citing negligible overhead of 45µs P50. This approach catches failures like wrong pricing or outdated data that pass standard checks.", "body_md": "Every AI gateway on the market — LiteLLM, Portkey, OpenRouter, Olla — checks the same things: HTTP status code, response time, token usage. If the backup provider returns HTTP 200 with valid JSON, the gateway declares success.\n\nBut HTTP 200 only tells you the request completed. It says nothing about whether the response is *correct*.\n\nIn production monitoring across multi-provider setups, a consistent pattern emerges during failover events:\n\nThe gateway logs show \"failover successful.\" Monitoring shows no errors. But the output is wrong.\n\nAll major LLM gateways operate at the **transport level**:\n\n``` python\n# Every gateway does this:\ndef handle_failover(request, providers):\n    for provider in providers:\n        try:\n            response = provider.complete(request)\n            if response.status_code == 200:\n                return response  # \"Success!\"\n        except Exception as e:\n            log(f\"Provider failed: {e}\")\n            continue  # Try next\n```\n\nTransport-level checks validate:\n\nWhat they don't validate:\n\nInstead of accepting any 200 OK, add a contract validation step after failover:\n\n``` python\nfrom dataclasses import dataclass\nfrom typing import List, Optional\nimport json\n\n@dataclass\nclass ResponseContract:\n    \"\"\"Define what a valid response looks like.\"\"\"\n    required_fields: List[str]\n    forbidden_patterns: List[str]\n    max_tokens: int\n    require_json: bool = True\n    field_constraints: dict = None\n\ndef validate_response(response: dict, contract: ResponseContract) -> dict:\n    \"\"\"Validate response against contract. Returns validation result.\"\"\"\n    issues = []\n\n    # 1. Structural checks (~45µs P50)\n    for field in contract.required_fields:\n        if field not in response:\n            issues.append(f\"Missing required field: {field}\")\n\n    # 2. Field type validation\n    if contract.field_constraints:\n        for field, expected_type in contract.field_constraints.items():\n            if field in response:\n                if not isinstance(response[field], expected_type):\n                    issues.append(f\"Field {field}: expected {expected_type.__name__}, got {type(response[field]).__name__}\")\n\n    # 3. Content pattern checks\n    if isinstance(response.get(\"content\", \"\"), str):\n        content = response[\"content\"]\n        for pattern in contract.forbidden_patterns:\n            if pattern.lower() in content.lower():\n                issues.append(f\"Forbidden pattern found: {pattern}\")\n\n    return {\n        \"valid\": len(issues) == 0,\n        \"issues\": issues,\n        \"issue_count\": len(issues),\n    }\n\n# Usage example\ndef validated_failover(request, providers, contract):\n    \"\"\"Failover with response validation.\"\"\"\n    for provider in providers:\n        try:\n            response = provider.complete(request)\n            result = validate_response(response, contract)\n            if result[\"valid\"]:\n                return response\n            else:\n                log(f\"Provider {provider.name}: contract validation failed - {result['issues']}\")\n                # Option: retry with next provider, or surface degradation\n        except Exception as e:\n            log(f\"Provider {provider.name} error: {e}\")\n    raise AllProvidersFailed(\"All providers failed or produced invalid responses\")\n```\n\nThis pattern adds **45µs P50 overhead** (diagnostic engine microbenchmark, 70,000 fault injections across 7 failure types) — negligible compared to the 700-900ms of a typical LLM API call.\n\nBased on the arXiv:2606.14589 taxonomy from a production LLM agent runtime:\n\nThe response is structurally perfect — all fields present, correct types, valid JSON. The content is just wrong.\n\n**Example**: You ask for pricing of GPT-4o. The backup model returns valid JSON with a plausible price that happens to be outdated or from a different model.\n\n**Detection**: Field-level constraints and cross-field validation (e.g., \"model_name + price must match known pricing table\").\n\nIn multi-step agent workflows, each individual response looks fine — but the combination produces contradictions.\n\n**Example**: Step 1 says \"user is in California.\" Step 3 says \"applying NY state tax.\" Each response is independently valid.\n\n**Detection**: Stateful validation across the conversation context, checking for logical consistency between steps.\n\nThe response is coherent, well-structured, and cites sources — but the citations don't exist or don't support the claim.\n\n**Example**: An analysis that cites specific research papers, but the papers don't contain the claimed findings.\n\n**Detection**: Structured predicates that verify assertions against known reference data.\n\nThe validation layer belongs **in the proxy**, not the application:\n\n```\n┌─────────────┐     ┌──────────────┐     ┌─────────────┐\n│  Application │────▶│    Gateway   │────▶│  Provider 1 │\n│  (Agent/App)  │     │  + Validation│     ├─────────────┤\n└─────────────┘     │     Layer    │     │  Provider 2 │\n                    │              │     ├─────────────┤\n                    │  After every  │     │  Provider 3 │\n                    │  response:    │     └─────────────┘\n                    │  1. Validate  │\n                    │  2. If fail → │\n                    │     retry or  │\n                    │     flag      │\n                    └──────────────┘\n```\n\nBenefits of proxy-level validation:\n\nWhen evaluating a gateway, add one more row to your comparison spreadsheet:\n\n| Capability | Any current gateway | Should be standard |\n|---|---|---|\n| Provider routing | ✅ | ✅ |\n| Failover | ✅ | ✅ |\n| Circuit breakers | ✅ | ✅ |\n| Rate limiting | ✅ | ✅ |\n| Cost tracking | ✅ | ✅ |\nResponse validation |\n❌ | ✅ Required |\nSemantic correctness |\n❌ | ✅ Required |\n\nThe microsecond-level overhead (45µs P50, 102µs P99) makes this a no-brainer addition to the proxy layer.\n\nThe validation approach shown above is simplified for illustration. A production-grade implementation — with configurable contracts, multi-provider support, and MCP integration — is what we're building at [Correctover](https://correctover.com).\n\nBut the pattern itself is framework-agnostic. You can add response validation to any gateway today with < 100 lines of Python.\n\n*References:*", "url": "https://wpnews.pro/news/that-200-ok-from-your-llm-gateway-probably-means-nothing", "canonical_source": "https://dev.to/correctover/that-200-ok-from-your-llm-gateway-probably-means-nothing-4ok1", "published_at": "2026-06-30 11:07:00+00:00", "updated_at": "2026-06-30 11:19:52.032358+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools", "ai-safety", "ai-agents"], "entities": ["LiteLLM", "Portkey", "OpenRouter", "Olla", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/that-200-ok-from-your-llm-gateway-probably-means-nothing", "markdown": "https://wpnews.pro/news/that-200-ok-from-your-llm-gateway-probably-means-nothing.md", "text": "https://wpnews.pro/news/that-200-ok-from-your-llm-gateway-probably-means-nothing.txt", "jsonld": "https://wpnews.pro/news/that-200-ok-from-your-llm-gateway-probably-means-nothing.jsonld"}}