{"slug": "openai-compatible-is-a-spectrum-not-a-boolean-12-things-that-silently-break-when", "title": "“OpenAI-compatible\" is a spectrum, not a boolean 12 things that silently break when you swap providers", "summary": "A developer's field notes reveal that the 'OpenAI-compatible' API standard is a spectrum, not a boolean, with at least 12 silent breakages when swapping LLM providers, including discarded parameters like frequency_penalty, inconsistent finish_reason values, and missing stream usage data. The author recommends a probe script to test each provider's actual capabilities before committing to a swap.", "body_md": "`POST /v1/chat/completions`\n\nhas become the closest thing we have to a wire protocol for LLM inference. vLLM speaks it, llama.cpp speaks it, Ollama speaks it, TGI speaks it, and every hosted provider advertises it. The pitch is that you change three values and keep your codebase:\n\npython\n\n```\nclient = OpenAI(\n    base_url=\"https://your-provider.example/v1\",\n    api_key=os.environ[\"PROVIDER_API_KEY\"],\n)\nresp = client.chat.completions.create(\n    model=\"some-other-model\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello\"}],\n)\n```\n\nThat genuinely works. But I’ve been keeping notes on where it stops working, and the list is longer than I expected. Sharing it here because I suspect people have hit gaps I haven’t.\n\nCore request shape (`model`\n\n, `messages`\n\n, `temperature`\n\n, `top_p`\n\n, `stop`\n\n), the three standard roles, the response envelope (`choices[0].message.content`\n\n, `finish_reason`\n\n, `usage`\n\n), SSE streaming with `data: [DONE]`\n\ntermination, and bearer auth. If your app only uses these, swaps really are close to free.\n\nThe ordering here is roughly by how long it took me to notice.\n\n`frequency_penalty`\n\nto a server that doesn’t implement it and nothing tells you it was discarded.`max_tokens`\n\nvs `max_completion_tokens`\n\n.`usage`\n\nis often absent from streams.`stream_options={\"include_usage\": True}`\n\n, and not every implementation honors it. If it doesn’t, you’re estimating tokens client-side with a tokenizer that may not match the server’s.`tools`\n\n+ `tool_choice: \"auto\"`\n\nis broadly supported. Forced function selection, parallel calls, streaming partial arguments, and strict schema adherence are not. Some servers emulate tool calling with prompt injection plus a parser, which fails differently — and worse — than native support.`{\"type\": \"json_object\"}`\n\n(valid JSON, no schema guarantee) is common. `{\"type\": \"json_schema\", \"strict\": true}`\n\nwith constrained decoding is rare. Plenty of docs say “JSON mode supported” and mean only the first.`finish_reason`\n\nvalues.`stop`\n\n/ `length`\n\n/ `tool_calls`\n\n, implementations add their own. Writing `if finish_reason == \"stop\"`\n\nas your only success path fails against a server that returns something else.`Retry-After`\n\n, 429 with nothing, 503, or a 200 with an error in the body. Retry logic keyed to one shape breaks against another.`temperature: 0`\n\n. Matters if you depend on near-determinism for evals.`image_url`\n\nwith base64 data URIs is increasingly standard, but size caps, MIME handling, and the `detail`\n\nparameter all vary.`/v1/embeddings`\n\nlooks the same, but dimensionality and normalization differ. Swapping the embedding model means an index rebuild, not a config change.A probe script, run once per provider, output committed to the repo. Roughly:\n\npython\n\n``` python\nimport json, os\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=os.environ[\"BASE_URL\"], api_key=os.environ[\"API_KEY\"])\nMODEL = os.environ[\"MODEL\"]\nresults = {}\n\ndef check(name):\n    def wrap(fn):\n        try:\n            results[name] = fn()\n        except Exception as e:\n            results[name] = f\"FAIL: {type(e).__name__}: {e}\"\n    return wrap\n\n@check(\"stream_usage\")\ndef _():\n    usage = None\n    for chunk in client.chat.completions.create(\n        model=MODEL,\n        messages=[{\"role\": \"user\", \"content\": \"Count to three.\"}],\n        stream=True,\n        stream_options={\"include_usage\": True},\n    ):\n        if getattr(chunk, \"usage\", None):\n            usage = chunk.usage\n    return \"present\" if usage else \"MISSING\"\n\n@check(\"tool_calling\")\ndef _():\n    r = client.chat.completions.create(\n        model=MODEL,\n        messages=[{\"role\": \"user\", \"content\": \"Weather in Kuala Lumpur?\"}],\n        tools=[{\n            \"type\": \"function\",\n            \"function\": {\n                \"name\": \"get_weather\",\n                \"parameters\": {\n                    \"type\": \"object\",\n                    \"properties\": {\"city\": {\"type\": \"string\"}},\n                    \"required\": [\"city\"],\n                },\n            },\n        }],\n        tool_choice=\"auto\",\n    )\n    calls = r.choices[0].message.tool_calls\n    return calls[0].function.name if calls else \"NOT CALLED\"\n\n@check(\"unknown_param\")\ndef _():\n    client.chat.completions.create(\n        model=MODEL, messages=[{\"role\": \"user\", \"content\": \"hi\"}], max_tokens=5,\n        extra_body={\"definitely_not_a_real_parameter\": True},\n    )\n    return \"silently accepted\"\n\nprint(json.dumps(results, indent=2, default=str))\n```\n\nModel versions move underneath you, so a re-run tells you what changed in seconds rather than in a production incident.", "url": "https://wpnews.pro/news/openai-compatible-is-a-spectrum-not-a-boolean-12-things-that-silently-break-when", "canonical_source": "https://discuss.huggingface.co/t/openai-compatible-is-a-spectrum-not-a-boolean-12-things-that-silently-break-when-you-swap-providers/179356#post_1", "published_at": "2026-08-28 01:46:58+00:00", "updated_at": "2026-08-28 02:17:48.315239+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure", "large-language-models"], "entities": ["OpenAI", "vLLM", "llama.cpp", "Ollama", "TGI"], "alternates": {"html": "https://wpnews.pro/news/openai-compatible-is-a-spectrum-not-a-boolean-12-things-that-silently-break-when", "markdown": "https://wpnews.pro/news/openai-compatible-is-a-spectrum-not-a-boolean-12-things-that-silently-break-when.md", "text": "https://wpnews.pro/news/openai-compatible-is-a-spectrum-not-a-boolean-12-things-that-silently-break-when.txt", "jsonld": "https://wpnews.pro/news/openai-compatible-is-a-spectrum-not-a-boolean-12-things-that-silently-break-when.jsonld"}}