“OpenAI-compatible" is a spectrum, not a boolean 12 things that silently break when you swap providers

A developer's field notes reveal that the 'OpenAI-compatible' API standard is a spectrum, not a boolean, with at least 12 silent breakages when swapping LLM providers, including discarded parameters like frequency_penalty, inconsistent finish_reason values, and missing stream usage data. The author recommends a probe script to test each provider's actual capabilities before committing to a swap.

POST /v1/chat/completions has become the closest thing we have to a wire protocol for LLM inference. vLLM speaks it, llama.cpp speaks it, Ollama speaks it, TGI speaks it, and every hosted provider advertises it. The pitch is that you change three values and keep your codebase: python client = OpenAI base url="https://your-provider.example/v1", api key=os.environ "PROVIDER API KEY" , resp = client.chat.completions.create model="some-other-model", messages= {"role": "user", "content": "Hello"} , That genuinely works. But I’ve been keeping notes on where it stops working, and the list is longer than I expected. Sharing it here because I suspect people have hit gaps I haven’t. Core request shape model , messages , temperature , top p , stop , the three standard roles, the response envelope choices 0 .message.content , finish reason , usage , SSE streaming with data: DONE termination, and bearer auth. If your app only uses these, swaps really are close to free. The ordering here is roughly by how long it took me to notice. frequency penalty to a server that doesn’t implement it and nothing tells you it was discarded. max tokens vs max completion tokens . usage is often absent from streams. stream options={"include usage": True} , and not every implementation honors it. If it doesn’t, you’re estimating tokens client-side with a tokenizer that may not match the server’s. tools + tool choice: "auto" is broadly supported. Forced function selection, parallel calls, streaming partial arguments, and strict schema adherence are not. Some servers emulate tool calling with prompt injection plus a parser, which fails differently — and worse — than native support. {"type": "json object"} valid JSON, no schema guarantee is common. {"type": "json schema", "strict": true} with constrained decoding is rare. Plenty of docs say “JSON mode supported” and mean only the first. finish reason values. stop / length / tool calls , implementations add their own. Writing if finish reason == "stop" as your only success path fails against a server that returns something else. Retry-After , 429 with nothing, 503, or a 200 with an error in the body. Retry logic keyed to one shape breaks against another. temperature: 0 . Matters if you depend on near-determinism for evals. image url with base64 data URIs is increasingly standard, but size caps, MIME handling, and the detail parameter all vary. /v1/embeddings looks the same, but dimensionality and normalization differ. Swapping the embedding model means an index rebuild, not a config change.A probe script, run once per provider, output committed to the repo. Roughly: python python import json, os from openai import OpenAI client = OpenAI base url=os.environ "BASE URL" , api key=os.environ "API KEY" MODEL = os.environ "MODEL" results = {} def check name : def wrap fn : try: results name = fn except Exception as e: results name = f"FAIL: {type e . name }: {e}" return wrap @check "stream usage" def : usage = None for chunk in client.chat.completions.create model=MODEL, messages= {"role": "user", "content": "Count to three."} , stream=True, stream options={"include usage": True}, : if getattr chunk, "usage", None : usage = chunk.usage return "present" if usage else "MISSING" @check "tool calling" def : r = client.chat.completions.create model=MODEL, messages= {"role": "user", "content": "Weather in Kuala Lumpur?"} , tools= { "type": "function", "function": { "name": "get weather", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": "city" , }, }, } , tool choice="auto", calls = r.choices 0 .message.tool calls return calls 0 .function.name if calls else "NOT CALLED" @check "unknown param" def : client.chat.completions.create model=MODEL, messages= {"role": "user", "content": "hi"} , max tokens=5, extra body={"definitely not a real parameter": True}, return "silently accepted" print json.dumps results, indent=2, default=str Model versions move underneath you, so a re-run tells you what changed in seconds rather than in a production incident.