# “OpenAI-compatible" is a spectrum, not a boolean 12 things that silently break when you swap providers

> Source: <https://discuss.huggingface.co/t/openai-compatible-is-a-spectrum-not-a-boolean-12-things-that-silently-break-when-you-swap-providers/179356#post_1>
> Published: 2026-08-28 01:46:58+00:00

`POST /v1/chat/completions`

has become the closest thing we have to a wire protocol for LLM inference. vLLM speaks it, llama.cpp speaks it, Ollama speaks it, TGI speaks it, and every hosted provider advertises it. The pitch is that you change three values and keep your codebase:

python

```
client = OpenAI(
    base_url="https://your-provider.example/v1",
    api_key=os.environ["PROVIDER_API_KEY"],
)
resp = client.chat.completions.create(
    model="some-other-model",
    messages=[{"role": "user", "content": "Hello"}],
)
```

That genuinely works. But I’ve been keeping notes on where it stops working, and the list is longer than I expected. Sharing it here because I suspect people have hit gaps I haven’t.

Core request shape (`model`

, `messages`

, `temperature`

, `top_p`

, `stop`

), the three standard roles, the response envelope (`choices[0].message.content`

, `finish_reason`

, `usage`

), SSE streaming with `data: [DONE]`

termination, and bearer auth. If your app only uses these, swaps really are close to free.

The ordering here is roughly by how long it took me to notice.

`frequency_penalty`

to a server that doesn’t implement it and nothing tells you it was discarded.`max_tokens`

vs `max_completion_tokens`

.`usage`

is often absent from streams.`stream_options={"include_usage": True}`

, and not every implementation honors it. If it doesn’t, you’re estimating tokens client-side with a tokenizer that may not match the server’s.`tools`

+ `tool_choice: "auto"`

is broadly supported. Forced function selection, parallel calls, streaming partial arguments, and strict schema adherence are not. Some servers emulate tool calling with prompt injection plus a parser, which fails differently — and worse — than native support.`{"type": "json_object"}`

(valid JSON, no schema guarantee) is common. `{"type": "json_schema", "strict": true}`

with constrained decoding is rare. Plenty of docs say “JSON mode supported” and mean only the first.`finish_reason`

values.`stop`

/ `length`

/ `tool_calls`

, implementations add their own. Writing `if finish_reason == "stop"`

as your only success path fails against a server that returns something else.`Retry-After`

, 429 with nothing, 503, or a 200 with an error in the body. Retry logic keyed to one shape breaks against another.`temperature: 0`

. Matters if you depend on near-determinism for evals.`image_url`

with base64 data URIs is increasingly standard, but size caps, MIME handling, and the `detail`

parameter all vary.`/v1/embeddings`

looks the same, but dimensionality and normalization differ. Swapping the embedding model means an index rebuild, not a config change.A probe script, run once per provider, output committed to the repo. Roughly:

python

``` python
import json, os
from openai import OpenAI

client = OpenAI(base_url=os.environ["BASE_URL"], api_key=os.environ["API_KEY"])
MODEL = os.environ["MODEL"]
results = {}

def check(name):
    def wrap(fn):
        try:
            results[name] = fn()
        except Exception as e:
            results[name] = f"FAIL: {type(e).__name__}: {e}"
    return wrap

@check("stream_usage")
def _():
    usage = None
    for chunk in client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": "Count to three."}],
        stream=True,
        stream_options={"include_usage": True},
    ):
        if getattr(chunk, "usage", None):
            usage = chunk.usage
    return "present" if usage else "MISSING"

@check("tool_calling")
def _():
    r = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": "Weather in Kuala Lumpur?"}],
        tools=[{
            "type": "function",
            "function": {
                "name": "get_weather",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"],
                },
            },
        }],
        tool_choice="auto",
    )
    calls = r.choices[0].message.tool_calls
    return calls[0].function.name if calls else "NOT CALLED"

@check("unknown_param")
def _():
    client.chat.completions.create(
        model=MODEL, messages=[{"role": "user", "content": "hi"}], max_tokens=5,
        extra_body={"definitely_not_a_real_parameter": True},
    )
    return "silently accepted"

print(json.dumps(results, indent=2, default=str))
```

Model versions move underneath you, so a re-run tells you what changed in seconds rather than in a production incident.