POST /v1/chat/completions
has become the closest thing we have to a wire protocol for LLM inference. vLLM speaks it, llama.cpp speaks it, Ollama speaks it, TGI speaks it, and every hosted provider advertises it. The pitch is that you change three values and keep your codebase:
python
client = OpenAI(
base_url="https://your-provider.example/v1",
api_key=os.environ["PROVIDER_API_KEY"],
)
resp = client.chat.completions.create(
model="some-other-model",
messages=[{"role": "user", "content": "Hello"}],
)
That genuinely works. But I’ve been keeping notes on where it stops working, and the list is longer than I expected. Sharing it here because I suspect people have hit gaps I haven’t.
Core request shape (model
, messages
, temperature
, top_p
, stop
), the three standard roles, the response envelope (choices[0].message.content
, finish_reason
, usage
), SSE streaming with data: [DONE]
termination, and bearer auth. If your app only uses these, swaps really are close to free.
The ordering here is roughly by how long it took me to notice.
frequency_penalty
to a server that doesn’t implement it and nothing tells you it was discarded.max_tokens
vs max_completion_tokens
.usage
is often absent from streams.stream_options={"include_usage": True}
, and not every implementation honors it. If it doesn’t, you’re estimating tokens client-side with a tokenizer that may not match the server’s.tools
tool_choice: "auto"
is broadly supported. Forced function selection, parallel calls, streaming partial arguments, and strict schema adherence are not. Some servers emulate tool calling with prompt injection plus a parser, which fails differently — and worse — than native support.{"type": "json_object"}
(valid JSON, no schema guarantee) is common. {"type": "json_schema", "strict": true}
with constrained decoding is rare. Plenty of docs say “JSON mode supported” and mean only the first.finish_reason
values.stop
/ length
/ tool_calls
, implementations add their own. Writing if finish_reason == "stop"
as your only success path fails against a server that returns something else.Retry-After
, 429 with nothing, 503, or a 200 with an error in the body. Retry logic keyed to one shape breaks against another.temperature: 0
. Matters if you depend on near-determinism for evals.image_url
with base64 data URIs is increasingly standard, but size caps, MIME handling, and the detail
parameter all vary./v1/embeddings
looks the same, but dimensionality and normalization differ. Swapping the embedding model means an index rebuild, not a config change.A probe script, run once per provider, output committed to the repo. Roughly:
python
import json, os
from openai import OpenAI
client = OpenAI(base_url=os.environ["BASE_URL"], api_key=os.environ["API_KEY"])
MODEL = os.environ["MODEL"]
results = {}
def check(name):
def wrap(fn):
try:
results[name] = fn()
except Exception as e:
results[name] = f"FAIL: {type(e).__name__}: {e}"
return wrap
@check("stream_usage")
def _():
usage = None
for chunk in client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": "Count to three."}],
stream=True,
stream_options={"include_usage": True},
):
if getattr(chunk, "usage", None):
usage = chunk.usage
return "present" if usage else "MISSING"
@check("tool_calling")
def _():
r = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": "Weather in Kuala Lumpur?"}],
tools=[{
"type": "function",
"function": {
"name": "get_weather",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}],
tool_choice="auto",
)
calls = r.choices[0].message.tool_calls
return calls[0].function.name if calls else "NOT CALLED"
@check("unknown_param")
def _():
client.chat.completions.create(
model=MODEL, messages=[{"role": "user", "content": "hi"}], max_tokens=5,
extra_body={"definitely_not_a_real_parameter": True},
)
return "silently accepted"
print(json.dumps(results, indent=2, default=str))
Model versions move underneath you, so a re-run tells you what changed in seconds rather than in a production incident.