# Beyond the Model List: A Practical Checklist for Testing an AI API Provider > Source: > Published: 2026-09-25 19:16:58+00:00 Disclosure: I build a coding agent, so I’m on the consuming side of these APIs rather than selling one. The issue that cost me the most time: advertised context length versus served context length. A provider lists the model at its full documented window because that’s what the model card says, but the actual deployment is configured lower. Nothing in the docs tells you. You find out when requests start erroring or silently truncating at some fraction of what you designed around. Worth making one of your first tests a deliberate long-context request near the documented limit, rather than only typical-sized inputs. The second one: the same model name from two providers is often not the same artifact. Different quantisation, different chat template, sometimes a different revision. Tool-call formatting is where that surfaces first, because a parser tuned against one provider’s output breaks on another’s even though both list the identical model name. If your workflow depends on structured output or function calling, that belongs on the checklist as its own line item rather than folded into “feature support.” Both are invisible on a model list and both are cheap to test for once you know to look.