LLM Provider Quirks: Build Compatibility Tests Before You Trust Model Abstraction

Developers building multi-model AI applications must create dedicated compatibility test suites for LLM provider quirks, as abstraction layers that work in demos frequently break on provider swaps due to differences in tool calls, structured outputs, streaming, and safety handling. Microsoft 365 Copilot now uses models from Microsoft, Anthropic, and OpenAI, while OpenAI, Anthropic, and Google ship increasingly divergent tool-use and agent APIs, making explicit, testable contracts essential for safe model diversity.

Every multi-model AI app eventually reaches the same uncomfortable moment: the abstraction layer works in the demo, then breaks on the first serious provider swap. The failure is rarely dramatic at first. A tool call arrives in a slightly different shape. A schema that passed with one model becomes too strict for another. A streaming parser assumes the wrong event order. A retry turns a harmless timeout into a duplicate write. The dashboard still says the provider returned a successful response, but the product quietly does the wrong thing. This is why LLM provider quirks deserve their own test suite. You need compatibility tests that prove your application contract survives the weird parts of each provider. That matters more now because the market is moving toward model diversity. Microsoft documentation says Microsoft 365 Copilot can use models from Microsoft, Anthropic, and OpenAI depending on the feature and need. Recent reporting also points to Microsoft expanding its in-house MAI model usage to improve cost efficiency and reduce dependence on external frontier labs. At the same time, OpenAI, Anthropic, and Google continue to ship richer tool use, structured output, and agent APIs with different assumptions baked into each surface. The practical lesson for developers is simple: model abstraction is useful, but only after you know what you are abstracting. The easy version of model abstraction looks like this: js const answer = await llm.generate { model: "some-model", prompt: userPrompt} ; That wrapper is fine for plain text. It becomes fragile when your app depends on tool calls, structured outputs, streaming, safety refusals, caching, files, images, function schemas, or provider-specific reasoning controls. Developers often discover this after they have already promised portability. The team starts with one provider. Finance asks for a cheaper model on high-volume tasks. Security asks for a model that keeps data in a specific environment. Product wants a long-context model for large documents. Leadership sees Microsoft-style model choice and asks, “Can we do that too?” The answer is yes, but the hidden work is not the adapter. The hidden work is the contract. A good LLM abstraction does not hide provider differences. It makes the differences explicit, testable, and safe to route around. OpenAI’s function calling guide describes tool calling as a multi-step loop where the model requests a tool, your application executes it, and you send the result back for a final response. Anthropic’s Claude docs describe a similar client-tool loop, but with Claude-specific content blocks, stop reasons, and tool-result handling. Google’s Gemini function calling docs frame the model as returning function arguments for external actions, knowledge augmentation, and capability extension. These are close enough to sound interchangeable and different enough to break code that assumes they are identical. A provider quirk is not a bug. It is a behavior your app must understand before it can depend on a model safely. Some quirks are documented API differences. Some are model behavior patterns. Some only show up under load, long context, unusual schemas, partial streams, or safety-sensitive prompts. You do not need to memorize every provider detail. You need to identify the categories that can break your product. One provider may return tool calls as separate response items. Another may return content blocks. A wrapper may normalize the happy path but leak provider-specific fields during parallel tool calls or partial failures. Your app should not assume that “tool call” means one universal object. Structured output sounds simple: give the model a JSON Schema and receive valid JSON. In practice, provider support differs by API, model, schema keyword, strictness mode, SDK helper, and whether the schema is used for a final answer or a tool call. OpenAI’s structured output docs distinguish schema adherence from older JSON mode and recommend Structured Outputs when available. Anthropic’s structured output docs use an output config.format pattern with schema limitations. Google's Gemini docs describe JSON Schema support for structured outputs, with separate API modes. The words look similar. The integration details are not the same. Streaming is where many abstractions start to leak. You may receive text deltas, reasoning metadata, tool-call deltas, content block starts, content block stops, usage events, safety events, or provider-specific keepalive messages. If your parser expects one event order, a provider swap can turn a smooth UI into missing tokens or broken tool arguments. An invalid schema, blocked prompt, rate limit, overloaded model, timeout, content filter, or provider-side incident should not all become new Error "LLM failed" . Your retry logic needs to know whether to retry, fall back, ask the user for different input, page an engineer, or stop because the action might be unsafe to repeat. Refusals are product behavior, not just model behavior. If one provider returns a structured refusal field and another returns refusal text, your app still needs one user-facing policy. Otherwise the same unsafe request may produce inconsistent UI states, logs, and analytics. Some models expose reasoning effort, caching, priority inference, batch options, or task-specific modes. Others expose different levers or none at all. If your abstraction only accepts temperature and max tokens, it may erase the exact controls that made a provider worth using. Provider compatibility should be tested as a release workflow, not guessed during a migration. Do not start by wrapping every SDK behind the smallest common denominator. That approach turns into a pile of exceptions. Start by writing the contract your product needs. The contract should describe what your app expects from any model, regardless of provider. Then each provider adapter must prove it can satisfy that contract for the use cases you plan to route to it. A practical contract includes: The most important part is the error contract. Many teams test output quality but forget operational behavior. That is backwards. A slightly weaker answer is annoying. A retried payment, deleted file, or invisible refusal is a production incident. Your compatibility suite should be small enough to run in CI and realistic enough to catch the failures that matter. It is not a replacement for deep model evals. It is the preflight check that says, “This provider can safely enter this route.” Pick the workflows your product actually runs: extract invoice fields, draft a pull request summary, classify support tickets, call a CRM tool, answer from documents, generate a UI component, or route an agent step. For each workflow, save a few golden prompts with expected behavior. Do not only include clean examples. Include messy inputs, missing fields, ambiguous user requests, long context, policy-sensitive requests, and cases where the model should ask a question instead of acting. Create schemas that represent your real payloads, then add edge cases. Test required fields, nested objects, arrays, enums, nullable values, descriptions, unknown properties, and large schemas. If you use Zod, Pydantic, or generated JSON Schema, test the generated schema exactly as your app sends it. Your goal is not to prove that every model supports every schema. Your goal is to classify where each provider is safe. Tool calling is a loop, so test the loop. The first model response is not enough. Verify that your adapter can: Use fake tools in CI. For example, a fake calendar tool can record whether a meeting would have been created without touching a real calendar. That lets you test dangerous paths without danger. Record streaming event samples from each provider and replay them through your parser. Include normal text, long output, tool calls, interrupted streams, provider errors, and user cancellation. If the UI depends on streaming, your compatibility suite should fail when an adapter drops tokens, emits duplicate chunks, or marks a response complete too early. Mock the ugly cases: rate limit, invalid request, bad schema, content policy refusal, empty response, network timeout, slow stream, malformed tool arguments, and provider outage. Then assert the product behavior. The answer should not be “log and move on.” It should be route-specific. The following sketch is not a full framework. It shows the shape of a contract-first adapter. Each provider can keep its native SDK internally, but the rest of your product sees one tested result shape. type LlmRequest = { route: "ticket triage" | "invoice extract" | "agent step"; messages: Array<{ role: "system" | "user" | "assistant"; content: string } ; tools?: ToolDefinition ; responseSchema?: unknown; idempotencyKey: string;}; type LlmResult = | { type: "final"; text: string; usage: Usage; providerMeta: unknown } | { type: "tool call"; calls: ToolCall ; usage: Usage; providerMeta: unknown } | { type: "refusal"; reason: string; usage: Usage; providerMeta: unknown }; type LlmFailure = | { type: "retryable"; reason: string; retryAfterMs?: number } | { type: "bad request"; reason: string } | { type: "policy"; reason: string } | { type: "unsafe retry"; reason: string }; interface ProviderAdapter { name: string; supports route: LlmRequest "route" : boolean; generate request: LlmRequest : Promise<LlmResult ;} Notice what is missing: no provider SDK types leak into the product route. That does not mean you throw away provider-specific features. It means you isolate them behind capabilities that tests can verify. A provider adapter can expose metadata for observability. It can support special controls such as reasoning effort, prompt caching, or priority inference. But those controls should be mapped through route-level policy, not scattered across feature code. The most common routing mistake is treating providers like interchangeable vendors with different prices. A better mental model is capability routing. For a simple classification task, you may care about cost, latency, and stable JSON. For a coding agent, you may care about long context, tool reliability, repository instruction following, and patch quality. For a customer support workflow, refusal behavior, citation quality, and privacy boundaries may matter more than raw reasoning strength. Build a route profile for each workflow: Then promote models into routes only after they pass compatibility and quality gates. This turns “Should we use OpenAI, Claude, Gemini, or a smaller model?” into a better question: “Which tested model can satisfy this route’s contract at the best cost, latency, and risk level?” If you search for LLM abstraction layers, you will find three common article types. The first says to use a unified API or gateway. The second compares model prices and context windows. The third explains how to migrate from one provider to another. Those are useful, but they often skip the practical failure layer. They do not show the exact compatibility tests a team should run before trusting the abstraction. They underplay streaming and error semantics. They rarely treat refusals, retries, duplicate tool execution, or schema keyword differences as first-class product risks. That is the gap this article fills: not “use an abstraction layer,” but “prove your abstraction layer handles provider quirks before it touches users.” You do not need to rebuild your AI stack in one sprint. Start with one route that already hurts. Good candidates are high-volume extraction, support triage, document summarization, coding-agent review, or any workflow where cost, latency, or provider reliability has become visible. First, capture twenty to fifty real examples with user data removed. Add expected outcomes and failure labels. Second, write the route contract. Third, implement one adapter for your current provider and make it pass. Fourth, add a second provider and run the same suite. Fifth, compare not only answer quality but parser behavior, retry safety, streaming stability, and cost per successful outcome. Only then should you turn on traffic splitting. Start with shadow mode if possible. Send the same request to the candidate model without showing its answer to the user. Log compatibility failures and quality scores. When the new provider performs well, move a small percentage of low-risk traffic. Keep rollback as a configuration change, not a deploy. The safest provider switch is boring: measured, reversible, and backed by route-level contracts. Provider compatibility is impossible to improve if your logs only store the final answer. For each request, capture the provider, model, route, adapter version, prompt template version, schema version, tool catalog version, latency, token usage, cache status, retry count, refusal state, tool-call count, validation errors, and final outcome. Do not log sensitive user content unless your policy allows it. You can often store hashes, redacted samples, metadata, and failure classes instead. The goal is to answer operational questions quickly: This is also where your abstraction pays off. Once every adapter reports the same normalized failure classes, your product team can compare behavior without reading raw provider responses all day. AI gateways and unified APIs can be valuable. They centralize keys, routing, rate limits, observability, and sometimes provider normalization. For many teams, using a gateway is better than wiring five SDKs directly into product code. But a gateway does not remove your responsibility for product-level compatibility. It can tell you that a provider returned a response. It may even normalize response shape. It cannot know whether your invoice extraction route treats a missing tax ID as a user-fixable issue, a model failure, or an acceptable blank field. That is your contract. The best pattern is layered: Before adding a new model to production, ask three questions. Can it satisfy the route contract? If not, do not ship it on that route, no matter how impressive the benchmark looks. Can the adapter explain failures in normalized terms? If not, support and operations will suffer during incidents. Can you roll back without a deploy? If not, you are not ready for live traffic. The teams that win with multi-model AI will not be the teams with the prettiest wrapper. They will be the teams with the clearest contracts, the most boring rollback plan, and the discipline to test provider quirks before users find them. LLM provider quirks are behavior differences between model APIs that can affect production apps. Common examples include tool-call shape, JSON Schema support, streaming event format, refusal handling, retry semantics, context limits, file support, and provider-specific model controls. Yes. A gateway can simplify routing, keys, observability, and some normalization, but it cannot define your product’s expected behavior. You still need route-level tests for schemas, tool calls, errors, refusals, latency, and safe rollback. No. An abstraction layer is useful when it is honest about provider differences. The risky version hides everything behind a tiny generic interface. The safer version exposes capabilities, normalizes failures, and requires every provider adapter to pass compatibility tests before use. Start with twenty to fifty high-quality examples for one product route. Include normal cases, messy cases, missing data, long context, safety-sensitive prompts, malformed tool arguments, timeouts, and retry scenarios. Add more cases as real failures appear. Model evals measure whether a model performs a task well. Compatibility tests measure whether a provider can safely fit your application’s contract. You need both. A model can score well in an eval and still break your app through streaming, schema, retry, or refusal behavior. There is no universal answer. OpenAI, Anthropic, Google Gemini, and other providers all support tool-like workflows, but their APIs and model behavior differ. The best provider is the one that passes your route’s tool-call, schema, error, latency, and quality tests at an acceptable cost. Build only the part that is specific to your product. You can use SDKs or gateways for transport and provider access, then build your own route contracts, compatibility tests, and rollout rules. That keeps your product behavior under your control without reinventing every infrastructure piece. LLM Provider Quirks: Build Compatibility Tests Before You Trust Model Abstraction https://pub.towardsai.net/llm-provider-quirks-build-compatibility-tests-before-you-trust-model-abstraction-53c9b0c9a6d0 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.