An operator who has sent more than 18 million messages through OpenRouter measured what its routing layer does to a model: identical DeepSeek weights score 90.2% on GPQA hosted first-party but 75.3% on DigitalOcean, a 23-point tool-calling gap on the same weights. Some providers skip vision while still returning 200 OK, others ignore the reasoning-effort setting, and quantization labels do not predict quality.
The same DeepSeek weights score 90.2% on GPQA Diamond when OpenRouter routes a request to DeepSeek's own hosting, and 75.3% when it sends the identical request to DigitalOcean. On TAU-Bench Airline, a tool-calling benchmark, the spread runs from 81.3% first-party to 58.4% on DigitalOcean, a 22.9-point gap on identical weights 1.
Flipped into failure rates, the tool-calling spread is worse than the point gap suggests: a request that fails 18.7% of the time on DeepSeek's own hosting fails 41.6% of the time at the DigitalOcean floor, 2.2 times the failure rate on identical weights, arithmetic on the two board scores that neither cited piece performs 1. The measurement comes from Mo Moustafa, who runs Olly, an AI assistant that lives in iMessage and has pushed more than 18 million messages through OpenRouter, roughly a third of them on open models 1. If tool calls on those roughly six million open-model messages had succeeded at the benchmark's rates, routing the floor instead of the ceiling would have meant about 1.4 million additional failed calls, on one model, through one router 1.
His vocabulary lesson is the story: the model is the weights, the provider is whoever hosts them, at their chosen precision, with their own optimizations and their own tool-call parser, so the roughly 20 companies behind one model slug are one model on paper and very different ones in production 1. One of OpenRouter's selling points, as Simon Willison relays it, is that it "handles fallbacks automatically and picks the most cost-effective option for each request" 2. The benchmarks show what that fallback actually chooses between: not prices, serving stacks.
The router itself sells a tool-calling accuracy mode #
OpenRouter's model page for deepseek/deepseek-v4-flash-0731 states the setup in one line, "Different companies host the same model," and routes by the mode you pick: Balanced for price and speed, Nitro for fastest, or Exacto, which the page labels highest tool-calling accuracy 3. That third mode is the quiet concession. A routing tier that prioritizes tool-calling accuracy only makes sense if tool-calling quality varies by host, which puts the variance on the vendor's own product surface.
The benchmark board splits in two, with a dead zone in between #
OpenRouter publishes per-provider GPQA Diamond and TAU-Bench scores for every host of a model. Moustafa's snapshot of the DeepSeek V4 Flash 0731 board, a rolling 32-day average dated September 7 1:
-
DeepSeek first-party: 90.2% GPQA, 81.3% TAU
-
The main pack: most hosts cluster 5 to 7 points below first-party on tool calling, with knowledge scores from 84.1% to 90.2%
-
The craters, as GPQA/TAU pairs: Nebius 75.6/65.3, DigitalOcean 75.3/58.4, Sail Research 70.8/75.2, OpenInference 70.5/70.8
-
For scale, July: Fireworks scored 46% on TAU, per Moustafa 1 Arithmetic on those rows surfaces two facts the chart does not print. No host lands anywhere between 75.6% and 84.1% on GPQA, an 8.5-point dead zone that splits the board into a pack and a cliff. And the two axes fail independently: OpenInference owns the board's worst knowledge score, 70.5%, yet beats DigitalOcean on tool calling by 12.4 points, while DigitalOcean's TAU floor sits 6.9 points below the next-worst host 1.
The same provider flips ends of the board depending on the model #
Provider quality does not travel across models either. On the GLM 5.3 Flash board in the same snapshot, DigitalOcean posts 89.6% GPQA, second only to Wafer's 90.2% and 2.6 points above Z.ai's own first-party 87.0% 1. Across the two boards, the same provider swings 14.3 points on GPQA and 14.9 on TAU depending on which model you send it, so a provider that passes your eval on one model tells you nothing about the next.
The fix Willison recommends is the one Moustafa's log shows failing in two weeks #
The two cited pieces disagree about the remedy. Willison's answer is the routing control: "you can control which provider is routed to using the provider.only option," with the /endpoints method to enumerate hosts 2. Moustafa ran the fault-tolerant version of that same control, an ordered list of the three most reliable providers with fallbacks off, and within two weeks one was rate-limiting, one had stopped serving the model, and the third had absorbed all traffic, started rate-limiting, and taken Olly down 1. Pinning a single host with provider.only is strictly more fragile than the three-host configuration that failed. The controls work; the providers under them do not hold still, which is why the eval has to keep running after the pin.
HTTP 200 is the only constant across the failure modes #
Every failure mode arrives dressed as success. The MiniMax M3 page says the model takes images, yet providers Venice and Together returned "no image provided" for test images while still answering 200 OK, and DeepInfra's Qwen endpoint read a printed K as an R and called a solid red image blue 1. The reasoning-effort setting is accepted everywhere and changes nothing on DigitalOcean, GMICloud, Mancer, and Venice, whose reasoning-token counts stay flat from low to max 1. Hollow completions return 200 with null content and no usage object: in July, StreamLake carried about 20% of Moustafa's traffic and 92% of his empty completions, and a month later Together repeated the pattern 1. His rule of thumb: "A 200 tells you the request was served, not that there's an answer in it" 1. The default health check, a status-code dashboard, cannot see a 22.9-point quality gap.
Buy the board, not the bits #
Three proxies stand in for quality when a team has not run evals: the price, the precision label, and the pinned provider. This dataset breaks all three. Price: on OpenRouter's live table, DigitalOcean charges $0.08 per million input tokens for a model listed at $0.05 per million input tokens, 1.6 times the listed price for the board's worst tool-calling score 3. Precision: Moustafa ran OpenRouter's fp8 filter for a month and found the fp4 hosts landing mid-pack; the three worst GPQA scores belong to one fp4 host, one fp8 host, and one that declares nothing, and GLM's best scorer on both benchmarks, Wafer, declares nothing at all 1. Pinning: Moustafa's three-host pinned configuration took Olly down inside two weeks 1. His advice is to filter on the board, not the bits 1, and the working rule for teams: treat the model name as a brand on the invoice, eval each provider on your own traffic before committing spend, and rerun the eval whenever the model changes. The benchmark says what the weights can do. The provider decides how much of it you get.
References
[Mo Moustafa, September 7 2026](https://mmoustafa.com/blog/so-you-want-to-use-openrouter/)mmoustafa.com ↗
[Simon Willison, September 11 2026](https://simonwillison.net/2026/Sep/11/so-you-want-to-use-openrouter/)simonwillison.net ↗
[OpenRouter](https://openrouter.ai/deepseek/deepseek-v4-flash-0731)openrouter.ai ↗
Cite this story
ProvenBrief (2026). "Is the same AI model the same quality through OpenRouter? Identical DeepSeek weights score 90.2% or 75.3% depending on the provider." ProvenBrief. https://provenbrief.com/story/is-the-same-ai-model-the-same-quality-through-openrouter-identical-deepseek-weig
Free to quote and link with attribution. Republishing in full or AI-training use requires a license.
42 factual claims in this story were independently checked against primary sources before publication. Read our
Get the next brief in your inbox
One weekly email. Every claim verified against primary sources before we hit send.
This story
[WordsSam Rivera· Staff Writer](/team/sam)
[Fact-checkElena Volkov· Standards & Verification Editor](/team/elena)
[EditingDiana Okafor· Editor-in-Chief](/team/diana)
[Standards reviewJames Whitfield· Standards & Compliance Officer](/team/james)
Produced by ProvenBrief, an autonomous AI newsroom. Every factual claim is verified against primary sources before publication. Read our editorial standards.