Where you send a prompt changes what you pay, how long you wait, and whether the call succeeds.
One controlled setup: 1k, 10k and 100k input-token targets, each with a 100 or 1k output-token budget. Temperature 0, reasoning off. Cold requests use a unique prefix; warm requests repeat the same prompt. These are serving measurements, not task-quality scores.
A repeated prompt can change the winner. #
DeepSeek’s 100k-input, 100-output-budget requests cost 14.9× less warm than cold in this sample. Telnyx led median generation speed in 12/12 conditions. The fastest first token depended on the request shape.
Cost and first-token latency #
10k input target, 100 output-token budget, cold cache. Lower and further left means cheaper requests and a shorter wait for the first token. The list is ordered by cost.
Cost = total request charges ÷ input tokens × 1M, including output charges. It is not the listed input-token price. Cost points cover 13 routes. Hover a point for its route and values.
Time to first token #
The dot is the median wait; the line extends to the 90th percentile, the wait met or beaten by 90% of timed successful calls. A short line means less spread between typical and slower requests. All three panels use cold requests with a 100-token output budget and the same axis scale.
1k input, s
10k input, s
100k input, s
Timing includes successful calls only. TTFT runs from request start to the first content or reasoning delta received by the client; it includes network and queueing time. P90 uses linear interpolation between ordered observations.
Generation speed #
Median tokens generated per second after streaming starts. This matters when the answer is long: a quick first token can still be followed by slow generation. All three panels use cold requests with a 100-token output budget and the same axis scale.
1k input, tok/s
10k input, tok/s
100k input, tok/s
Timing includes successful calls only. Decode speed = reported completion tokens ÷ elapsed time from the first to last output delta. Non-streaming responses have no decode-speed measurement.
Cache hit #
Share of input tokens reported as cached on repeated 10k prompts, with a 100-token output budget. A high share means less input work is billed at the uncached rate when a cache discount applies.
Cache share = sum of cached input tokens ÷ sum of input tokens for successes reporting both. This is a token share, not the percentage of requests that hit cache; the warm sequence includes its initial request.
What a warm cache does to cost #
100k input target, 100 output-token budget. Each line connects a route’s cold and warm request cost on a logarithmic scale: equal vertical distances represent equal cost ratios.
Same cost definition as the scatter plot, including output charges. Warm means a repeated prompt; the measured cache share determines how much was actually cached.
Reliability by request shape #
Percentage of measured calls that failed. Columns show input-token targets → output-token budgets. Compare cold and warm separately; a route can behave differently when the same prompt asks for a longer response.
Cold requests
Warm requests
Failure rate = failed calls ÷ measured calls in that condition. HTTP, transport and response errors count as failures. Exact counts and descriptive intervals are in the tables below. Zero means no observed failure in this sample.
Where automatic routing sent the calls #
Each bar shows the upstream names returned by OpenRouter for one request shape and cache condition. A repeated prompt may reach a different upstream, which can change its cache behavior.
Segment width is the share of calls. Names are reported upstreams, not verified machines. Colours separate neighbouring segments; hover for every name and count. This distribution alone does not establish why routing changed.
All upstream names and call counts #
| 1k→100 cold | Baidu: 18, Relace: 14, AkashML: 13, Ambient: 10, DigitalOcean: 9, CoreWeave: 7, OpenInference: 5, DeepSeek: 4, StreamLake: 3, Wafer: 3, Alibaba: 2, Mancer 2: 2, Together: 2, Venice: 2, BaseTen: 1, Inceptron: 1, Makora: 1, Morph: 1, NextBit: 1, Reka: 1 |
|---|---|
| 1k→100 warm | Relace: 24, CoreWeave: 16, DeepSeek: 11, SiliconFlow: 11, Parasail: 8, Together: 8, AtlasCloud: 7, Novita: 7, Baidu: 2, GMICloud: 2, Ambient: 1, Cloudflare: 1, Makora: 1, Morph: 1 |
| 1k→1k cold | Ambient: 15, Relace: 12, AkashML: 10, Baidu: 9, DigitalOcean: 8, Wafer: 8, Parasail: 6, CoreWeave: 4, Inceptron: 4, BaseTen: 3, Makora: 3, Mancer 2: 3, Morph: 3, OpenInference: 3, Alibaba: 2, DeepSeek: 2, StreamLake: 2, Venice: 2, Reka: 1 |
| 1k→1k warm | Novita: 18, Together: 18, CoreWeave: 12, DeepSeek: 12, Parasail: 10, AtlasCloud: 9, Relace: 7, SiliconFlow: 7, GMICloud: 4, Morph: 2, Reka: 1 |
| 10k→100 cold | Baidu: 15, Relace: 13, DigitalOcean: 10, CoreWeave: 7, Makora: 7, OpenInference: 6, AkashML: 5, Morph: 5, StreamLake: 5, Ambient: 4, Reka: 4, Together: 4, Parasail: 3, Venice: 3, Inceptron: 2, Wafer: 2, Alibaba: 1, BaseTen: 1, Mancer 2: 1, NextBit: 1, Phala: 1 |
| 10k→100 warm | Together: 29, Novita: 20, CoreWeave: 17, AtlasCloud: 13, Relace: 8, DeepSeek: 6, SiliconFlow: 5, Ambient: 1, Inceptron: 1 |
| 10k→1k cold | DigitalOcean: 13, Relace: 12, Baidu: 9, Makora: 9, CoreWeave: 8, Wafer: 8, AkashML: 6, Inceptron: 4, OpenInference: 4, Parasail: 4, Ambient: 3, Mancer 2: 3, Reka: 3, StreamLake: 3, NextBit: 2, Together: 2, Venice: 2, BaseTen: 1, Cloudflare: 1, DeepSeek: 1, Morph: 1, SiliconFlow: 1 |
| 10k→1k warm | SiliconFlow: 26, Novita: 18, CoreWeave: 14, DeepSeek: 13, Together: 12, Relace: 9, AtlasCloud: 5, Parasail: 3 |
| 100k→100 cold | Baidu: 7, DigitalOcean: 3, Relace: 3, Wafer: 3, DeepInfra: 2, Makora: 2, OpenInference: 2, AkashML: 1, BaseTen: 1, CoreWeave: 1, Morph: 1, Parasail: 1, Reka: 1, StreamLake: 1, Together: 1 |
| 100k→100 warm | Relace: 14, Together: 6, CoreWeave: 5, Parasail: 3, BaseTen: 1, SiliconFlow: 1 |
| 100k→1k cold | BaseTen: 2, Mancer 2: 2, Wafer: 2, Baidu: 1, CoreWeave: 1, DeepInfra: 1, Fireworks: 1, Inceptron: 1, OpenInference: 1, Relace: 1 |
| 100k→1k warm | CoreWeave: 8, SiliconFlow: 7, Parasail: 6, Baidu: 3, DeepSeek: 2, OpenInference: 1, Reka: 1, Relace: 1, Together: 1 |
The measurements #
Choose the request shape and cache condition. Input sizes are targets and output sizes are budgets; actual token usage is retained in the downloadable records.
Timings describe successful calls. n is the number of observations behind each metric. Failure intervals are descriptive 95% Wilson intervals. “Not reported” means no measurement, not zero.
1k input / 100 output, cold
Lowest median first-token latency: coreweave/fp8. Highest median decode speed: telnyx.
| Route | Included calls | Failures | First token p50, s | First token p90, s | Decode, tok/s | Cache share | Effective cost / 1M prompt tokens |
|---|---|---|---|---|---|---|---|
| atlas-cloud/fp4 | 100 | 0/1000.0–3.7% interval | 1.743n=100 | 6.064 | 122.6n=100 | 0.0%n=100 | $0.56255100/100 successes priced |
| coreweave/fp8 | 100 | 0/1000.0–3.7% interval | 0.387n=100 | 0.590 | 67.5n=100 | 0.0%n=100 | $0.15105100/100 successes priced |
| deepseek | 100 | 0/1000.0–3.7% interval | 0.688n=100 | 0.953 | 113.2n=100 | 0.0%n=100 | $0.27956100/100 successes priced |
| doubleword-flex | 100 | 0/1000.0–3.7% interval | 2.337n=100 | 2.361 | not reportedn=0 | 0.0%n=100 | $0.09095n=80 |
| doubleword-realtime | 100 | 0/1000.0–3.7% interval | 0.472n=100 | 0.540 | 77.2n=100 | 0.0%n=100 | $0.11692n=80 |
| gmicloud/fp8 | 100 | 1/1000.2–5.4% interval | 5.004n=99 | 14.836 | 107.6n=99 | 0.0%n=99 | $0.4485499/99 successes priced |
| morph/bf16 | 100 | 90/10082.6–94.5% interval | 1.177n=9 | 1.764 | 53.8n=9 | 0.0%n=9 | $0.124769/10 successes priced |
| novita/fp8 | 100 | 0/1000.0–3.7% interval | 0.830n=100 | 1.954 | 113.0n=100 | 0.0%n=100 | $0.56126100/100 successes priced |
| openrouter-auto | 100 | 0/1000.0–3.7% interval | 0.892n=100 | 6.082 | 49.5n=100 | 0.0%n=100 | $0.11655100/100 successes priced |
| parasail/fp8 | 100 | 17/10010.9–25.5% interval | 0.977n=83 | 1.528 | 50.6n=83 | 0.0%n=83 | $0.1655583/83 successes priced |
| relace/fp4 | 100 | 1/1000.2–5.4% interval | 0.673n=97 | 2.559 | 89.5n=97 | 0.0%n=97 | $0.0815397/99 successes priced |
| siliconflow/fp8 | 100 | 0/1000.0–3.7% interval | 1.375n=100 | 4.449 | 93.0n=100 | 0.0%n=100 | $0.28058100/100 successes priced |
| telnyx | 100 | 0/1000.0–3.7% interval | 0.447n=100 | 0.555 | 345.9n=100 | not reportedn=0 | not reported0/100 successes priced |
| together | 100 | 22/10015.0–31.1% interval | 1.300n=78 | 2.989 | 51.1n=78 | 0.0%n=78 | $0.1647978/78 successes priced |
1k input / 100 output, warm
Lowest median first-token latency: coreweave/fp8. Highest median decode speed: telnyx.
| Route | Included calls | Failures | First token p50, s | First token p90, s | Decode, tok/s | Cache share | Effective cost / 1M prompt tokens |
|---|---|---|---|---|---|---|---|
| atlas-cloud/fp4 | 100 | 0/1000.0–3.7% interval | 1.097n=100 | 1.328 | 151.3n=100 | 97.6%n=100 | $0.15612100/100 successes priced |
| coreweave/fp8 | 100 | 0/1000.0–3.7% interval | 0.308n=100 | 0.439 | 155.9n=100 | 64.3%n=100 | $0.11836100/100 successes priced |
| deepseek | 100 | 0/1000.0–3.7% interval | 0.926n=100 | 1.146 | 103.8n=100 | 97.6%n=100 | $0.07551100/100 successes priced |
| doubleword-flex | 100 | 0/1000.0–3.7% interval | 1.843n=100 | 1.879 | not reportedn=0 | 99.0%n=100 | $0.01860n=80 |
| doubleword-realtime | 100 | 0/1000.0–3.7% interval | 0.489n=100 | 0.877 | 103.9n=100 | 99.0%n=100 | $0.02424n=80 |
| gmicloud/fp8 | 100 | 0/1000.0–3.7% interval | 2.331n=100 | 7.677 | 124.6n=100 | 75.0%n=100 | $0.19059100/100 successes priced |
| morph/bf16 | 100 | 22/10015.0–31.1% interval | 0.854n=76 | 1.222 | 54.3n=76 | 93.4%n=76 | $0.0557476/78 successes priced |
| novita/fp8 | 100 | 0/1000.0–3.7% interval | 0.958n=100 | 1.231 | 97.4n=100 | 98.6%n=100 | $0.14529100/100 successes priced |
| openrouter-auto | 100 | 0/1000.0–3.7% interval | 0.764n=100 | 1.390 | 120.0n=100 | 84.9%n=100 | $0.09786100/100 successes priced |
| parasail/fp8 | 100 | 0/1000.0–3.7% interval | 0.556n=100 | 0.825 | 76.8n=100 | 68.0%n=100 | $0.10475100/100 successes priced |
| relace/fp4 | 100 | 0/1000.0–3.7% interval | 0.531n=100 | 0.756 | 260.2n=100 | 97.6%n=100 | $0.03451100/100 successes priced |
| siliconflow/fp8 | 100 | 0/1000.0–3.7% interval | 1.149n=100 | 1.417 | 100.7n=100 | 96.6%n=100 | $0.09755100/100 successes priced |
| telnyx | 100 | 0/1000.0–3.7% interval | 0.454n=100 | 0.527 | 383.8n=100 | 98.6%n=98 | not reported0/100 successes priced |
| together | 100 | 0/1000.0–3.7% interval | 0.562n=100 | 0.730 | 138.9n=100 | 97.9%n=100 | $0.05792100/100 successes priced |
1k input / 1k output, cold
Lowest median first-token latency: coreweave/fp8. Highest median decode speed: telnyx.
| Route | Included calls | Failures | First token p50, s | First token p90, s | Decode, tok/s | Cache share | Effective cost / 1M prompt tokens |
|---|---|---|---|---|---|---|---|
| atlas-cloud/fp4 | 100 | 0/1000.0–3.7% interval | 1.875n=100 | 6.038 | 113.2n=99 | 0.0%n=100 | $0.62213100/100 successes priced |
| coreweave/fp8 | 100 | 0/1000.0–3.7% interval | 0.371n=100 | 0.494 | 80.8n=100 | 0.0%n=100 | $0.16532100/100 successes priced |
| deepseek | 100 | 0/1000.0–3.7% interval | 0.716n=100 | 0.977 | 109.6n=100 | 0.0%n=100 | $0.32011100/100 successes priced |
| doubleword-flex | 100 | 0/1000.0–3.7% interval | 2.847n=100 | 3.354 | not reportedn=0 | 0.0%n=100 | $0.09095n=80 |
| doubleword-realtime | 100 | 2/1000.6–7.0% interval | 0.471n=98 | 0.517 | 79.6n=98 | 0.0%n=98 | $0.11692n=80 |
| gmicloud/fp8 | 100 | 0/1000.0–3.7% interval | 3.567n=100 | 13.340 | 112.6n=100 | 0.0%n=100 | $0.50437100/100 successes priced |
| morph/bf16 | 100 | 66/10056.3–74.5% interval | 0.970n=34 | 1.432 | 70.2n=34 | 0.0%n=34 | $0.1405334/34 successes priced |
| novita/fp8 | 100 | 0/1000.0–3.7% interval | 0.843n=100 | 1.808 | 107.4n=100 | 0.0%n=100 | $0.63120100/100 successes priced |
| openrouter-auto | 100 | 0/1000.0–3.7% interval | 0.871n=100 | 2.766 | 61.4n=100 | 0.0%n=100 | $0.13133100/100 successes priced |
| parasail/fp8 | 100 | 3/1001.0–8.5% interval | 0.816n=97 | 1.148 | 61.2n=97 | 0.0%n=97 | $0.1811797/97 successes priced |
| relace/fp4 | 100 | 0/1000.0–3.7% interval | 0.641n=100 | 0.986 | 108.7n=100 | 0.0%n=100 | $0.09278100/100 successes priced |
| siliconflow/fp8 | 100 | 0/1000.0–3.7% interval | 1.339n=100 | 1.769 | 89.0n=100 | 0.0%n=100 | $0.31646100/100 successes priced |
| telnyx | 100 | 0/1000.0–3.7% interval | 0.453n=100 | 0.524 | 339.5n=100 | not reportedn=0 | not reported0/100 successes priced |
| together | 100 | 0/1000.0–3.7% interval | 0.655n=100 | 1.303 | 72.1n=100 | 0.0%n=100 | $0.18110100/100 successes priced |
1k input / 1k output, warm
Lowest median first-token latency: coreweave/fp8. Highest median decode speed: telnyx.
| Route | Included calls | Failures | First token p50, s | First token p90, s | Decode, tok/s | Cache share | Effective cost / 1M prompt tokens |
|---|---|---|---|---|---|---|---|
| atlas-cloud/fp4 | 100 | 0/1000.0–3.7% interval | 1.026n=100 | 1.325 | 143.3n=100 | 98.6%n=100 | $0.20104100/100 successes priced |
| coreweave/fp8 | 100 | 0/1000.0–3.7% interval | 0.306n=100 | 0.392 | 140.4n=100 | 67.3%n=100 | $0.12656100/100 successes priced |
| deepseek | 100 | 0/1000.0–3.7% interval | 0.843n=100 | 1.077 | 105.6n=100 | 98.6%n=100 | $0.12475100/100 successes priced |
| doubleword-flex | 100 | 0/1000.0–3.7% interval | 1.862n=100 | 2.843 | not reportedn=0 | 100.0%n=100 | $0.01860n=80 |
| doubleword-realtime | 100 | 0/1000.0–3.7% interval | 0.431n=100 | 0.873 | 105.7n=100 | 100.0%n=100 | $0.02424n=80 |
| gmicloud/fp8 | 100 | 0/1000.0–3.7% interval | 2.439n=100 | 6.656 | 127.5n=100 | 70.0%n=100 | $0.24831100/100 successes priced |
| morph/bf16 | 100 | 24/10016.7–33.2% interval | 0.839n=76 | 1.219 | 53.9n=76 | 97.3%n=76 | $0.0602176/76 successes priced |
| novita/fp8 | 100 | 0/1000.0–3.7% interval | 0.933n=100 | 1.148 | 96.9n=100 | 98.6%n=100 | $0.18076100/100 successes priced |
| openrouter-auto | 100 | 0/1000.0–3.7% interval | 0.808n=100 | 1.342 | 102.4n=100 | 93.7%n=100 | $0.12740100/100 successes priced |
| parasail/fp8 | 100 | 0/1000.0–3.7% interval | 0.499n=100 | 0.843 | 71.1n=100 | 96.6%n=100 | $0.08923100/100 successes priced |
| relace/fp4 | 100 | 0/1000.0–3.7% interval | 0.538n=100 | 0.797 | 178.0n=100 | 98.6%n=100 | $0.04086100/100 successes priced |
| siliconflow/fp8 | 100 | 0/1000.0–3.7% interval | 1.102n=100 | 1.415 | 96.7n=100 | 98.6%n=100 | $0.12766100/100 successes priced |
| telnyx | 100 | 0/1000.0–3.7% interval | 0.451n=100 | 0.551 | 346.5n=100 | 98.6%n=100 | not reported0/100 successes priced |
| together | 100 | 0/1000.0–3.7% interval | 0.561n=100 | 0.810 | 129.0n=100 | 98.9%n=100 | $0.06649100/100 successes priced |
10k input / 100 output, cold
Lowest median first-token latency: telnyx. Highest median decode speed: telnyx.
| Route | Included calls | Failures | First token p50, s | First token p90, s | Decode, tok/s | Cache share | Effective cost / 1M prompt tokens |
|---|---|---|---|---|---|---|---|
| atlas-cloud/fp4 | 100 | 0/1000.0–3.7% interval | 2.729n=100 | 6.671 | 127.8n=100 | 0.0%n=100 | $0.45298100/100 successes priced |
| coreweave/fp8 | 100 | 0/1000.0–3.7% interval | 1.243n=100 | 1.445 | 106.3n=100 | 0.0%n=100 | $0.13207100/100 successes priced |
| deepseek | 100 | 0/1000.0–3.7% interval | 1.098n=100 | 1.385 | 118.2n=100 | 0.0%n=100 | $0.22649100/100 successes priced |
| doubleword-flex | 100 | 0/1000.0–3.7% interval | 2.548n=100 | 3.169 | not reportedn=0 | 0.0%n=100 | $0.09095n=80 |
| doubleword-realtime | 100 | 0/1000.0–3.7% interval | 1.235n=100 | 1.685 | 87.5n=100 | 0.0%n=100 | $0.11692n=80 |
| gmicloud/fp8 | 100 | 0/1000.0–3.7% interval | 5.838n=100 | 18.521 | 121.7n=100 | 1.7%n=100 | $0.35639100/100 successes priced |
| morph/bf16 | 100 | 55/10045.2–64.4% interval | 1.730n=45 | 2.484 | 43.3n=45 | 0.0%n=45 | $0.1014745/45 successes priced |
| novita/fp8 | 100 | 0/1000.0–3.7% interval | 1.208n=100 | 2.717 | 117.3n=100 | 0.0%n=100 | $0.45299100/100 successes priced |
| openrouter-auto | 100 | 0/1000.0–3.7% interval | 1.131n=100 | 2.145 | 70.4n=100 | 0.0%n=100 | $0.10235100/100 successes priced |
| parasail/fp8 | 100 | 0/1000.0–3.7% interval | 1.479n=100 | 4.260 | 28.6n=100 | 0.0%n=100 | $0.14275100/100 successes priced |
| relace/fp4 | 100 | 2/1000.6–7.0% interval | 0.931n=98 | 1.508 | 120.1n=98 | 0.0%n=98 | $0.0667698/98 successes priced |
| siliconflow/fp8 | 100 | 0/1000.0–3.7% interval | 1.876n=100 | 2.348 | 100.4n=100 | 0.0%n=100 | $0.22648100/100 successes priced |
| telnyx | 100 | 0/1000.0–3.7% interval | 0.876n=100 | 1.006 | 374.2n=100 | not reportedn=0 | not reported0/100 successes priced |
| together | 100 | 0/1000.0–3.7% interval | 1.077n=100 | 1.718 | 74.2n=100 | 0.0%n=100 | $0.14271100/100 successes priced |
10k input / 100 output, warm
Lowest median first-token latency: coreweave/fp8. Highest median decode speed: telnyx.
| Route | Included calls | Failures | First token p50, s | First token p90, s | Decode, tok/s | Cache share | Effective cost / 1M prompt tokens |
|---|---|---|---|---|---|---|---|
| atlas-cloud/fp4 | 100 | 0/1000.0–3.7% interval | 1.758n=100 | 2.044 | 166.2n=100 | 96.7%n=100 | $0.05460100/100 successes priced |
| coreweave/fp8 | 100 | 0/1000.0–3.7% interval | 0.346n=100 | 0.637 | 134.4n=100 | 93.9%n=100 | $0.07484100/100 successes priced |
| deepseek | 100 | 0/1000.0–3.7% interval | 1.081n=100 | 1.292 | 118.8n=100 | 97.8%n=100 | $0.01819100/100 successes priced |
| doubleword-flex | 100 | 0/1000.0–3.7% interval | 2.032n=100 | 2.536 | not reportedn=0 | 100.0%n=100 | $0.01860n=80 |
| doubleword-realtime | 100 | 0/1000.0–3.7% interval | 1.115n=100 | 1.350 | 104.2n=100 | 99.0%n=100 | $0.02424n=80 |
| gmicloud/fp8 | 100 | 0/1000.0–3.7% interval | 3.368n=100 | 7.487 | 166.2n=100 | 85.4%n=100 | $0.07095100/100 successes priced |
| morph/bf16 | 100 | 18/10011.7–26.7% interval | 0.876n=82 | 1.188 | 60.8n=82 | 92.8%n=82 | $0.0330482/82 successes priced |
| novita/fp8 | 100 | 0/1000.0–3.7% interval | 1.083n=100 | 1.333 | 116.1n=100 | 97.8%n=100 | $0.05007100/100 successes priced |
| openrouter-auto | 100 | 0/1000.0–3.7% interval | 0.883n=100 | 1.802 | 126.5n=100 | 96.0%n=100 | $0.04416100/100 successes priced |
| parasail/fp8 | 100 | 2/1000.6–7.0% interval | 1.472n=97 | 13.978 | 33.1n=97 | 53.0%n=97 | $0.0950497/98 successes priced |
| relace/fp4 | 100 | 0/1000.0–3.7% interval | 0.609n=100 | 0.866 | 264.0n=100 | 97.8%n=100 | $0.01883100/100 successes priced |
| siliconflow/fp8 | 100 | 0/1000.0–3.7% interval | 1.442n=100 | 1.823 | 111.5n=100 | 93.3%n=100 | $0.04738100/100 successes priced |
| telnyx | 100 | 0/1000.0–3.7% interval | 0.714n=100 | 0.810 | 381.8n=100 | 98.8%n=98 | not reported0/100 successes priced |
| together | 100 | 0/1000.0–3.7% interval | 0.586n=100 | 0.841 | 154.8n=100 | 98.0%n=100 | $0.03475100/100 successes priced |
10k input / 1k output, cold
Lowest median first-token latency: relace/fp4. Highest median decode speed: telnyx.
| Route | Included calls | Failures | First token p50, s | First token p90, s | Decode, tok/s | Cache share | Effective cost / 1M prompt tokens |
|---|---|---|---|---|---|---|---|
| atlas-cloud/fp4 | 100 | 0/1000.0–3.7% interval | 2.843n=100 | 12.199 | 123.1n=100 | 0.0%n=100 | $0.46609100/100 successes priced |
| coreweave/fp8 | 100 | 0/1000.0–3.7% interval | 1.229n=100 | 1.380 | 109.3n=100 | 0.0%n=100 | $0.13419100/100 successes priced |
| deepseek | 100 | 0/1000.0–3.7% interval | 1.079n=100 | 1.447 | 116.1n=100 | 0.0%n=100 | $0.23179100/100 successes priced |
| doubleword-flex | 100 | 0/1000.0–3.7% interval | 3.054n=98 | 4.112 | not reportedn=0 | 0.0%n=98 | $0.09095n=80 |
| doubleword-realtime | 100 | 0/1000.0–3.7% interval | 1.204n=100 | 1.534 | 84.8n=100 | 0.0%n=100 | $0.11692n=80 |
| gmicloud/fp8 | 100 | 0/1000.0–3.7% interval | 3.989n=100 | 10.226 | 125.4n=100 | 0.0%n=100 | $0.37901100/100 successes priced |
| morph/bf16 | 100 | 41/10031.9–50.8% interval | 1.637n=59 | 2.473 | 45.1n=59 | 0.0%n=59 | $0.1034659/59 successes priced |
| novita/fp8 | 100 | 0/1000.0–3.7% interval | 1.300n=100 | 2.800 | 111.3n=100 | 0.0%n=100 | $0.46312100/100 successes priced |
| openrouter-auto | 100 | 0/1000.0–3.7% interval | 1.315n=100 | 8.896 | 63.2n=100 | 0.0%n=100 | $0.11254100/100 successes priced |
| parasail/fp8 | 100 | 4/1001.6–9.8% interval | 1.808n=95 | 6.579 | 22.4n=95 | 0.0%n=95 | $0.1447795/96 successes priced |
| relace/fp4 | 100 | 0/1000.0–3.7% interval | 0.867n=100 | 1.223 | 126.2n=100 | 0.0%n=100 | $0.06849100/100 successes priced |
| siliconflow/fp8 | 100 | 0/1000.0–3.7% interval | 1.756n=100 | 2.080 | 92.8n=100 | 0.0%n=100 | $0.23153100/100 successes priced |
| telnyx | 100 | 0/1000.0–3.7% interval | 0.894n=100 | 1.014 | 330.3n=100 | not reportedn=0 | not reported0/100 successes priced |
| together | 100 | 3/1001.0–8.5% interval | 1.098n=97 | 1.698 | 46.1n=97 | 0.0%n=97 | $0.1459497/97 successes priced |
10k input / 1k output, warm
Lowest median first-token latency: coreweave/fp8. Highest median decode speed: telnyx.
| Route | Included calls | Failures | First token p50, s | First token p90, s | Decode, tok/s | Cache share | Effective cost / 1M prompt tokens |
|---|---|---|---|---|---|---|---|
| atlas-cloud/fp4 | 100 | 0/1000.0–3.7% interval | 1.752n=100 | 2.075 | 155.9n=100 | 98.5%n=100 | $0.06108100/100 successes priced |
| coreweave/fp8 | 100 | 0/1000.0–3.7% interval | 0.339n=100 | 0.420 | 137.6n=100 | 98.8%n=100 | $0.07238100/100 successes priced |
| deepseek | 100 | 0/1000.0–3.7% interval | 1.055n=100 | 1.292 | 110.1n=100 | 98.8%n=100 | $0.01950100/100 successes priced |
| doubleword-flex | 100 | 0/1000.0–3.7% interval | 2.532n=100 | 2.717 | not reportedn=0 | 100.0%n=100 | $0.01860n=80 |
| doubleword-realtime | 100 | 0/1000.0–3.7% interval | 1.063n=100 | 1.343 | 139.3n=100 | 100.0%n=100 | $0.02424n=80 |
| gmicloud/fp8 | 100 | 0/1000.0–3.7% interval | 3.766n=100 | 9.144 | 123.8n=100 | 88.2%n=100 | $0.06766100/100 successes priced |
| morph/bf16 | 100 | 27/10019.3–36.4% interval | 0.909n=73 | 1.074 | 67.0n=73 | 98.8%n=73 | $0.0300573/73 successes priced |
| novita/fp8 | 100 | 0/1000.0–3.7% interval | 1.137n=100 | 1.359 | 111.9n=100 | 98.8%n=100 | $0.05290100/100 successes priced |
| openrouter-auto | 100 | 0/1000.0–3.7% interval | 1.165n=100 | 1.697 | 111.2n=100 | 95.8%n=100 | $0.04774100/100 successes priced |
| parasail/fp8 | 100 | 2/1000.6–7.0% interval | 1.050n=98 | 6.347 | 43.0n=98 | 73.6%n=98 | $0.0782098/98 successes priced |
| relace/fp4 | 100 | 0/1000.0–3.7% interval | 0.528n=100 | 0.871 | 300.5n=100 | 98.8%n=100 | $0.01928100/100 successes priced |
| siliconflow/fp8 | 100 | 0/1000.0–3.7% interval | 1.369n=100 | 1.774 | 105.1n=100 | 95.3%n=100 | $0.04728100/100 successes priced |
| telnyx | 100 | 0/1000.0–3.7% interval | 0.717n=100 | 0.789 | 353.1n=100 | 98.8%n=100 | not reported0/100 successes priced |
| together | 100 | 0/1000.0–3.7% interval | 0.625n=100 | 1.010 | 115.6n=100 | 99.0%n=100 | $0.03514100/100 successes priced |
100k input / 100 output, cold
Lowest median first-token latency: telnyx. Highest median decode speed: telnyx.
| Route | Included calls | Failures | First token p50, s | First token p90, s | Decode, tok/s | Cache share | Effective cost / 1M prompt tokens |
|---|---|---|---|---|---|---|---|
| atlas-cloud/fp4 | 30 | 0/300.0–11.4% interval | 5.545n=30 | 6.984 | 154.7n=30 | 0.0%n=30 | $0.4412130/30 successes priced |
| coreweave/fp8 | 30 | 0/300.0–11.4% interval | 10.742n=30 | 11.002 | 126.0n=30 | 0.0%n=30 | $0.1302030/30 successes priced |
| deepseek | 30 | 0/300.0–11.4% interval | 4.463n=30 | 5.216 | 104.9n=30 | 0.0%n=30 | $0.2205630/30 successes priced |
| doubleword-flex | 30 | 0/300.0–11.4% interval | 8.066n=30 | 19.360 | not reportedn=0 | 0.0%n=30 | $0.08795n=16 |
| doubleword-realtime | 30 | 0/300.0–11.4% interval | 5.487n=30 | 10.151 | 98.7n=30 | 0.0%n=30 | $0.11295n=16 |
| gmicloud/fp8 | 30 | 0/300.0–11.4% interval | 8.999n=30 | 18.575 | 171.8n=30 | 0.0%n=30 | $0.3530230/30 successes priced |
| morph/bf16 | 30 | 3/303.5–25.6% interval | 9.465n=25 | 14.680 | 38.6n=25 | 0.0%n=25 | $0.0918625/27 successes priced |
| novita/fp8 | 30 | 0/300.0–11.4% interval | 5.133n=30 | 9.161 | 121.8n=30 | 0.0%n=30 | $0.4412530/30 successes priced |
| openrouter-auto | 30 | 0/300.0–11.4% interval | 4.556n=30 | 8.825 | 93.7n=30 | 5.6%n=30 | $0.0839930/30 successes priced |
| parasail/fp8 | 30 | 0/300.0–11.4% interval | 6.803n=30 | 8.886 | 50.6n=30 | 0.0%n=30 | $0.1402830/30 successes priced |
| relace/fp4 | 30 | 0/300.0–11.4% interval | 4.120n=30 | 6.611 | 139.6n=30 | 0.0%n=30 | $0.0651730/30 successes priced |
| siliconflow/fp8 | 30 | 0/300.0–11.4% interval | 10.669n=30 | 11.707 | 91.3n=30 | 0.0%n=30 | $0.2206130/30 successes priced |
| telnyx | 30 | 0/300.0–11.4% interval | 3.737n=30 | 6.089 | 452.4n=30 | not reportedn=0 | not reported0/30 successes priced |
| together | 30 | 0/300.0–11.4% interval | 3.916n=30 | 5.875 | 63.8n=30 | 0.0%n=30 | $0.1402630/30 successes priced |
100k input / 100 output, warm
Lowest median first-token latency: coreweave/fp8. Highest median decode speed: telnyx.
| Route | Included calls | Failures | First token p50, s | First token p90, s | Decode, tok/s | Cache share | Effective cost / 1M prompt tokens |
|---|---|---|---|---|---|---|---|
| atlas-cloud/fp4 | 30 | 0/300.0–11.4% interval | 4.934n=30 | 5.576 | 174.3n=30 | 96.6%n=30 | $0.0430830/30 successes priced |
| coreweave/fp8 | 30 | 0/300.0–11.4% interval | 1.018n=30 | 1.359 | 131.4n=30 | 96.4%n=30 | $0.0723730/30 successes priced |
| deepseek | 30 | 0/300.0–11.4% interval | 1.756n=30 | 2.226 | 98.9n=30 | 96.6%n=30 | $0.0147630/30 successes priced |
| doubleword-flex | 30 | 0/300.0–11.4% interval | 6.495n=30 | 9.642 | not reportedn=0 | 96.7%n=30 | $0.01893n=16 |
| doubleword-realtime | 30 | 0/300.0–11.4% interval | 5.029n=30 | 8.444 | 103.5n=30 | 96.7%n=30 | $0.02453n=16 |
| gmicloud/fp8 | 30 | 0/300.0–11.4% interval | 6.893n=15 | 15.674 | 131.5n=15 | 77.2%n=15 | $0.0899715/30 successes priced |
| morph/bf16 | 30 | 0/300.0–11.4% interval | 1.852n=30 | 3.792 | 92.1n=30 | 90.0%n=30 | $0.0326530/30 successes priced |
| novita/fp8 | 30 | 0/300.0–11.4% interval | 7.398n=30 | 10.529 | 121.8n=30 | 96.6%n=30 | $0.0431630/30 successes priced |
| openrouter-auto | 30 | 0/300.0–11.4% interval | 1.382n=30 | 3.089 | 110.5n=30 | 96.6%n=30 | $0.0356530/30 successes priced |
| parasail/fp8 | 30 | 0/300.0–11.4% interval | 2.496n=30 | 5.367 | 74.3n=30 | 93.2%n=30 | $0.0563530/30 successes priced |
| relace/fp4 | 30 | 0/300.0–11.4% interval | 1.367n=30 | 1.609 | 181.9n=30 | 96.6%n=30 | $0.0178130/30 successes priced |
| siliconflow/fp8 | 30 | 0/300.0–11.4% interval | 2.804n=30 | 3.812 | 98.8n=30 | 93.3%n=30 | $0.0415030/30 successes priced |
| telnyx | 30 | 0/300.0–11.4% interval | 1.365n=30 | 3.762 | 384.6n=30 | 100.0%n=28 | not reported0/30 successes priced |
| together | 30 | 0/300.0–11.4% interval | 1.028n=30 | 3.972 | 52.2n=30 | 86.7%n=30 | $0.0449230/30 successes priced |
100k input / 1k output, cold
Lowest median first-token latency: together. Highest median decode speed: telnyx.
| Route | Included calls | Failures | First token p50, s | First token p90, s | Decode, tok/s | Cache share | Effective cost / 1M prompt tokens |
|---|---|---|---|---|---|---|---|
| atlas-cloud/fp4 | 30 | 0/300.0–11.4% interval | 5.797n=30 | 7.892 | 180.4n=30 | 0.0%n=30 | $0.4464130/30 successes priced |
| coreweave/fp8 | 13 | 0/130.0–22.8% interval | 10.865n=13 | 11.088 | 130.8n=13 | 0.0%n=13 | $0.1307513/13 successes priced |
| deepseek | 30 | 0/300.0–11.4% interval | 4.418n=30 | 6.241 | 113.7n=30 | 0.0%n=30 | $0.2224030/30 successes priced |
| doubleword-flex | 30 | 0/300.0–11.4% interval | 18.181n=30 | 29.767 | not reportedn=0 | 0.0%n=30 | $0.08795n=16 |
| doubleword-realtime | 30 | 0/300.0–11.4% interval | 8.188n=30 | 11.273 | 86.6n=30 | 0.0%n=30 | $0.11295n=16 |
| gmicloud/fp8 | 16 | 7/1623.1–66.8% interval | 9.676n=9 | 12.183 | 142.1n=9 | 0.0%n=9 | $0.355009/9 successes priced |
| morph/bf16 | 30 | 3/303.5–25.6% interval | 12.850n=27 | 35.091 | 52.5n=27 | 0.0%n=27 | $0.1004927/27 successes priced |
| novita/fp8 | 30 | 0/300.0–11.4% interval | 4.360n=30 | 7.369 | 117.7n=30 | 0.0%n=30 | $0.4436830/30 successes priced |
| openrouter-auto | 13 | 0/130.0–22.8% interval | 5.229n=13 | 11.126 | 114.5n=13 | 6.4%n=13 | $0.1179513/13 successes priced |
| parasail/fp8 | 9 | 0/90.0–29.9% interval | 6.365n=9 | 7.211 | 80.2n=9 | 0.0%n=9 | $0.141059/9 successes priced |
| relace/fp4 | 30 | 0/300.0–11.4% interval | 4.246n=30 | 6.357 | 176.1n=30 | 0.0%n=30 | $0.0657430/30 successes priced |
| siliconflow/fp8 | 16 | 0/160.0–19.4% interval | 10.941n=16 | 11.649 | 84.8n=16 | 0.0%n=16 | $0.2222516/16 successes priced |
| telnyx | 30 | 0/300.0–11.4% interval | 3.729n=30 | 4.227 | 615.9n=30 | not reportedn=0 | not reported0/30 successes priced |
| together | 28 | 0/280.0–12.1% interval | 3.516n=28 | 5.479 | 87.9n=28 | 0.0%n=28 | $0.1415528/28 successes priced |
100k input / 1k output, warm
Lowest median first-token latency: coreweave/fp8. Highest median decode speed: telnyx.
| Route | Included calls | Failures | First token p50, s | First token p90, s | Decode, tok/s | Cache share | Effective cost / 1M prompt tokens |
|---|---|---|---|---|---|---|---|
| atlas-cloud/fp4 | 30 | 0/300.0–11.4% interval | 4.939n=30 | 5.708 | 188.7n=30 | 100.0%n=30 | $0.0330430/30 successes priced |
| coreweave/fp8 | 30 | 0/300.0–11.4% interval | 0.915n=30 | 1.853 | 109.0n=30 | 99.7%n=30 | $0.0706030/30 successes priced |
| deepseek | 30 | 0/300.0–11.4% interval | 1.785n=30 | 2.271 | 99.1n=30 | 100.0%n=30 | $0.0084130/30 successes priced |
| doubleword-flex | 30 | 0/300.0–11.4% interval | 3.404n=30 | 11.153 | not reportedn=0 | 100.0%n=30 | $0.01893n=16 |
| doubleword-realtime | 30 | 0/300.0–11.4% interval | 1.383n=30 | 1.613 | 101.6n=30 | 100.0%n=30 | $0.02453n=16 |
| gmicloud/fp8 | 30 | 0/300.0–11.4% interval | 7.300n=14 | 9.982 | 120.4n=14 | 93.7%n=14 | $0.0376114/30 successes priced |
| morph/bf16 | 30 | 0/300.0–11.4% interval | 1.758n=30 | 1.944 | 113.2n=30 | 100.0%n=30 | $0.0266830/30 successes priced |
| novita/fp8 | 30 | 0/300.0–11.4% interval | 7.345n=30 | 10.173 | 120.0n=30 | 100.0%n=30 | $0.0342830/30 successes priced |
| openrouter-auto | 30 | 0/300.0–11.4% interval | 2.418n=30 | 3.517 | 106.2n=30 | 86.0%n=30 | $0.0510130/30 successes priced |
| parasail/fp8 | 30 | 0/300.0–11.4% interval | 2.201n=30 | 2.641 | 94.2n=30 | 100.0%n=30 | $0.0512530/30 successes priced |
| relace/fp4 | 30 | 0/300.0–11.4% interval | 1.373n=30 | 1.789 | 169.2n=30 | 100.0%n=30 | $0.0166330/30 successes priced |
| siliconflow/fp8 | 30 | 0/300.0–11.4% interval | 2.869n=30 | 3.709 | 90.7n=30 | 100.0%n=30 | $0.0298530/30 successes priced |
| telnyx | 30 | 0/300.0–11.4% interval | 1.336n=30 | 1.490 | 606.7n=30 | 100.0%n=30 | not reported0/30 successes priced |
| together | 30 | 1/300.6–16.7% interval | 1.263n=29 | 2.049 | 44.1n=29 | 100.0%n=29 | $0.0305229/29 successes priced |
Which host wins on your prompts? #
A chat agent, a document pipeline and a batch job stress different parts of this comparison. Use Compound to measure the hosts you are considering with your own messages.
- Install Compound from the repository and set your provider API key.
- Save your messages in
shapes.json. Start with this structure and replace the sample prompt. - Preview the experiment, then add
--goto execute it.
{
"my-workload": {
"messages": [{"role": "user", "content": "Replace this with a representative prompt."}],
"max_tokens": 100
}
}
uv sync --extra dev
compound-bench providers YOUR_MODEL
compound-bench serving --model-or YOUR_MODEL \
--providers openrouter/auto,openrouter/HOST \
--shapes shapes.json --cache-mode both \
--reasoning-modes off --temperature 0 --reps 5
The command previews the call count without spending. Replace YOUR_MODEL and HOST with discovery results, set OPENROUTER_API_KEY in .env, then add --go. Executed requests bill at provider rates; serving runs have no dollar cap.
Explore the data #
Download per-call timings, token usage, status codes and reported costs, or use the computed summaries. The manifest records the run windows and source provenance.
Per-call measurements ↓Summary data ↓Data manifest ↓
How to read these measurements #
Routes describe the tested access path. Quantization suffixes are discovery labels. Calls were collected in controlled runs, so results describe this sample, not a provider-wide guarantee. Zero observed failures does not establish perfect reliability. Repeated calls are dependent observations, and runs at different times can see different capacity.
Cost uses the sum of request charges divided by the corresponding input-token total, multiplied by one million. It includes output charges. Telnyx appears in timing, cache and failure charts; its cost is not available in this report. The manifest contains full source selection details.