In week one of production deployment, systems rarely fail because a model isn't smart enough. They fail because of an unhandled HTTP 429, a streaming response that drops the final usage frame, or a monthly bill that doesn't match local token estimates.
In this benchmark (2026-11-17, 240 direct requests against dashscope.aliyuncs.com/compatible-mode/v1 with enable_thinking=false and zero SDK wrappers), we put the cloud inference gateway layer under a microscope.
| Dimension | Measured Result | Sample & Scope |
|---|---|---|
Streaming c_ttft |
qwen3.8-max : P500.585s / P950.857s ;qwen3.8-flash : P500.521s / P950.762s |
20 sequential requests/model, max_tokens=64 |
| Total Request Latency | qwen3.8-max : P501.349s / P951.794s ;qwen3.8-flash : P501.155s / P951.599s |
Same as above |
Streaming usage Frame |
40/40 returned completeusage in the final chunk withstream_options.include_usage |
20 streams/model |
tools Protocol |
5/5 per model returned standardtool_calls (query_order ,ORD-77 ) |
Non-streaming |
json_object Mode |
5/5 per model returned validjson.loads() output (amount_cents=19950 ) |
Non-streaming |
| **10 -> 100-Way Burst** | `180/180` HTTP 200 on`qwen3.8-flash` (`0` HTTP 429s) | `max_tokens=16` , stopped at 100-way burst |
| **Rate-Limit Headers** | Only `x-request-id` and`x-dashscope-timeout` present; no`x-ratelimit-*` or`Retry-After` | Observed on HTTP 200 responses |
prompt_tokens Across Models
Across all 40 sequential T1 requests with the exact same prompt:
qwen3.8-max billed 22–23 qwen3.8-flash billed 35–36 Never estimate costs using a local tokenizer or a single model's token count. Always compute spend from each endpoint's returned usage object.
Stepping concurrency from 10 -> 20 -> 50 -> 100 concurrent short requests yielded 180/180 HTTP 200s (100-way wall time: 2.200s, c_ttft P50 0.565s / P95 1.005s, max 2.187s).
Because no x-ratelimit-* headers are returned on 200s and default burst headroom is high, a runaway Agent loop will burn through your budget long before hitting a platform 429. Per-key rate limits and hard daily spend caps must live in your own gateway.
On DashScope's published pricing, qwen3.8-max Batch File inference is priced at half of real-time (6 / 18 RMB per 1M input/output tokens vs 12 / 36 real-time; context-cached input drops to 1.5). Routing overnight summarization and ETL jobs through real-time endpoints doubles your bill unnecessarily.
usage payload.
Next in Production AI Benchmarks: Testing 128K Long-Context "Needle in a Haystack" & Multi-Hop Contract Audits.
p02_gateway)