Production MaaS Gateway Benchmark: Streaming TTFT, 100-Way Burst Concurrency & Token Billing Quirks A developer benchmarked the DashScope cloud inference gateway (compatible-mode/v1) with 240 direct requests, measuring streaming time-to-first-token, burst concurrency, and token billing behavior. The tests found qwen3.8-max and qwen3.8-flash handled a 100-way burst with 180/180 HTTP 200 responses and no 429s, but the gateway returned no x-ratelimit-* or Retry-After headers, and the same prompt was billed at 22–23 prompt tokens on max versus 35–36 on flash. The author recommends computing spend from each endpoint's returned usage object and enforcing per-key rate limits and daily spend caps in your own gateway. In week one of production deployment, systems rarely fail because a model isn't smart enough. They fail because of an unhandled HTTP 429, a streaming response that drops the final usage frame, or a monthly bill that doesn't match local token estimates. In this benchmark 2026-11-17 , 240 direct requests against dashscope.aliyuncs.com/compatible-mode/v1 with enable thinking=false and zero SDK wrappers , we put the cloud inference gateway layer under a microscope. | Dimension | Measured Result | Sample & Scope | |---|---|---| | Streaming c ttft | qwen3.8-max : P50 0.585s / P95 0.857s ; qwen3.8-flash : P50 0.521s / P95 0.762s | 20 sequential requests/model, max tokens=64 | | Total Request Latency | qwen3.8-max : P50 1.349s / P95 1.794s ; qwen3.8-flash : P50 1.155s / P95 1.599s | Same as above | | Streaming usage Frame | 40/40 returned complete usage in the final chunk with stream options.include usage | 20 streams/model | | tools Protocol | 5/5 per model returned standard tool calls query order , ORD-77 | Non-streaming | | json object Mode | 5/5 per model returned valid json.loads output amount cents=19950 | Non-streaming | | 10 - 100-Way Burst | 180/180 HTTP 200 on qwen3.8-flash 0 HTTP 429s | max tokens=16 , stopped at 100-way burst | | Rate-Limit Headers | Only x-request-id and x-dashscope-timeout present; no x-ratelimit- or Retry-After | Observed on HTTP 200 responses | prompt tokens Across Models Across all 40 sequential T1 requests with the exact same prompt: qwen3.8-max billed 22–23 qwen3.8-flash billed 35–36 Never estimate costs using a local tokenizer or a single model's token count. Always compute spend from each endpoint's returned usage object. Stepping concurrency from 10 - 20 - 50 - 100 concurrent short requests yielded 180/180 HTTP 200s 100-way wall time: 2.200s , c ttft P50 0.565s / P95 1.005s , max 2.187s . Because no x-ratelimit- headers are returned on 200s and default burst headroom is high, a runaway Agent loop will burn through your budget long before hitting a platform 429. Per-key rate limits and hard daily spend caps must live in your own gateway. On DashScope's published pricing, qwen3.8-max Batch File inference is priced at half of real-time 6 / 18 RMB per 1M input/output tokens vs 12 / 36 real-time; context-cached input drops to 1.5 . Routing overnight summarization and ETL jobs through real-time endpoints doubles your bill unnecessarily. usage payload. Next in Production AI Benchmarks : Testing 128K Long-Context "Needle in a Haystack" & Multi-Hop Contract Audits. p02 gateway