{"slug": "production-maas-gateway-benchmark-streaming-ttft-100-way-burst-concurrency-token", "title": "Production MaaS Gateway Benchmark: Streaming TTFT, 100-Way Burst Concurrency & Token Billing Quirks", "summary": "A developer benchmarked the DashScope cloud inference gateway (compatible-mode/v1) with 240 direct requests, measuring streaming time-to-first-token, burst concurrency, and token billing behavior. The tests found qwen3.8-max and qwen3.8-flash handled a 100-way burst with 180/180 HTTP 200 responses and no 429s, but the gateway returned no x-ratelimit-* or Retry-After headers, and the same prompt was billed at 22–23 prompt tokens on max versus 35–36 on flash. The author recommends computing spend from each endpoint's returned usage object and enforcing per-key rate limits and daily spend caps in your own gateway.", "body_md": "In week one of production deployment, systems rarely fail because a model isn't smart enough. They fail because of an unhandled HTTP 429, a streaming response that drops the final `usage` frame, or a monthly bill that doesn't match local token estimates.\n\nIn this benchmark (`2026-11-17`, 240 direct requests against `dashscope.aliyuncs.com/compatible-mode/v1` with `enable_thinking=false` and zero SDK wrappers), we put the **cloud inference gateway layer** under a microscope.\n\n| Dimension | Measured Result | Sample & Scope | \n|---|---|---|\n| **Streaming `c_ttft`** | `qwen3.8-max` : P50`0.585s` / P95`0.857s` ;`qwen3.8-flash` : P50`0.521s` / P95`0.762s` | 20 sequential requests/model, `max_tokens=64` | \n| **Total Request Latency** | `qwen3.8-max` : P50`1.349s` / P95`1.794s` ;`qwen3.8-flash` : P50`1.155s` / P95`1.599s` | Same as above | \n| **Streaming `usage` Frame** | `40/40` returned complete`usage` in the final chunk with`stream_options.include_usage` | 20 streams/model | \n| **`tools` Protocol** | `5/5` per model returned standard`tool_calls` (`query_order` ,`ORD-77` ) | Non-streaming | \n| **`json_object` Mode** | `5/5` per model returned valid`json.loads()` output (`amount_cents=19950` ) | Non-streaming | \n| **10 -> 100-Way Burst** | `180/180` HTTP 200 on`qwen3.8-flash` (`0` HTTP 429s) | `max_tokens=16` , stopped at 100-way burst | \n| **Rate-Limit Headers** | Only `x-request-id` and`x-dashscope-timeout` present; no`x-ratelimit-*` or`Retry-After` | Observed on HTTP 200 responses | \n\n`prompt_tokens` Across Models\nAcross all 40 sequential `T1` requests with the exact same prompt:\n\n`qwen3.8-max` billed `22–23` `qwen3.8-flash` billed `35–36` Never estimate costs using a local tokenizer or a single model's token count. Always compute spend from each endpoint's returned `usage` object.\n\nStepping concurrency from `10 -> 20 -> 50 -> 100` concurrent short requests yielded `180/180` HTTP 200s (100-way wall time: `2.200s`, `c_ttft` P50 `0.565s` / P95 `1.005s`, max `2.187s`).\n\nBecause **no `x-ratelimit-*` headers** are returned on 200s and default burst headroom is high, a runaway Agent loop will burn through your budget long before hitting a platform 429. Per-key rate limits and hard daily spend caps must live in your own gateway.\n\nOn DashScope's published pricing, `qwen3.8-max` Batch File inference is priced at half of real-time (`6 / 18` RMB per 1M input/output tokens vs `12 / 36` real-time; context-cached input drops to `1.5`). Routing overnight summarization and ETL jobs through real-time endpoints doubles your bill unnecessarily.\n\n`usage` payload.\n*Next in **Production AI Benchmarks**: Testing 128K Long-Context \"Needle in a Haystack\" & Multi-Hop Contract Audits.*\n\n`p02_gateway`)", "url": "https://wpnews.pro/news/production-maas-gateway-benchmark-streaming-ttft-100-way-burst-concurrency-token", "canonical_source": "https://dev.to/wangguode/production-maas-gateway-benchmark-streaming-ttft-100-way-burst-concurrency-token-billing-quirks-1cm1", "published_at": "2026-11-21 03:01:51+00:00", "updated_at": "2026-11-21 03:12:00.904093+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "mlops", "ai-tools"], "entities": ["DashScope", "Alibaba Cloud", "qwen3.8-max", "qwen3.8-flash"], "also_reported_by": [], "takeaway": "DashScope returned no x-ratelimit-* or Retry-After headers on 200s and absorbed a 100-way burst with 180/180 successes, while the same prompt billed 22–23 prompt tokens on qwen3.8-max versus 35–36 on qwen3.8-flash, so compute spend from each endpoint's usage object and enforce rate limits and spend ", "recommendation": {"advice": "Compute costs from each endpoint's returned usage object rather than a local tokenizer, and enforce per-key rate limits and hard daily spend caps in your own gateway since the provider returns no rate-limit headers.", "playbook": "llm-cost", "playbook_url": "https://wpnews.pro/playbooks/llm-cost"}, "alternates": {"html": "https://wpnews.pro/news/production-maas-gateway-benchmark-streaming-ttft-100-way-burst-concurrency-token", "markdown": "https://wpnews.pro/news/production-maas-gateway-benchmark-streaming-ttft-100-way-burst-concurrency-token.md", "text": "https://wpnews.pro/news/production-maas-gateway-benchmark-streaming-ttft-100-way-burst-concurrency-token.txt", "jsonld": "https://wpnews.pro/news/production-maas-gateway-benchmark-streaming-ttft-100-way-burst-concurrency-token.jsonld"}}