# Production MaaS Gateway Benchmark: Streaming TTFT, 100-Way Burst Concurrency & Token Billing Quirks

> Source: <https://dev.to/wangguode/production-maas-gateway-benchmark-streaming-ttft-100-way-burst-concurrency-token-billing-quirks-1cm1>
> Published: 2026-11-21 03:01:51+00:00

**Recommendation:** Compute costs from each endpoint's returned usage object rather than a local tokenizer, and enforce per-key rate limits and hard daily spend caps in your own gateway since the provider returns no rate-limit headers.
More in the playbook "Controlling LLM cost": <https://wpnews.pro/playbooks/llm-cost>

In week one of production deployment, systems rarely fail because a model isn't smart enough. They fail because of an unhandled HTTP 429, a streaming response that drops the final `usage` frame, or a monthly bill that doesn't match local token estimates.

In this benchmark (`2026-11-17`, 240 direct requests against `dashscope.aliyuncs.com/compatible-mode/v1` with `enable_thinking=false` and zero SDK wrappers), we put the **cloud inference gateway layer** under a microscope.

| Dimension | Measured Result | Sample & Scope | 
|---|---|---|
| **Streaming `c_ttft`** | `qwen3.8-max` : P50`0.585s` / P95`0.857s` ;`qwen3.8-flash` : P50`0.521s` / P95`0.762s` | 20 sequential requests/model, `max_tokens=64` | 
| **Total Request Latency** | `qwen3.8-max` : P50`1.349s` / P95`1.794s` ;`qwen3.8-flash` : P50`1.155s` / P95`1.599s` | Same as above | 
| **Streaming `usage` Frame** | `40/40` returned complete`usage` in the final chunk with`stream_options.include_usage` | 20 streams/model | 
| **`tools` Protocol** | `5/5` per model returned standard`tool_calls` (`query_order` ,`ORD-77` ) | Non-streaming | 
| **`json_object` Mode** | `5/5` per model returned valid`json.loads()` output (`amount_cents=19950` ) | Non-streaming | 
| **10 -> 100-Way Burst** | `180/180` HTTP 200 on`qwen3.8-flash` (`0` HTTP 429s) | `max_tokens=16` , stopped at 100-way burst | 
| **Rate-Limit Headers** | Only `x-request-id` and`x-dashscope-timeout` present; no`x-ratelimit-*` or`Retry-After` | Observed on HTTP 200 responses | 

`prompt_tokens` Across Models
Across all 40 sequential `T1` requests with the exact same prompt:

`qwen3.8-max` billed `22–23` `qwen3.8-flash` billed `35–36` Never estimate costs using a local tokenizer or a single model's token count. Always compute spend from each endpoint's returned `usage` object.

Stepping concurrency from `10 -> 20 -> 50 -> 100` concurrent short requests yielded `180/180` HTTP 200s (100-way wall time: `2.200s`, `c_ttft` P50 `0.565s` / P95 `1.005s`, max `2.187s`).

Because **no `x-ratelimit-*` headers** are returned on 200s and default burst headroom is high, a runaway Agent loop will burn through your budget long before hitting a platform 429. Per-key rate limits and hard daily spend caps must live in your own gateway.

On DashScope's published pricing, `qwen3.8-max` Batch File inference is priced at half of real-time (`6 / 18` RMB per 1M input/output tokens vs `12 / 36` real-time; context-cached input drops to `1.5`). Routing overnight summarization and ETL jobs through real-time endpoints doubles your bill unnecessarily.

`usage` payload.
*Next in **Production AI Benchmarks**: Testing 128K Long-Context "Needle in a Haystack" & Multi-Hop Contract Audits.*

`p02_gateway`)
