cd /news/ai-infrastructure/production-maas-gateway-benchmark-st… · home › topics › ai-infrastructure › article
[ARTICLE · art-173609] src=dev.to ↗ pub= topic=ai-infrastructure verified=true sentiment=· neutral

Production MaaS Gateway Benchmark: Streaming TTFT, 100-Way Burst Concurrency & Token Billing Quirks

A developer benchmarked the DashScope cloud inference gateway (compatible-mode/v1) with 240 direct requests, measuring streaming time-to-first-token, burst concurrency, and token billing behavior. The tests found qwen3.8-max and qwen3.8-flash handled a 100-way burst with 180/180 HTTP 200 responses and no 429s, but the gateway returned no x-ratelimit-* or Retry-After headers, and the same prompt was billed at 22–23 prompt tokens on max versus 35–36 on flash. The author recommends computing spend from each endpoint's returned usage object and enforcing per-key rate limits and daily spend caps in your own gateway.

recommendation

Compute costs from each endpoint's returned usage object rather than a local tokenizer, and enforce per-key rate limits and hard daily spend caps in your own gateway since the provider returns no rate-limit headers.

more in playbook: Controlling LLM cost
by read2 min views5 publishedNov 21, 2026

In week one of production deployment, systems rarely fail because a model isn't smart enough. They fail because of an unhandled HTTP 429, a streaming response that drops the final usage frame, or a monthly bill that doesn't match local token estimates.

In this benchmark (2026-11-17, 240 direct requests against dashscope.aliyuncs.com/compatible-mode/v1 with enable_thinking=false and zero SDK wrappers), we put the cloud inference gateway layer under a microscope.

Dimension Measured Result Sample & Scope
Streaming c_ttft qwen3.8-max : P500.585s / P950.857s ;qwen3.8-flash : P500.521s / P950.762s 20 sequential requests/model, max_tokens=64
Total Request Latency qwen3.8-max : P501.349s / P951.794s ;qwen3.8-flash : P501.155s / P951.599s Same as above
Streaming usage Frame 40/40 returned completeusage in the final chunk withstream_options.include_usage 20 streams/model
tools Protocol 5/5 per model returned standardtool_calls (query_order ,ORD-77 ) Non-streaming
json_object Mode 5/5 per model returned validjson.loads() output (amount_cents=19950 ) Non-streaming
| **10 -> 100-Way Burst** | `180/180` HTTP 200 on`qwen3.8-flash` (`0` HTTP 429s) | `max_tokens=16` , stopped at 100-way burst | 
| **Rate-Limit Headers** | Only `x-request-id` and`x-dashscope-timeout` present; no`x-ratelimit-*` or`Retry-After` | Observed on HTTP 200 responses | 

prompt_tokens Across Models Across all 40 sequential T1 requests with the exact same prompt:

qwen3.8-max billed 22–23 qwen3.8-flash billed 35–36 Never estimate costs using a local tokenizer or a single model's token count. Always compute spend from each endpoint's returned usage object.

Stepping concurrency from 10 -> 20 -> 50 -> 100 concurrent short requests yielded 180/180 HTTP 200s (100-way wall time: 2.200s, c_ttft P50 0.565s / P95 1.005s, max 2.187s). Because no x-ratelimit-* headers are returned on 200s and default burst headroom is high, a runaway Agent loop will burn through your budget long before hitting a platform 429. Per-key rate limits and hard daily spend caps must live in your own gateway.

On DashScope's published pricing, qwen3.8-max Batch File inference is priced at half of real-time (6 / 18 RMB per 1M input/output tokens vs 12 / 36 real-time; context-cached input drops to 1.5). Routing overnight summarization and ETL jobs through real-time endpoints doubles your bill unnecessarily.

usage payload. Next in Production AI Benchmarks: Testing 128K Long-Context "Needle in a Haystack" & Multi-Hop Contract Audits.

p02_gateway)

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @dashscope 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/production-maas-gate…] indexed:0 read:2min 2026-11-21 · —