AI API Latency Metrics: TTFT, First Response, and Output Speed

Modelflare documented a request-timeline approach for measuring AI API latency, breaking the single "TTFT" figure into distinct milestones: upstream headers, first non-empty SSE event, first effective response, first visible text, upstream completion, and handler finish. The company's gateway records request-level metadata under other.timing, including auth_ms, upstream_headers_ms, first_sse_event_ms, first_response_ms, first_text_delta_ms, upstream_done_ms, total_handler_ms, and visible_output_tps. It warns that subtracting timestamps with different origins produces wrong latency numbers and that absent milestones should be stored as absent rather than zero.

A request-timeline guide to upstream headers, first SSE event, first effective response, first visible text, end-to-end latency, and output speed. AI API latency is not one number. For a streaming request, record when the upstream headers arrive, when the first non-empty SSE event arrives, when the first meaningful content or action appears, when the first visible text appears, and when the response completes. These milestones answer different questions and should not all be labeled “TTFT.” For a text-only chat, time to first visible text is usually the user-facing latency. For a reasoning or tool-using workflow, the first effective output can occur earlier as reasoning metadata or function-call arguments. Generation speed is a separate measurement taken after output starts. Use a monotonic clock and one origin for all client- or gateway-level timestamps. A streaming request can be represented as follows: t0 gateway receives request t1 gateway starts upstream request t2 upstream response headers arrive t3 first non-empty SSE event arrives t4 first effective content or action arrives t5 first visible text delta arrives t6 upstream body completes or closes t7 gateway handler finishes Not every request has every milestone. A non-streaming response has no useful SSE timeline. A function-call-only response may never produce visible text. A cancelled stream may have a start but no normal completion. Store absent values as absent rather than as zero, because zero would falsely imply that the event happened immediately. Modelflare records request-level metadata under other.timing . The fields map to the timeline like this: | Field | What it measures | Important boundary | |---|---|---| | auth ms | Token and user authentication duration | Local phase duration, not upstream time | | distribution ms | Channel selection and routing-policy duration | Local phase duration | | body read ms | Time spent reading the downstream request body | Can expose a slow upload before routing | | upstream headers ms | Upstream request start to response headers | Its origin is upstream start, not initial gateway receipt | | first sse event ms | Gateway receipt to first non-empty upstream SSE event | An event can contain metadata rather than useful output | | first response ms | Gateway receipt to first effective content or action delta | Includes visible text, reasoning summary, function arguments, or custom tool input | | first text delta ms | Gateway receipt to first visible text delta | Exists only when visible streaming text appears | | upstream done ms | Gateway receipt to upstream body completion or close | A close is not automatically a successful completion | | total handler ms | Gateway receipt to the latest known relay completion point | Closest gateway-side end-to-end duration | | visible output tps | Output tokens divided by time from first visible text to upstream completion | A visible-output diagnostic, not system-wide TPS | This mapping prevents a common error: subtracting timestamps that do not share an origin. For example, upstream headers ms is a duration beginning when the upstream request starts, while first response ms begins when the gateway receives the request. The conventional definition of time to first token is the duration from query submission to the first received output token. The NVIDIA NIM benchmarking guide https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html notes that this generally includes network latency, queueing, and prompt prefill, and that empty initial responses should not count. That definition works for a text-generation benchmark, but modern API streams can emit several things before visible text: Use explicit names in dashboards and incident reports: | Name | Recommended meaning | Best use | |---|---|---| | Time to headers | Request start to upstream headers | Network, proxy, and upstream admission diagnosis | | Time to first event | Request start to first non-empty SSE event | Transport liveness only | | Time to first effective response | Request start to useful content or action | Agent and reasoning workflow responsiveness | | Time to first visible text | Request start to text a user can render | Chat and user-perceived responsiveness | If a chart says only “TTFT,” document which row it means. Otherwise two teams can report different values for the same request and both appear correct. End-to-end latency measures the whole request from submission until the full response is received. It includes queueing, prefill, decoding, network transfer, tool pauses represented within the upstream stream, and gateway work visible inside the chosen boundary. For a simple completed text stream: end to end latency = final response time - request start time visible generation window = final response time - first visible text time visible output tps = output tokens / visible generation window seconds Inter-token latency, also called time per output token, is normally the average spacing between output tokens after the first token. NVIDIA’s AIPerf definition excludes TTFT and divides the remaining duration by output tokens - 1 . Its reciprocal approximates per-user decode throughput for sufficiently long outputs. Do not calculate token-level ITL by counting SSE events or text Delta objects. One event can contain zero, one, or several tokens, and token boundaries depend on the model tokenizer. Modelflare’s visible output tps uses the recorded output-token count and the visible generation window; it does not claim to be a per-token event-spacing measurement or total system throughput. These metrics answer separate questions: In a text-only flow, first response ms and first text delta ms may be nearly identical. In a reasoning or function-calling flow, the gap can be meaningful. Suppose a model emits function-call arguments at 1.8 seconds, the application or provider-side tool flow continues, and visible text begins at 6.4 seconds. The system has produced an actionable result at 1.8 seconds, but the user has not seen prose until 6.4 seconds. Calling both values TTFT hides whether the delay occurred before planning, during a tool step, or before answer rendering. Interpret the fields according to the product experience: The AI API Streaming Guide https://modelflare.dev/articles/ai-api-streaming-guide covers SSE parsing, cancellation, and idle-timeout behavior. Latency metrics should be computed only after the parser distinguishes envelopes from effective output. Start at the upstream boundary, then move toward local phases. This avoids blaming authentication or routing when the request spent most of its time waiting for the model. | Symptom | Primary metric | Likely layer to inspect | Next check | |---|---|---|---| | Slow before headers | upstream headers ms | Network path, upstream admission, provider queue, proxy route | Compare channel, region, status, and concurrency | | Headers are fast, useful output is late | first response ms minus early phases | Model queueing, prompt prefill, reasoning, upstream scheduling | Compare input tokens, model, route, and inflight count | | First event is fast, effective output is late | Gap from first sse event ms to first response ms | Metadata-only events, heartbeats, reasoning startup | Inspect safe event types, not raw content | | Effective action is fast, visible text is late | Gap from first response ms to first text delta ms | Tool/reasoning phase or response composition | Check requested tools and output mode | | Text begins quickly, then crawls | visible output tps and upstream done ms | Decode throughput, contention, long context, network backpressure | Compare output length and channel inflight count | | All upstream metrics are normal | auth ms , distribution ms , body read ms | Local auth, policy selection, client upload | Inspect only the phase that is elevated | No single metric proves a root cause. High upstream headers ms , for example, groups several possibilities that require route, region, provider, and concurrent-load evidence to separate. The following values are illustrative diagnostics, not Modelflare production averages or provider benchmarks. | Metric | Value | |---|---| | upstream headers ms | 6,100 ms | | first sse event ms | 6,300 ms | | first response ms | 6,350 ms | | first text delta ms | 6,400 ms | | total handler ms | 9,200 ms | | visible output tps | 42 | Most of the wait happens before headers. Generation after visible text is comparatively healthy. Check the selected upstream route, provider admission, network path, region, and concurrent load before optimizing the client renderer. | Metric | Value | |---|---| | upstream headers ms | 240 ms | | first sse event ms | 310 ms | | first response ms | 2,900 ms | | first text delta ms | 8,700 ms | | total handler ms | 10,200 ms | | visible output tps | 55 | Transport is alive early and an effective action appears at 2.9 seconds, but visible text waits another 5.8 seconds. If the request uses reasoning or tools, inspect that phase. Increasing a header timeout would not address this pattern. | Metric | Value | |---|---| | upstream headers ms | 260 ms | | first sse event ms | 330 ms | | first response ms | 420 ms | | first text delta ms | 430 ms | | total handler ms | 20,430 ms | | visible output tps | 9.8 | The request becomes visible quickly but spends about 20 seconds generating. Compare output length, context length, selected channel, inflight count, and provider behavior. A first-output timeout would pass and therefore cannot detect this failure mode. A fair comparison holds the workload contract stable. At minimum, record: Compare p50, p95, and p99 rather than presenting one average. Keep failures visible: removing timeouts and errors can make a route look faster precisely when it is less reliable. Never compare two providers with different prompts, output lengths, concurrency, or endpoint behavior and present the result as a model-speed ranking. Latency diagnosis does not require storing prompts, responses, API keys, or plaintext client identities. A useful request record can contain: upstream headers slow or generation slow tps . Apply retention limits and access controls even to metadata. Request IDs, route choices, timing patterns, and token counts can still reveal operational behavior when combined. Begin an investigation with the request timeline, not a single TTFT label. Then use Reliable AI API Routing https://modelflare.dev/articles/reliable-ai-api-routing to compare the exact route selected for each attempt and AI API Error Troubleshooting https://modelflare.dev/articles/ai-api-error-troubleshooting to pair timing with the terminal status. This produces evidence that can separate admission delay, model startup, reasoning or tool work, visible generation, and local gateway overhead. Originally published at modelflare.dev https://modelflare.dev/articles/ai-api-latency-metrics?utm source=devto&utm medium=syndication&utm campaign=blog . Disclosure: I'm part of the Modelflare team, which builds the AI API gateway mentioned in this article.