# AI API Latency Metrics: TTFT, First Response, and Output Speed

> Source: <https://dev.to/modelflare_dev/ai-api-latency-metrics-ttft-first-response-and-output-speed-mja>
> Published: 2026-10-20 02:09:35+00:00

A request-timeline guide to upstream headers, first SSE event, first effective response, first visible text, end-to-end latency, and output speed.

AI API latency is not one number. For a streaming request, record when the upstream headers arrive, when the first non-empty SSE event arrives, when the first meaningful content or action appears, when the first visible text appears, and when the response completes. These milestones answer different questions and should not all be labeled “TTFT.”

For a text-only chat, time to first visible text is usually the user-facing latency. For a reasoning or tool-using workflow, the first effective output can occur earlier as reasoning metadata or function-call arguments. Generation speed is a separate measurement taken after output starts.

Use a monotonic clock and one origin for all client- or gateway-level timestamps. A streaming request can be represented as follows:

```
t0  gateway receives request
t1  gateway starts upstream request
t2  upstream response headers arrive
t3  first non-empty SSE event arrives
t4  first effective content or action arrives
t5  first visible text delta arrives
t6  upstream body completes or closes
t7  gateway handler finishes
```

Not every request has every milestone. A non-streaming response has no useful SSE timeline. A function-call-only response may never produce visible text. A cancelled stream may have a start but no normal completion. Store absent values as absent rather than as zero, because zero would falsely imply that the event happened immediately.

Modelflare records request-level metadata under `other.timing`. The fields map to the timeline like this:

| Field | What it measures | Important boundary | 
|---|---|---|
| `auth_ms` | Token and user authentication duration | Local phase duration, not upstream time | 
| `distribution_ms` | Channel selection and routing-policy duration | Local phase duration | 
| `body_read_ms` | Time spent reading the downstream request body | Can expose a slow upload before routing | 
| `upstream_headers_ms` | Upstream request start to response headers | Its origin is upstream start, not initial gateway receipt | 
| `first_sse_event_ms` | Gateway receipt to first non-empty upstream SSE event | An event can contain metadata rather than useful output | 
| `first_response_ms` | Gateway receipt to first effective content or action delta | Includes visible text, reasoning summary, function arguments, or custom tool input | 
| `first_text_delta_ms` | Gateway receipt to first visible text delta | Exists only when visible streaming text appears | 
| `upstream_done_ms` | Gateway receipt to upstream body completion or close | A close is not automatically a successful completion | 
| `total_handler_ms` | Gateway receipt to the latest known relay completion point | Closest gateway-side end-to-end duration | 
| `visible_output_tps` | Output tokens divided by time from first visible text to upstream completion | A visible-output diagnostic, not system-wide TPS | 

This mapping prevents a common error: subtracting timestamps that do not share an origin. For example, `upstream_headers_ms` is a duration beginning when the upstream request starts, while `first_response_ms` begins when the gateway receives the request.

The conventional definition of time to first token is the duration from query submission to the first received output token. The [NVIDIA NIM benchmarking guide](https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html) notes that this generally includes network latency, queueing, and prompt prefill, and that empty initial responses should not count.

That definition works for a text-generation benchmark, but modern API streams can emit several things before visible text:

Use explicit names in dashboards and incident reports:

| Name | Recommended meaning | Best use | 
|---|---|---|
| Time to headers | Request start to upstream headers | Network, proxy, and upstream admission diagnosis | 
| Time to first event | Request start to first non-empty SSE event | Transport liveness only | 
| Time to first effective response | Request start to useful content or action | Agent and reasoning workflow responsiveness | 
| Time to first visible text | Request start to text a user can render | Chat and user-perceived responsiveness | 

If a chart says only “TTFT,” document which row it means. Otherwise two teams can report different values for the same request and both appear correct.

End-to-end latency measures the whole request from submission until the full response is received. It includes queueing, prefill, decoding, network transfer, tool pauses represented within the upstream stream, and gateway work visible inside the chosen boundary.

For a simple completed text stream:

```
end_to_end_latency = final_response_time - request_start_time
visible_generation_window = final_response_time - first_visible_text_time
visible_output_tps = output_tokens / visible_generation_window_seconds
```

Inter-token latency, also called time per output token, is normally the average spacing between output tokens after the first token. NVIDIA’s AIPerf definition excludes TTFT and divides the remaining duration by `output_tokens - 1`. Its reciprocal approximates per-user decode throughput for sufficiently long outputs.

Do not calculate token-level ITL by counting SSE events or text Delta objects. One event can contain zero, one, or several tokens, and token boundaries depend on the model tokenizer. Modelflare’s `visible_output_tps` uses the recorded output-token count and the visible generation window; it does not claim to be a per-token event-spacing measurement or total system throughput.

These metrics answer separate questions:

In a text-only flow, `first_response_ms` and `first_text_delta_ms` may be nearly identical. In a reasoning or function-calling flow, the gap can be meaningful.

Suppose a model emits function-call arguments at 1.8 seconds, the application or provider-side tool flow continues, and visible text begins at 6.4 seconds. The system has produced an actionable result at 1.8 seconds, but the user has not seen prose until 6.4 seconds. Calling both values TTFT hides whether the delay occurred before planning, during a tool step, or before answer rendering.

Interpret the fields according to the product experience:

The [AI API Streaming Guide](https://modelflare.dev/articles/ai-api-streaming-guide) covers SSE parsing, cancellation, and idle-timeout behavior. Latency metrics should be computed only after the parser distinguishes envelopes from effective output.

Start at the upstream boundary, then move toward local phases. This avoids blaming authentication or routing when the request spent most of its time waiting for the model.

| Symptom | Primary metric | Likely layer to inspect | Next check | 
|---|---|---|---|
| Slow before headers | `upstream_headers_ms` | Network path, upstream admission, provider queue, proxy route | Compare channel, region, status, and concurrency | 
| Headers are fast, useful output is late | `first_response_ms` minus early phases | Model queueing, prompt prefill, reasoning, upstream scheduling | Compare input tokens, model, route, and inflight count | 
| First event is fast, effective output is late | Gap from `first_sse_event_ms` to`first_response_ms` | Metadata-only events, heartbeats, reasoning startup | Inspect safe event types, not raw content | 
| Effective action is fast, visible text is late | Gap from `first_response_ms` to`first_text_delta_ms` | Tool/reasoning phase or response composition | Check requested tools and output mode | 
| Text begins quickly, then crawls | `visible_output_tps` and`upstream_done_ms` | Decode throughput, contention, long context, network backpressure | Compare output length and channel inflight count | 
| All upstream metrics are normal | `auth_ms` ,`distribution_ms` ,`body_read_ms` | Local auth, policy selection, client upload | Inspect only the phase that is elevated | 

No single metric proves a root cause. High `upstream_headers_ms`, for example, groups several possibilities that require route, region, provider, and concurrent-load evidence to separate.

The following values are illustrative diagnostics, not Modelflare production averages or provider benchmarks.

| Metric | Value | 
|---|---|
| `upstream_headers_ms` | 6,100 ms | 
| `first_sse_event_ms` | 6,300 ms | 
| `first_response_ms` | 6,350 ms | 
| `first_text_delta_ms` | 6,400 ms | 
| `total_handler_ms` | 9,200 ms | 
| `visible_output_tps` | 42 | 

Most of the wait happens before headers. Generation after visible text is comparatively healthy. Check the selected upstream route, provider admission, network path, region, and concurrent load before optimizing the client renderer.

| Metric | Value | 
|---|---|
| `upstream_headers_ms` | 240 ms | 
| `first_sse_event_ms` | 310 ms | 
| `first_response_ms` | 2,900 ms | 
| `first_text_delta_ms` | 8,700 ms | 
| `total_handler_ms` | 10,200 ms | 
| `visible_output_tps` | 55 | 

Transport is alive early and an effective action appears at 2.9 seconds, but visible text waits another 5.8 seconds. If the request uses reasoning or tools, inspect that phase. Increasing a header timeout would not address this pattern.

| Metric | Value | 
|---|---|
| `upstream_headers_ms` | 260 ms | 
| `first_sse_event_ms` | 330 ms | 
| `first_response_ms` | 420 ms | 
| `first_text_delta_ms` | 430 ms | 
| `total_handler_ms` | 20,430 ms | 
| `visible_output_tps` | 9.8 | 

The request becomes visible quickly but spends about 20 seconds generating. Compare output length, context length, selected channel, inflight count, and provider behavior. A first-output timeout would pass and therefore cannot detect this failure mode.

A fair comparison holds the workload contract stable. At minimum, record:

Compare p50, p95, and p99 rather than presenting one average. Keep failures visible: removing timeouts and errors can make a route look faster precisely when it is less reliable. Never compare two providers with different prompts, output lengths, concurrency, or endpoint behavior and present the result as a model-speed ranking.

Latency diagnosis does not require storing prompts, responses, API keys, or plaintext client identities. A useful request record can contain:

`upstream_headers_slow` or `generation_slow_tps`.
Apply retention limits and access controls even to metadata. Request IDs, route choices, timing patterns, and token counts can still reveal operational behavior when combined.

Begin an investigation with the request timeline, not a single TTFT label. Then use [Reliable AI API Routing](https://modelflare.dev/articles/reliable-ai-api-routing) to compare the exact route selected for each attempt and [AI API Error Troubleshooting](https://modelflare.dev/articles/ai-api-error-troubleshooting) to pair timing with the terminal status. This produces evidence that can separate admission delay, model startup, reasoning or tool work, visible generation, and local gateway overhead.

*Originally published at [modelflare.dev](https://modelflare.dev/articles/ai-api-latency-metrics?utm_source=devto&utm_medium=syndication&utm_campaign=blog).*

*Disclosure: I'm part of the Modelflare team, which builds the AI API gateway mentioned in this article.*