# Model-neutral Pi extension for live decode TPS, TTFT, and prompt-rate estimates

> Source: <https://gist.github.com/Anemll/f95a14877862f289e19b12586850eded>
> Published: 2026-09-13 06:29:42+00:00

This guide adds a model-neutral throughput indicator to
[Pi](https://github.com/earendil-works/pi). It was validated against
`@earendil-works/pi-coding-agent` 0.85.1.

The extension works with local and hosted models because it observes Pi's
standard assistant-stream events. It does **not** read vLLM logs or metrics and
does not depend on a particular model name.

During generation, the footer looks similar to:

```
⚡ TTFT: 1.24s · Decode: ~42.1 tok/s · ~312 tok
```

After generation, when the provider reports token usage, it becomes:

```
⚡ TTFT: 1.24s · Input/TTFT: ~1850 tok/s · Decode: 39.8 tok/s · 842 tok
```

- **TTFT** is measured from Pi's`before_provider_request` event to the first
output delta observed by the client.
- **Live decode TPS** is a`characters / 4` estimate because most providers do
not report a cumulative token count on every stream chunk.
- **Final decode TPS** uses the provider's reported output-token count and the
client-observed interval from the first output delta to the last. The first
token is excluded from the numerator because it defines the start boundary.
- **Input/TTFT** is only a client-side prompt-rate estimate. It divides uncached
input tokens by TTFT, which also contains network, queue, scheduling, and
stream-start overhead. It is**not** authoritative server prefill throughput.

For true per-request prefill throughput, use metrics or timing emitted by the inference server. Pi's message events do not expose a server-only prefill duration.

Create `~/.pi/agent/extensions/live-throughput-status.ts` with this content:

``` python
import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent";

const STATUS_KEY = "live-throughput";
const CHARS_PER_TOKEN = 4;
const UPDATE_INTERVAL_MS = 200;

function setStatus(ctx: ExtensionContext, text?: string): void {
	if (ctx.mode !== "tui") return;
	ctx.ui.setStatus(
		STATUS_KEY,
		text === undefined ? undefined : ctx.ui.theme.fg("accent", `⚡ ${text}`),
	);
}

function positiveNumber(value: unknown): number | undefined {
	return typeof value === "number" && Number.isFinite(value) && value > 0
		? value
		: undefined;
}

function deltaChars(event: unknown): number {
	if (!event || typeof event !== "object") return 0;
	const streamEvent = event as { type?: string; delta?: unknown };
	if (
		streamEvent.type !== "text_delta" &&
		streamEvent.type !== "thinking_delta" &&
		streamEvent.type !== "toolcall_delta"
	) {
		return 0;
	}
	return typeof streamEvent.delta === "string" ? streamEvent.delta.length : 0;
}

function seconds(milliseconds: number): number {
	return Math.max(0, milliseconds) / 1000;
}

function rate(value: number, durationSeconds: number): string {
	return (value / Math.max(0.001, durationSeconds)).toFixed(1);
}

export default function (pi: ExtensionAPI) {
	let requestStartedAt: number | undefined;
	let firstOutputAt: number | undefined;
	let lastOutputAt: number | undefined;
	let ttftSeconds: number | undefined;
	let streamedChars = 0;
	let lastDisplayAt = 0;

	function reset(ctx: ExtensionContext, waiting = true): void {
		requestStartedAt = undefined;
		firstOutputAt = undefined;
		lastOutputAt = undefined;
		ttftSeconds = undefined;
		streamedChars = 0;
		lastDisplayAt = 0;
		setStatus(ctx, waiting ? "TTFT: waiting · Decode: waiting" : undefined);
	}

	pi.on("session_start", async (_event, ctx) => reset(ctx));
	pi.on("model_select", async (_event, ctx) => reset(ctx));

	// This event occurs immediately before Pi sends a provider payload. The
	// handler observes time only and deliberately returns no payload rewrite.
	pi.on("before_provider_request", async (_event, _ctx) => {
		requestStartedAt = Date.now();
	});

	pi.on("message_start", async (event, ctx) => {
		if (event.message.role !== "assistant") return;
		firstOutputAt = undefined;
		lastOutputAt = undefined;
		ttftSeconds = undefined;
		streamedChars = 0;
		lastDisplayAt = 0;
		// Fallback for a custom provider that does not emit the request hook.
		requestStartedAt ??= Date.now();
		setStatus(ctx, "TTFT: waiting · Decode: waiting for first token…");
	});

	pi.on("message_update", async (event, ctx) => {
		if (event.message.role !== "assistant") return;
		const chars = deltaChars(event.assistantMessageEvent);
		if (chars <= 0) return;

		const now = Date.now();
		if (firstOutputAt === undefined) {
			firstOutputAt = now;
			ttftSeconds = seconds(now - (requestStartedAt ?? now));
		}
		lastOutputAt = now;
		streamedChars += chars;

		if (now - lastDisplayAt < UPDATE_INTERVAL_MS) return;
		lastDisplayAt = now;
		const decodeSeconds = seconds(now - firstOutputAt);
		const estimatedTokens = streamedChars / CHARS_PER_TOKEN;
		setStatus(
			ctx,
			`TTFT: ${ttftSeconds!.toFixed(2)}s · Decode: ~${rate(estimatedTokens, decodeSeconds)} tok/s · ~${Math.round(estimatedTokens)} tok`,
		);
	});

	pi.on("message_end", async (event, ctx) => {
		if (event.message.role !== "assistant") return;

		const usage = event.message.usage;
		const outputTokens = positiveNumber(usage?.output);
		const uncachedInputTokens = positiveNumber(usage?.input);
		const cacheWriteTokens = positiveNumber(usage?.cacheWrite) ?? 0;
		const observedTtft = ttftSeconds;
		const decodeSeconds =
			firstOutputAt !== undefined && lastOutputAt !== undefined
				? seconds(lastOutputAt - firstOutputAt)
				: undefined;

		const parts: string[] = [];
		if (observedTtft !== undefined) {
			parts.push(`TTFT: ${observedTtft.toFixed(2)}s`);
		}

		// Pi normalizes usage.input to uncached input for supported providers.
		// Cache reads are excluded because they were not prefetched by the model.
		const processedInputTokens = (uncachedInputTokens ?? 0) + cacheWriteTokens;
		if (processedInputTokens > 0 && observedTtft !== undefined && observedTtft > 0) {
			parts.push(`Input/TTFT: ~${rate(processedInputTokens, observedTtft)} tok/s`);
		}

		if (outputTokens !== undefined && outputTokens > 1 && decodeSeconds !== undefined && decodeSeconds > 0) {
			parts.push(`Decode: ${rate(outputTokens - 1, decodeSeconds)} tok/s · ${outputTokens} tok`);
		} else if (outputTokens !== undefined) {
			parts.push(`Decode: ${outputTokens} tok · rate unavailable`);
		} else if (streamedChars > 0 && decodeSeconds !== undefined && decodeSeconds > 0) {
			const estimatedTokens = streamedChars / CHARS_PER_TOKEN;
			parts.push(`Decode: ~${rate(estimatedTokens, decodeSeconds)} tok/s · ~${Math.round(estimatedTokens)} tok`);
		} else {
			parts.push("Decode: no output tokens");
		}

		setStatus(ctx, parts.join(" · "));
		requestStartedAt = undefined;
	});
}
```

Run `/reload` in Pi. The global extension directory is auto-discovered, so no
additional settings entry is required.

The extension itself is model-neutral. Accuracy depends on the provider:

| Provider behavior | Live display | Final display | 
|---|---|---|
| Reports final output usage | Approximate TPS | Exact token count with client-timed TPS | 
| Reports no output usage | Approximate TPS | Approximate TPS | 
| Buffers output instead of streaming deltas | No useful live TPS | Token count; TPS may be unavailable | 

For an OpenAI-compatible model, Pi requests streaming usage by default. Keep
`supportsUsageInStreaming` enabled only when the server accepts
`stream_options: { "include_usage": true }`:

```
{
  "providers": {
    "local-llm": {
      "baseUrl": "http://HOST:PORT/v1",
      "api": "openai-completions",
      "apiKey": "local",
      "compat": {
        "supportsUsageInStreaming": true
      },
      "models": [
        {
          "id": "MODEL_ID",
          "name": "Local model",
          "contextWindow": 128000,
          "maxTokens": 16384
        }
      ]
    }
  }
}
```

If the server rejects `stream_options`, set `supportsUsageInStreaming` to
`false`; the extension will continue using its explicit approximate fallback.

OpenAI-compatible servers commonly send usage once, in a final usage-only SSE
chunk. Pi exposes that count on `message_end`, not necessarily as multiple
`message_update` samples. Therefore a design that waits for two increasing
mid-stream `usage.output` samples often never obtains an exact rate.

This implementation instead records the first and last output arrival times, then combines that interval with the final reported output-token count. It works with the normal one-sample behavior used by vLLM, llama.cpp, and other OpenAI-compatible servers.

1. Run `/reload` and confirm no extension error appears.
2. Submit a prompt that generates at least 100 tokens.
3. Confirm the footer first shows `TTFT: waiting` .
4. Confirm live output contains `Decode: ~...` with tildes.
5. Confirm the final output loses the tildes when usage is reported.
6. Compare the final token count with the server response or request log.
7. Test a tool-calling turn; `toolcall_delta` is included in the live estimate.
8. Repeat with a second model. No filename, provider ID, or model ID is hard-coded in the extension.

- The live `characters / 4` estimate varies with prose, code, JSON, and CJK.
- Even with an exact final token count, client timing can be affected by stream buffering and network jitter.
- TTFT is end-to-end from Pi's provider hook, not GPU-only prefill latency.
- `Input/TTFT` is a useful comparable client metric, not true server prefill TPS.
- Very short outputs do not have enough first-to-last span for a stable rate.

Authoritative Pi references: