Model-neutral Pi extension for live decode TPS, TTFT, and prompt-rate estimates

A developer published a model-neutral Pi extension that displays live throughput metrics—time to first token (TTFT), decode tokens per second, and a client-side prompt-rate estimate—in the coding agent's status footer. The extension, validated against @earendil-works/pi-coding-agent 0.85.1, works with both local and hosted models by observing Pi's standard assistant-stream events rather than reading vLLM logs or metrics. It estimates live decode TPS using a characters-divided-by-four heuristic and notes that its input/TTFT figure is not authoritative server prefill throughput.

This guide adds a model-neutral throughput indicator to Pi https://github.com/earendil-works/pi . It was validated against @earendil-works/pi-coding-agent 0.85.1. The extension works with local and hosted models because it observes Pi's standard assistant-stream events. It does not read vLLM logs or metrics and does not depend on a particular model name. During generation, the footer looks similar to: ⚡ TTFT: 1.24s · Decode: ~42.1 tok/s · ~312 tok After generation, when the provider reports token usage, it becomes: ⚡ TTFT: 1.24s · Input/TTFT: ~1850 tok/s · Decode: 39.8 tok/s · 842 tok - TTFT is measured from Pi's before provider request event to the first output delta observed by the client. - Live decode TPS is a characters / 4 estimate because most providers do not report a cumulative token count on every stream chunk. - Final decode TPS uses the provider's reported output-token count and the client-observed interval from the first output delta to the last. The first token is excluded from the numerator because it defines the start boundary. - Input/TTFT is only a client-side prompt-rate estimate. It divides uncached input tokens by TTFT, which also contains network, queue, scheduling, and stream-start overhead. It is not authoritative server prefill throughput. For true per-request prefill throughput, use metrics or timing emitted by the inference server. Pi's message events do not expose a server-only prefill duration. Create ~/.pi/agent/extensions/live-throughput-status.ts with this content: python import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent"; const STATUS KEY = "live-throughput"; const CHARS PER TOKEN = 4; const UPDATE INTERVAL MS = 200; function setStatus ctx: ExtensionContext, text?: string : void { if ctx.mode == "tui" return; ctx.ui.setStatus STATUS KEY, text === undefined ? undefined : ctx.ui.theme.fg "accent", ⚡ ${text} , ; } function positiveNumber value: unknown : number | undefined { return typeof value === "number" && Number.isFinite value && value 0 ? value : undefined; } function deltaChars event: unknown : number { if event || typeof event == "object" return 0; const streamEvent = event as { type?: string; delta?: unknown }; if streamEvent.type == "text delta" && streamEvent.type == "thinking delta" && streamEvent.type == "toolcall delta" { return 0; } return typeof streamEvent.delta === "string" ? streamEvent.delta.length : 0; } function seconds milliseconds: number : number { return Math.max 0, milliseconds / 1000; } function rate value: number, durationSeconds: number : string { return value / Math.max 0.001, durationSeconds .toFixed 1 ; } export default function pi: ExtensionAPI { let requestStartedAt: number | undefined; let firstOutputAt: number | undefined; let lastOutputAt: number | undefined; let ttftSeconds: number | undefined; let streamedChars = 0; let lastDisplayAt = 0; function reset ctx: ExtensionContext, waiting = true : void { requestStartedAt = undefined; firstOutputAt = undefined; lastOutputAt = undefined; ttftSeconds = undefined; streamedChars = 0; lastDisplayAt = 0; setStatus ctx, waiting ? "TTFT: waiting · Decode: waiting" : undefined ; } pi.on "session start", async event, ctx = reset ctx ; pi.on "model select", async event, ctx = reset ctx ; // This event occurs immediately before Pi sends a provider payload. The // handler observes time only and deliberately returns no payload rewrite. pi.on "before provider request", async event, ctx = { requestStartedAt = Date.now ; } ; pi.on "message start", async event, ctx = { if event.message.role == "assistant" return; firstOutputAt = undefined; lastOutputAt = undefined; ttftSeconds = undefined; streamedChars = 0; lastDisplayAt = 0; // Fallback for a custom provider that does not emit the request hook. requestStartedAt ??= Date.now ; setStatus ctx, "TTFT: waiting · Decode: waiting for first token…" ; } ; pi.on "message update", async event, ctx = { if event.message.role == "assistant" return; const chars = deltaChars event.assistantMessageEvent ; if chars <= 0 return; const now = Date.now ; if firstOutputAt === undefined { firstOutputAt = now; ttftSeconds = seconds now - requestStartedAt ?? now ; } lastOutputAt = now; streamedChars += chars; if now - lastDisplayAt < UPDATE INTERVAL MS return; lastDisplayAt = now; const decodeSeconds = seconds now - firstOutputAt ; const estimatedTokens = streamedChars / CHARS PER TOKEN; setStatus ctx, TTFT: ${ttftSeconds .toFixed 2 }s · Decode: ~${rate estimatedTokens, decodeSeconds } tok/s · ~${Math.round estimatedTokens } tok , ; } ; pi.on "message end", async event, ctx = { if event.message.role == "assistant" return; const usage = event.message.usage; const outputTokens = positiveNumber usage?.output ; const uncachedInputTokens = positiveNumber usage?.input ; const cacheWriteTokens = positiveNumber usage?.cacheWrite ?? 0; const observedTtft = ttftSeconds; const decodeSeconds = firstOutputAt == undefined && lastOutputAt == undefined ? seconds lastOutputAt - firstOutputAt : undefined; const parts: string = ; if observedTtft == undefined { parts.push TTFT: ${observedTtft.toFixed 2 }s ; } // Pi normalizes usage.input to uncached input for supported providers. // Cache reads are excluded because they were not prefetched by the model. const processedInputTokens = uncachedInputTokens ?? 0 + cacheWriteTokens; if processedInputTokens 0 && observedTtft == undefined && observedTtft 0 { parts.push Input/TTFT: ~${rate processedInputTokens, observedTtft } tok/s ; } if outputTokens == undefined && outputTokens 1 && decodeSeconds == undefined && decodeSeconds 0 { parts.push Decode: ${rate outputTokens - 1, decodeSeconds } tok/s · ${outputTokens} tok ; } else if outputTokens == undefined { parts.push Decode: ${outputTokens} tok · rate unavailable ; } else if streamedChars 0 && decodeSeconds == undefined && decodeSeconds 0 { const estimatedTokens = streamedChars / CHARS PER TOKEN; parts.push Decode: ~${rate estimatedTokens, decodeSeconds } tok/s · ~${Math.round estimatedTokens } tok ; } else { parts.push "Decode: no output tokens" ; } setStatus ctx, parts.join " · " ; requestStartedAt = undefined; } ; } Run /reload in Pi. The global extension directory is auto-discovered, so no additional settings entry is required. The extension itself is model-neutral. Accuracy depends on the provider: | Provider behavior | Live display | Final display | |---|---|---| | Reports final output usage | Approximate TPS | Exact token count with client-timed TPS | | Reports no output usage | Approximate TPS | Approximate TPS | | Buffers output instead of streaming deltas | No useful live TPS | Token count; TPS may be unavailable | For an OpenAI-compatible model, Pi requests streaming usage by default. Keep supportsUsageInStreaming enabled only when the server accepts stream options: { "include usage": true } : { "providers": { "local-llm": { "baseUrl": "http://HOST:PORT/v1", "api": "openai-completions", "apiKey": "local", "compat": { "supportsUsageInStreaming": true }, "models": { "id": "MODEL ID", "name": "Local model", "contextWindow": 128000, "maxTokens": 16384 } } } } If the server rejects stream options , set supportsUsageInStreaming to false ; the extension will continue using its explicit approximate fallback. OpenAI-compatible servers commonly send usage once, in a final usage-only SSE chunk. Pi exposes that count on message end , not necessarily as multiple message update samples. Therefore a design that waits for two increasing mid-stream usage.output samples often never obtains an exact rate. This implementation instead records the first and last output arrival times, then combines that interval with the final reported output-token count. It works with the normal one-sample behavior used by vLLM, llama.cpp, and other OpenAI-compatible servers. 1. Run /reload and confirm no extension error appears. 2. Submit a prompt that generates at least 100 tokens. 3. Confirm the footer first shows TTFT: waiting . 4. Confirm live output contains Decode: ~... with tildes. 5. Confirm the final output loses the tildes when usage is reported. 6. Compare the final token count with the server response or request log. 7. Test a tool-calling turn; toolcall delta is included in the live estimate. 8. Repeat with a second model. No filename, provider ID, or model ID is hard-coded in the extension. - The live characters / 4 estimate varies with prose, code, JSON, and CJK. - Even with an exact final token count, client timing can be affected by stream buffering and network jitter. - TTFT is end-to-end from Pi's provider hook, not GPU-only prefill latency. - Input/TTFT is a useful comparable client metric, not true server prefill TPS. - Very short outputs do not have enough first-to-last span for a stable rate. Authoritative Pi references: