Improving tail latency in parallel LLM calls

HOAi reduced tail latency in its parallel LLM-based document search by racing two GPT-5.4-mini default-tier requests per batch instead of a single GPT-4.1-mini priority request, cutting p95 search times from 18.12 seconds to an unmeasured lower value at roughly twice the cost. The company's production traces showed one straggler batch of 79 seconds in a 113-second search, and the dual-race approach aborts the slower request to eliminate that bottleneck.

Our AI agents at HOAi search a community association’s documents. Postgres retrieves candidate pages; model calls filter them for relevance and summarize the ones that pass. In our previous post, we fixed the Postgres retrieval plan /posts/the-postgres-index-our-query-never-used . Searches still occasionally took 30 seconds or more. The remaining problem came from a design choice meant to make them fast: we ran the model calls in parallel. The slowest batch sets the latency We split candidate pages into batches of five and run every batch concurrently. The phase finishes only when its slowest batch does. In one 113-second trace, most of the 21 batches finished in two to seven seconds. One took 79 seconds. That request set the reranking latency. The same pattern appears across production searches on GPT-4.1-mini priority: | Percentile | Search time | Candidate pages | |---|---|---| | p50 | 4.80 s | 19 | | p95 | 18.12 s | 79 | | p99 | 24.98 s | 69 | | max | 37.07 s | 72 | At p95, 79 candidates create about 16 chances for one provider stall. Race each batch We moved reranking from one GPT-4.1-mini priority request to two GPT-5.4-mini default-tier requests. Each batch launches both, takes the first successful result, and aborts the loser. It’s sending every request twice /posts/a-simple-fix-for-llm-tail-latency , applied per batch. We also moved summarization from GPT-4.1-mini priority to GPT-5.4-mini priority. The latency table above is the old baseline; we do not yet have enough post-rollout data to publish a new p95. What it costs We priced the same production token mix under the old and new configurations. Each column summarizes the costs of individually priced calls or searches: | Summarization configuration | Mean/call | Median/call | p95/call | |---|---|---|---| | GPT-4.1-mini priority, same-token counterfactual | $0.0156 | $0.0097 | $0.0525 | | GPT-5.4-mini priority, observed path | $0.0339 | $0.0213 | $0.1127 | | Reranking configuration | Mean/search | Median/search | p95/search | |---|---|---|---| | GPT-4.1-mini priority, single request | $0.0293 | $0.0211 | $0.0795 | | GPT-5.4-mini default, dual-race lower bound | $0.0603 | $0.0430 | $0.1626 | | GPT-5.4-mini default, dual-race upper bound | $0.0674 | $0.0495 | $0.1856 | The reranking bounds differ in how much output we charge to the aborted request. Cached-input discounts and partial usage from aborted requests may change the realized cost. The takeaway When a search waits for many parallel LLM calls, one straggler sets the latency. Racing each batch cuts that tail for roughly twice the cost.