A simple fix for LLM tail latency
Sending every LLM request twice and taking the faster response outperforms paid priority tiers, cutting worst-case time to complete response from 9.8s to 3.5s and worst-case time to first token from 4.2s to 1.2s in tests…