Measuring Agent Reliability: pass@k, pass^k, and LLM Judges
An agent that passes 75 out of 100 test runs has a 75% pass@1 rate, but reliability depends on the product: pass@k (at least one success in k tries) yields 98.4% for k=3, while pass^k (all k succeed) yields only 42.2%, a…