{"slug": "beyond-pass-k-measuring-reliability-and-security-of-agentic-code-generation", "title": "Beyond Pass@k: Measuring Reliability and Security of Agentic Code Generation", "summary": "A new arXiv paper (2608.14711v1) reveals that AI coding agent benchmarks misapply the pass@k estimator, inflating reported scores by 0.85-0.97 in absolute terms (0.96-0.98 reported vs. 0.00-0.12 corrected). The authors propose reliability@k and security-adjusted reliability@k metrics, and a preliminary 5-task SWE-bench Verified pilot found a macro-averaged hidden-test pass rate of 0.80 but strict task resolution of only 0.20.", "body_md": "arXiv:2608.14711v1 Announce Type: new\nAbstract: AI coding agent benchmarks rank agents with the Chen et al. (2021) pass@k estimator, but current implementations misapply it: they set n to the number of unit tests in a single submission rather than the number of independent rollout attempts, conflating test-suite size with attempt independence. We diagnose this operationalization error, prove it by counterexample, and propose reliability@k, the same estimator applied correctly, with n = independent rollouts and c = fully-passing rollouts per (task, agent) pair. In a synthetic multi-rollout benchmark, the misapplied metric inflates reported scores by 0.85-0.97 in absolute terms (0.96-0.98 reported vs. 0.00-0.12 corrected), and a cheap single-rollout proxy fails to substitute for repeated runs (Spearman $\\rho = 0.417$). Motivated by evidence that functional correctness does not imply security safety, we additionally propose security-adjusted reliability@k, which counts only rollouts that are both functionally correct and free of high-severity insecure patterns. In an initial live-API test with three agents, the adjustment did not change any ranking under our current scanner and threshold, so we present it as a proposed complementary lens whose decisive evaluation requires better-powered future runs. Finally, a preliminary 5-task SWE-bench Verified pilot observes the same core concern in a real repository setting: macro-averaged hidden-test pass rate was 0.80 while strict task resolution was 0.20.", "url": "https://wpnews.pro/news/beyond-pass-k-measuring-reliability-and-security-of-agentic-code-generation", "canonical_source": "https://www.machinebrief.com/news/beyond-passk-measuring-reliability-and-security-of-agentic-c-xy4g", "published_at": "2026-08-18 04:00:00+00:00", "updated_at": "2026-08-18 05:41:17.724368+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-safety"], "entities": ["arXiv", "Chen et al. (2021)", "SWE-bench Verified"], "alternates": {"html": "https://wpnews.pro/news/beyond-pass-k-measuring-reliability-and-security-of-agentic-code-generation", "markdown": "https://wpnews.pro/news/beyond-pass-k-measuring-reliability-and-security-of-agentic-code-generation.md", "text": "https://wpnews.pro/news/beyond-pass-k-measuring-reliability-and-security-of-agentic-code-generation.txt", "jsonld": "https://wpnews.pro/news/beyond-pass-k-measuring-reliability-and-security-of-agentic-code-generation.jsonld"}}