ProgramBench

A study by the coding agent company Droid found that single-agent runs stop early on long-horizon software tasks because they validate work locally without a complete standard of completion. In a test…

ProgramBench, a new reverse-engineering benchmark, compiles and fuzzes LLM-generated code against original binaries, with GPT-4o achieving a 34% pass@1 rate, Claude 3.5 Sonnet 28%, and DeepSeek-Coder-…

Coding agents Claude Code and Codex consistently over-predict their own wall-clock runtime, according to research conducted as part of MATS 10 with Maksym Andriushchenko. The study introduced AgentTim…