A new analysis finds that long-horizon terminal-task benchmarks commonly measure task length by the number of turns a model needs to pass and difficulty by labels an LLM assigns during generation, and the authors report those proxies break down. The work proposes measuring and synthesizing depth in terminal tasks, addressing the scarcity of human-curated datasets used to evaluate and train agents. The finding matters for teams that build or select agent benchmarks, since turn counts and LLM-assigned difficulty labels may not capture how deep a task actually is.
Long-horizon terminal tasks are used to evaluate and train agents. Human-curated datasets are scarce, so many synthesis methods have emerged. They usually measure the length of a task by the turns a model needs to pass it, and its difficulty by labels the LLM sets during generation. We find, however
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.