$\tau^\tau$-Bench: An Environment for End-To-End, Realistic Agent Construction

Researchers introduced ττ-bench, a benchmark that tests AI systems' ability to construct customer-service agents from realistic business records, client requirements, and production APIs. Across 53 tasks in four domains, the strongest configuration, Claude Opus 5 under Claude Code, passed only 23.9% of evaluation simulations, while an expert-authored reference ceiling scored 82.2%. The benchmark aims to make cooperative agent building a measurable target for coding agents.

arXiv:2609.04611v1 Announce Type: new Abstract: LLM agents are rapidly becoming production software, deployed to handle customer service, adjudicate disputes, and operate internal systems. Notably, the work of building them is increasingly handed to coding agents, yet existing benchmarks say little about whether an AI system can deliver one under the conditions of a real client engagement. We introduce $\tau^\tau$-bench pronounced hyper-tau-bench , a benchmark that makes agent construction the task. A developer agent is given the records a business actually keeps, a client who holds requirements, a production API that operations must run through, a codebase to inherit, and limits on serving cost and models: the same starting point a real engagement provides. From these it must deliver a complete customer-service agent, scored by deploying that agent against held-out simulated users. Across 53 tasks spanning four domains, the strongest configuration, Claude Opus 5 under Claude Code, passes just 23.9% of evaluation simulations. Meanwhile, an expert-authored reference ceiling scores 82.2%. The failures mirror ones human agent developers see: models issue shallow queries in place of deep comprehension of the records, communicate almost nothing to the client, and experiment too little with agent architecture and serving spend, shipping the first design that runs. We aim for $\tau^\tau$-bench to turn the work of cooperative agent building into a measurable target for coding agents.