ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model

IBM Research's ALTK-Evolve reached 67.5% Pass^4 on the τ²-bench retail benchmark using GPT-4.1, topping the leaderboard without a frontier model or any weight updates. The method automatically distills a ReAct agent's own failed conversations into system-prompt behavioral rules, letting teams improve policy-bound customer service agents through offline prompt evolution instead of fine-tuning or model swaps. The approach shifts the optimization bottleneck from model capability to system-prompt token length, according to the Hugging Face post.

Hugging Face https://huggingface.co/blog/ibm-research/altk-evolve-bc ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated. ALTK-Evolve achieves 67.5% Pass^4 solves tasks correctly four times consecutively on τ²-bench retail using GPT-4.1 by distilling past attempts into behavioral rules, proving lightweight, in-context learning can match frontier-model performance for policy-bound tasks. This lets teams upgrade existing agents without costly fine-tuning or model swaps by dynamically generating and applying domain-specific guidelines—cutting deployment time and compute costs while maintaining strict compliance. By automatically distilling its own conversation failures into system-prompt behavioral rules, a basic ReAct agent using the older GPT-4.1 model achieved 67.5% Pass^4 on the τ²-bench retail benchmark, outperforming all frontier entries without any weight updates. This proves that policy-bound customer service agents can be optimized for reliability and strict guardrails entirely through automated offline prompt evolution rather than costly fine-tuning or upgrading to reasoning models. For teams shipping agents, this enables a self-improving pipeline where models dynamically write their own domain playbooks, shifting the optimization bottleneck from model capability to system-prompt token length.