{"slug": "cypherturn-a-multi-turn-benchmark-for-conversational-text-to-cypher-evaluation", "title": "CypherTurn: A Multi-Turn Benchmark for Conversational Text-to-Cypher Evaluation and the Autonomy Divergence", "summary": "A new benchmark called CypherTurn, comprising 721 sessions and 5,927 turns across 7 knowledge graphs and 13 conversational phenomena, shows the best of 15 evaluated models reaches only 64.7% execution accuracy on conversational Text-to-Cypher tasks, with session-level correctness below 5%. The benchmark's authors report that frontier models reorder at the top of the leaderboard under a fully autonomous agentic protocol versus a guided oracle protocol — a phenomenon they term the Autonomy Divergence — and that scaling action budgets from x3 to x10 fails to close the autonomy gap because the strongest frontier models self-limit to approximately two actions per turn. Single-turn Cypher fine-tuning degrades multi-turn instruction following, while architecture-appropriate specialization outperforms several frontier models, according to the paper.", "body_md": "arXiv:2609.36987v1 Announce Type: new \nAbstract: Graph databases are increasingly queried through natural language, yet every existing benchmark evaluates isolated single-turn queries rather than the multi-turn sessions through which analysts actually work. We introduce CypherTurn, the first benchmark for conversational Text-to-Cypher evaluation, comprising 721 sessions and 5,927 turns across 7 knowledge graphs and 13 conversational phenomena. We evaluate 15 models under a guided oracle protocol and a fully autonomous agentic protocol, yielding four findings. First, the best model reaches only 64.7% execution accuracy, and session-level correctness remains below 5%. Second, despite strong overall rank correlation, frontier models exhibit a consequential reordering of the top of the leaderboard under autonomous operation, a phenomenon we term the Autonomy Divergence, which reveals error-management as a partially independent capability from raw generation skill. Third, scaling action budgets from x3 to x10 fails to close the autonomy gap, as the strongest frontier models self-limit to approximately two actions per turn regardless of available budget. Fourth, single-turn Cypher fine-tuning degrades multi-turn instruction following, while architecture-appropriate specialization outperforms several frontier models. These results establish CypherTurn as an open challenge for conversational graph database reasoning. Code and data are available at https://github.com/BarryQ/CypherTurn.", "url": "https://wpnews.pro/news/cypherturn-a-multi-turn-benchmark-for-conversational-text-to-cypher-evaluation", "canonical_source": "https://www.machinebrief.com/news/cypherturn-a-multi-turn-benchmark-for-conversational-text-to-izej", "published_at": "2026-09-30 04:00:00+00:00", "updated_at": "2026-09-30 04:47:03.076237+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-agents", "natural-language-processing", "ai-safety"], "entities": ["CypherTurn", "Cypher", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/cypherturn-a-multi-turn-benchmark-for-conversational-text-to-cypher-evaluation", "markdown": "https://wpnews.pro/news/cypherturn-a-multi-turn-benchmark-for-conversational-text-to-cypher-evaluation.md", "text": "https://wpnews.pro/news/cypherturn-a-multi-turn-benchmark-for-conversational-text-to-cypher-evaluation.txt", "jsonld": "https://wpnews.pro/news/cypherturn-a-multi-turn-benchmark-for-conversational-text-to-cypher-evaluation.jsonld"}}