{"slug": "testing-the-planning-claims-of-llm-based-agents-against-formal-semantics", "title": "Testing the Planning Claims of LLM-Based Agents against Formal Semantics", "summary": "A study posted to arXiv (2610.22469v1) found that the best LLM agent configuration solved 78% of solvable planning tasks offline but only 40% online, and recognized unsolvable tasks in under 10% of episodes. The authors grounded natural-language tasks in hidden PDDL models across five domains, four models, six prompting strategies and three agent architectures, and found heavier prompting strategies consumed 5 to 22 times as many tokens as direct prompting with no clear gain. Planning performance varied more across models than across prompting strategies or agent architectures, leading the authors to conclude that claims agents can plan require evidence from validated action traces.", "body_md": "arXiv:2610.22469v1 Announce Type: new \nAbstract: LLM-based agents are increasingly credited with the competences of practical reasoning: planning, monitoring, and correcting their own actions. Their success is attributed to the prompting strategies and agent architectures built around the model, such as chain-of-thought, search-based prompting, self-refinement, reflection, critics and tool-using loops. Yet these claims are usually supported by end-to-end success rates, which show only whether a task was completed, or by LLM judges, which are not verifiers. We test LLM planning against formal semantics. Tasks are posed in natural language, but each is grounded in a hidden PDDL model that decides whether every action is applicable, whether the goal is reached, and whether a plan exists at all. Across five domains (two of them new, all released with the paper), four models, six prompting strategies and three agent architectures, the best configuration solves 78% of the solvable tasks offline but only 40% online, where agents mostly act without inspecting the state. Unsolvable tasks are recognised in under 10% of episodes, and heavier prompting strategies use 5 to 22 times as many tokens as direct prompting with no clear gain. In our experiments, planning performance varied more across models than across prompting strategies or agent architectures. Claims that agents can plan require evidence from validated action traces.", "url": "https://wpnews.pro/news/testing-the-planning-claims-of-llm-based-agents-against-formal-semantics", "canonical_source": "https://arxiv.org/abs/2610.22469", "published_at": "2026-10-19 04:00:00+00:00", "updated_at": "2026-10-19 04:18:31.517631+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "large-language-models", "ai-research"], "entities": ["arXiv", "PDDL"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/testing-the-planning-claims-of-llm-based-agents-against-formal-semantics", "markdown": "https://wpnews.pro/news/testing-the-planning-claims-of-llm-based-agents-against-formal-semantics.md", "text": "https://wpnews.pro/news/testing-the-planning-claims-of-llm-based-agents-against-formal-semantics.txt", "jsonld": "https://wpnews.pro/news/testing-the-planning-claims-of-llm-based-agents-against-formal-semantics.jsonld"}}