{"slug": "bap-v2-an-enhanced-task-framework-for-instruction-following-in-minecraft", "title": "BAP v2: An Enhanced Task Framework for Instruction Following in Minecraft Dialogues", "summary": "Researchers led by Prashant Jayannavar published BAP v2, an enhanced benchmark for the Builder Action Prediction subtask of the Minecraft Collaborative Building Task, in Computational Linguistics Volume 52, Issue 3 (September 2026, pages 943–1000). The team introduced a new state-of-the-art model, Llama-CRAFTS, which reaches an F1 score of 53.0 on BAP v2 — a 6-point improvement over prior work — and showed that training LLM-based models on generated synthetic MCBT data improves performance across the board, while current SOTA models trained on human BAP dialogues fail on the simpler synthetic tasks. The authors identify spatial reasoning as the primary performance bottleneck and position BAP v2 as a measure of the spatial capabilities of text-only LLMs in embodied tasks.", "body_md": "##### Abstract\n\nDeveloping interactive agents that can understand language, perceive their surroundings, and act within the physical world is a long-standing goal of AI research. The Minecraft Collaborative Building Task (MCBT) (Narayan-Chen, Jayannavar, and Hockenmaier, 2019), a two-player game in which an Architect (A) instructs a Builder (B) to construct a target structure in a simulated 3D Blocks World environment, offers a rich platform to work towards this goal. In this work, we focus on the Builder Action Prediction (BAP) subtask: predicting B’s actions in a multimodal game context (Jayannavar, Narayan-Chen, and Hockenmaier, 2020)—a challenging testbed for grounded instruction following, with limited training data. We holistically re-examine this task and introduce BAP v2 to address key challenges in evaluation, training data, and modeling. Specifically, we define an enhanced evaluation benchmark, featuring a cleaner test set and fairer, more insightful metrics that also reveal spatial reasoning as the primary performance bottleneck. To address data scarcity and to teach models basic spatial skills, we generate different types of synthetic MCBT data. We observe that current, LLM-based SOTA models trained on the human BAP dialogues fail on these simpler, synthetic BAP ones, but show that training models on this synthetic data improves their performance across the board. We also introduce a new SOTA model, Llama-CRAFTS, which leverages richer input representations, and achieves an F1 score of 53.0 on the BAP v2 task and strong performance on the synthetic data. While this result marks a notable 6 points improvement over previous work, it also underscores the task’s remaining difficulty, establishing BAP v2 as a fertile ground for future research, and providing a useful measure of the spatial capabilities of current text-only LLMs in such embodied tasks.\n- Anthology ID:\n- 2026.cl-3.3\n- Volume:\n- [Computational Linguistics, Volume 52, Issue 3 - September 2026](https://aclanthology.org/volumes/2026.cl-3/)\n- Month:\n- September\n- Year:\n- 2026\n- Address:\n- Cambridge, MA\n- Venue:\n- [CL](https://aclanthology.org/venues/cl/)\n- SIG:\n- Publisher:\n- MIT Press\n- Note:\n- Pages:\n- 943–1000\n- Language:\n- URL:\n- [https://aclanthology.org/2026.cl-3.3/](https://aclanthology.org/2026.cl-3.3/)\n- DOI:\n- [10.1162/coli.a.602](https://doi.org/10.1162/coli.a.602)\n- Cite (ACL):\n- Prashant Jayannavar, Liliang Ren, Marisa Hudspeth, Risham Sidhu, Charlotte Lambert, Ariel Cordes, Elizabeth Kaplan, Anjali Narayan-Chen, and Julia Hockenmaier. 2026. [BAP v2: An Enhanced Task Framework for Instruction Following in Minecraft Dialogues](https://aclanthology.org/2026.cl-3.3/) .*Computational Linguistics* , 52(3):943–1000.\n- Cite (Informal):\n- [BAP v2: An Enhanced Task Framework for Instruction Following in Minecraft Dialogues](https://aclanthology.org/2026.cl-3.3/) (Jayannavar et al., CL 2026)\n- PDF:\n- [https://aclanthology.org/2026.cl-3.3.pdf](https://aclanthology.org/2026.cl-3.3.pdf)", "url": "https://wpnews.pro/news/bap-v2-an-enhanced-task-framework-for-instruction-following-in-minecraft", "canonical_source": "https://aclanthology.org/2026.cl-3.3/", "published_at": "2026-10-26 00:00:00+00:00", "updated_at": "2026-10-26 23:12:46.904404+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "natural-language-processing", "ai-agents"], "entities": ["Prashant Jayannavar", "Llama-CRAFTS", "Minecraft Collaborative Building Task", "Builder Action Prediction", "Computational Linguistics", "MIT Press", "Julia Hockenmaier", "Anjali Narayan-Chen"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/bap-v2-an-enhanced-task-framework-for-instruction-following-in-minecraft", "markdown": "https://wpnews.pro/news/bap-v2-an-enhanced-task-framework-for-instruction-following-in-minecraft.md", "text": "https://wpnews.pro/news/bap-v2-an-enhanced-task-framework-for-instruction-following-in-minecraft.txt", "jsonld": "https://wpnews.pro/news/bap-v2-an-enhanced-task-framework-for-instruction-following-in-minecraft.jsonld"}}