FDE-Gym: The Last Mile of AI Delivery

A new arXiv paper submitted 15 Oct 2026 introduces FDE-Gym, a benchmark of 95 end-to-end delivery tasks drawn from 20 customer scenarios across 18 sectors, built to test forward-deployed engineers' ability to clarify ambiguous requirements and uncover unstated constraints in live systems. Evaluating 13 model configurations from seven providers, the best configuration scored a mean FDE-Gym score of 0.56 and resolved only 14 of the 95 tasks, with discovering unstated constraints identified as the primary bottleneck. Even on the 23 tasks where the best configuration obtained every key fact from stakeholders, it reached a normalized business uplift of 0.8 or above on only 10, revealing a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers.

Computer Science Artificial Intelligence Submitted on 15 Oct 2026 Title:FDE-Gym: The Last Mile of AI Delivery View PDF http://arxiv.org/pdf/2610.22350v1 HTML experimental https://arxiv.org/html/2610.22350v1 Abstract:Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers FDEs must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions. FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 unchanged system and 1 reference solution , with penalties for stakeholder interactions messages, trial runs, and waiting time . An online platform lets humans practice FDE skills under the same grading criteria. We evaluate 13 model configurations from seven providers. In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks a score of at least 0.8 with no incident . Discovering unstated constraints is the primary bottleneck: performance drops sharply when models must uncover them without explicit guidance. Yet information discovery alone does not account for the gap: even on the 23 tasks where the best configuration obtains every key fact from stakeholders, it reaches a normalized business uplift of 0.8 or above on only 10. These results reveal a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers. Additional Features Current browse context: cs.AI References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .