Designing for Imperfect Prompts: A Simulation-Based Evaluation of an LLM-Based Enterprise Data Assistant Using Knowledge Graphs

A Design Science Research study of an LLM-based enterprise data assistant deployed in the CFO unit's data lake at an industry partner found that knowledge graph-based semantic mediation outperformed schema-based mediation via semantic similarity on Precision, Recall, F1, and preservation of user intent. The study, arXiv:2610.23304v1, used simulation-based evaluation with real-world data, supported by LLM-as-a-Judge assessments and human validation, after observing that business users frequently submit imperfect prompts containing domain-specific vocabulary and lacking schema references or structural information. The authors propose a design principle for knowledge graph-based mediation, arguing that structured knowledge representation bridges user intent and database structure and can guide enterprise data assistant design across skill levels.

arXiv:2610.23304v1 Announce Type: cross Abstract: Enterprise data assistants using large language models LLMs promise democratized data access in organizations. Yet in our real-world deployment within the CFO unit's data lake at an industry partner, we observed that business users frequently pose imperfect prompts that include domain-specific vocabulary and lack schema references or structural information. This Design Science Research study examines automatic resolution using a semantic mediation layer. We design and evaluate two design variants: schema-based mediation via semantic similarity, and knowledge graph-based mediation leveraging an automatically constructed knowledge graph from the data lake. Our simulation-based evaluation using real-world data, supported by LLM-as-a-Judge assessments and human validation, shows that knowledge graph-based mediation outperforms the schema-based approach in Precision, Recall, F1, and preserving user intent. We propose a design principle for knowledge graph-based mediation, demonstrating that structured knowledge representation bridges user intent and database structure and guides the design of enterprise data assistants across skill levels.