{"slug": "designing-for-imperfect-prompts-a-simulation-based-evaluation-of-an-llm-based", "title": "Designing for Imperfect Prompts: A Simulation-Based Evaluation of an LLM-Based Enterprise Data Assistant Using Knowledge Graphs", "summary": "A Design Science Research study of an LLM-based enterprise data assistant deployed in the CFO unit's data lake at an industry partner found that knowledge graph-based semantic mediation outperformed schema-based mediation via semantic similarity on Precision, Recall, F1, and preservation of user intent. The study, arXiv:2610.23304v1, used simulation-based evaluation with real-world data, supported by LLM-as-a-Judge assessments and human validation, after observing that business users frequently submit imperfect prompts containing domain-specific vocabulary and lacking schema references or structural information. The authors propose a design principle for knowledge graph-based mediation, arguing that structured knowledge representation bridges user intent and database structure and can guide enterprise data assistant design across skill levels.", "body_md": "arXiv:2610.23304v1 Announce Type: cross \nAbstract: Enterprise data assistants using large language models (LLMs) promise democratized data access in organizations. Yet in our real-world deployment within the CFO unit's data lake at an industry partner, we observed that business users frequently pose imperfect prompts that include domain-specific vocabulary and lack schema references or structural information. This Design Science Research study examines automatic resolution using a semantic mediation layer. We design and evaluate two design variants: schema-based mediation via semantic similarity, and knowledge graph-based mediation leveraging an automatically constructed knowledge graph from the data lake. Our simulation-based evaluation using real-world data, supported by LLM-as-a-Judge assessments and human validation, shows that knowledge graph-based mediation outperforms the schema-based approach in Precision, Recall, F1, and preserving user intent. We propose a design principle for knowledge graph-based mediation, demonstrating that structured knowledge representation bridges user intent and database structure and guides the design of enterprise data assistants across skill levels.", "url": "https://wpnews.pro/news/designing-for-imperfect-prompts-a-simulation-based-evaluation-of-an-llm-based", "canonical_source": "https://www.machinebrief.com/news/designing-for-imperfect-prompts-a-simulation-based-evaluatio-9hlk", "published_at": "2026-10-19 04:00:00+00:00", "updated_at": "2026-10-19 08:18:34.070288+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-agents"], "entities": ["arXiv", "CFO unit", "LLM-as-a-Judge"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/designing-for-imperfect-prompts-a-simulation-based-evaluation-of-an-llm-based", "markdown": "https://wpnews.pro/news/designing-for-imperfect-prompts-a-simulation-based-evaluation-of-an-llm-based.md", "text": "https://wpnews.pro/news/designing-for-imperfect-prompts-a-simulation-based-evaluation-of-an-llm-based.txt", "jsonld": "https://wpnews.pro/news/designing-for-imperfect-prompts-a-simulation-based-evaluation-of-an-llm-based.jsonld"}}