CANDI: Contextual Alignment for Niche Domains Question Answering

Researchers introduced CANDI-QA, a new dataset designed to evaluate large language models on context-sensitive question answering in specialized domains like medical diagnostics and financial advisory. Testing over ten models, they found that current LLMs struggle with contextual alignment, highlighting the need for enhanced symbolic integration. The study also presents MTSS-Net, a lightweight neuro-symbolic framework, as a baseline for future work.

arXiv:2607.11891v1 Announce Type: new Abstract: The deployment of large language models LLMs in specialized domains like medical diagnostics and financial advisory necessitates evaluating capabilities beyond general knowledge. Traditional question-answering benchmarks often fail to capture the nuanced contextual grounding, user awareness, and domain understanding these fields require. To address this, we introduce CANDI-QA Contextual Alignment for Niche Domains Question Answering , a novel dataset evaluating LLMs on delivering accurate, context-sensitive, and user-aligned answers in specialized settings. CANDI-QA features expert-curated question-answer pairs structured into two categories: 1 Information Assistance Questions, which are direct, factual queries requiring precise extraction, and 2 Applied Inference Questions, which are multi-hop reasoning tasks needing situational inference to generate actionable insights. We evaluate over ten diverse language models, from compact open-source to state-of-the-art proprietary systems. As a robust baseline, we present MTSS-Net, a lightweight neuro-symbolic framework combining neural retrieval with rule-based reasoning. Our findings highlight the profound challenges of achieving contextual alignment in niche domains, revealing the limitations of current LLMs without enhanced contextual or symbolic integration. Ultimately, CANDI-QA serves as a critical benchmark for advancing research in context-aware language models, stimulating the development of robust, trustworthy AI for high-stakes domains.