CORTEX: Token-Level Hallucination Detection in RAG via Comparative Internal Representations

Researchers propose CORTEX, a token-level hallucination detection method for Retrieval-Augmented Generation (RAG) that identifies ungrounded content by comparing internal representations of a large language model with and without retrieved documents. The method improves fine-grained localization of hallucinations and reduces false positives by leveraging document-grounded information propagation and post-processing smoothing. Experiments on two RAG benchmarks and three LLMs show substantial performance gains.

arXiv:2606.31033v1 Announce Type: new Abstract: In this paper, we propose CORTEX, a token-level hallucination detection method for Retrieval-Augmented Generation RAG . In long-form RAG outputs, hallucinations often arise in localized spans rather than throughout an entire response. CORTEX therefore identifies ungrounded content at the token level, enabling fine-grained localization of hallucinations. The key intuition behind CORTEX is that tokens grounded in retrieved documents should be more strongly influenced by those documents than hallucinated tokens. To capture this document-induced effect, CORTEX compares internal representations of a large language model LLM under two conditions: with and without the retrieved documents. Instead of relying solely on each token's immediate sensitivity to the retrieved documents, CORTEX also leverages the propagation of document-grounded information through preceding tokens, reducing false positives for tokens whose evidence has already been absorbed into the context. Finally, CORTEX applies post-processing smoothing step that models the tendency of hallucination labels to persist over contiguous spans, reducing local noise and encouraging span-consistent predictions. Experiments on two RAG benchmarks and three LLMs show that CORTEX substantially improves token-level hallucination detection, with each component consistently contributing to performance gains.