Fluent but Fallible: Benchmarking Hallucination and Factual Reliability in Large Language Models

A benchmark of ten large language models on 70 academic prompts found Perplexity and DeepSeek V3 achieved the highest overall accuracy, while Falcon reached perfect precision with zero hallucinations but had the lowest recall and highest response deviation, according to arXiv paper 2610.27034v1. Three reviewers scored responses from ChatGPT, Gemini, Claude 3.5 Sonnet, LLaMA 3, Mistral, Grok, Perplexity, DeepSeek V3, Falcon and Gemma in February 2025 across five metrics — Accuracy, Precision, Recall, Response Deviation Index and Hallucination Index — on prompts at four complexity levels. Advanced-level prompts proved most challenging for all models, while binary true/false Tricky prompts were handled more reliably, and the authors report a fundamental trade-off between response completeness and factual reliability that intensifies with prompt complexity.

arXiv:2610.27034v1 Announce Type: new Abstract: Large Language Models LLMs have significantly advanced natural language processing and artificial intelligence applications through their capabilities in reasoning, text generation and knowledge retrieval. However, LLMs are still vulnerable to two important challenges: hallucination, generating factually incorrect or fabricated content, and response deviation, meaning systematic deviations from ground-truth responses. These limitations raise serious concerns regarding reliability and trustworthiness in academic and research environments. This study offers a thorough comparative analysis using a variety of metrics of ten advanced LLMs ChatGPT, Gemini, Claude~3.5~Sonnet, LLaMA~3, Mistral, Grok, Perplexity, DeepSeek~V3, Falcon, and Gemma based on a benchmark consisting of 70 carefully designed academic prompts across four complexity levels: Basic, Explanation, Advanced, and Tricky, evaluated in February 2025 on specific model versions. Each response was independently evaluated by three reviewers using five metrics: Accuracy, Precision, Recall, Response Deviation Index, and Hallucination Index. Perplexity and DeepSeek~V3 achieved the highest overall accuracy, while Gemini demonstrated strong recall and a well-balanced performance profile across all five metrics. LLaMA~3 showed strong resistance against hallucinations through a conservative method of generation. Advanced-level prompts proved most challenging across all models, while Tricky prompts were handled more reliably owing to their binary true/false structure. Falcon achieved perfect precision and zero hallucinations, but at the cost of the lowest recall and the highest response deviation. The results show a fundamental trade-off between response completeness and factual reliability, intensified by prompt complexity.