{"slug": "do-we-need-an-llm-to-know-why-sources-conflict-decision-models-for-knowledge-in", "title": "Do We Need an LLM to Know Why Sources Conflict? Decision Models for Knowledge-Conflict Typing in RAG", "summary": "Typed decision models served by a single commercial inference gateway classified knowledge conflicts in RAG at 50.0–67.0% accuracy, and 10 of the 11 matched or beat Gemini 2.5 Flash's 61.3% at 4–39x lower billed cost and 2.6–6.0x lower latency, according to an arXiv paper (2610.36915v1) evaluated on the CONFLICTS benchmark. The study, which used the original paper's prompt verbatim across 11 decision models from 7 providers, found Gemini 3.8 Flash reached 70.0% accuracy, but routing each decision model's least confident quarter of queries to it produced a cascade statistically tied with Gemini 3.8 Flash for 8 of the 11 models at 25–37% of its cost. The authors report that decomposing the taxonomy into yes/no questions lowered accuracy for every model, a repair fitted on one model did not transfer, and dates played no measurable role, with code, prompts and all model outputs released on GitHub.", "body_md": "arXiv:2610.36915v1 Announce Type: new \nAbstract: When the sources retrieved for a query disagree, the right response depends on why they disagree: a question with several valid answers, a contested topic, stale information and a false source each call for different behaviour. Cattan et al. (2025) showed that telling a generator the conflict type improves its responses by about 24 points, but that predicting the type is hard even for frontier LLMs. We study whether typed decision models, which return probabilities over fixed options in a single forward pass instead of generating text, can do this classification fast and cheaply. Using the CONFLICTS benchmark and the original paper's prompt verbatim, we evaluate all 11 decision models served by one commercial inference gateway, from 7 providers, against Gemini models on identical inputs. Asked directly, the decision models reach 50.0--67.0% accuracy; 10 of the 11 match or exceed Gemini 2.5 Flash, the paper's model (61.3%), at 4--39x lower billed cost (2 were free) and 2.6--6.0x lower latency. The current Gemini 3.8 Flash (70.0%) is significantly more accurate than 8 of them, but sending each decision model's least confident quarter of queries to it gives a cascade statistically tied with it for 8 of the 11, at 25--37% of its cost. End to end, answers generated with the types of 10 of the 11 follow the expected behaviour as often as with the LLM's types. Decomposing the taxonomy into yes/no questions lowers accuracy for every model, and a repair fitted on one model does not transfer. Pre-specified diagnostics show that dates play no measurable role and that sensitivity to source order varies widely between models. Decision models detect a planted false source less reliably than the LLM as hard labels, but their probabilities still separate planted from clean sources. We release code, prompts and all model outputs at https://github.com/Parv17k/decision-models-conflict-typing.", "url": "https://wpnews.pro/news/do-we-need-an-llm-to-know-why-sources-conflict-decision-models-for-knowledge-in", "canonical_source": "https://www.machinebrief.com/news/do-we-need-an-llm-to-know-why-sources-conflict-decision-mode-6zzt", "published_at": "2026-10-28 04:00:00+00:00", "updated_at": "2026-10-28 04:41:51.817608+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-tools"], "entities": ["Cattan et al.", "Gemini 2.5 Flash", "Gemini 3.8 Flash", "CONFLICTS benchmark", "arXiv", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/do-we-need-an-llm-to-know-why-sources-conflict-decision-models-for-knowledge-in", "markdown": "https://wpnews.pro/news/do-we-need-an-llm-to-know-why-sources-conflict-decision-models-for-knowledge-in.md", "text": "https://wpnews.pro/news/do-we-need-an-llm-to-know-why-sources-conflict-decision-models-for-knowledge-in.txt", "jsonld": "https://wpnews.pro/news/do-we-need-an-llm-to-know-why-sources-conflict-decision-models-for-knowledge-in.jsonld"}}