VeriQuote · open-source library and CLI
Language models put citations like [1] behind their claims, and checking one means opening the source
and finding the passage. A recent study on clinical guideline questions found that a frontier model
quoted its sources word for word for 98% of its claims, yet only 37% of the claims were fully supported
by those quotes.
VeriQuote makes the model attach a verbatim quote to every citation and checks each quote twice. A deterministic matcher looks for the quote in the source. An LLM judge reads the quote next to the claim and decides whether it supports it. A citation passes only if both let it through.
Pick an example. The matcher compares the quote with the source and ignores the claim. The judge compares the quote with the claim; it sees a little of the source as context but does not check that the quote is really there. Each misses what the other catches. The source texts were written for this demo; the trial they describe is fictional.
Source
Quote attached
Claim in the answer
The matcher reruns as you type. The judge needs a model: any OpenAI-compatible endpoint that accepts browser requests, such as OpenRouter. Your key stays in this tab and goes only to that endpoint.
The examples above were picked by hand. The results below were not: thousands of generated quotes for the
matcher, citations labelled by human annotators for the judge, [and
four open answering models following the citation instructions. All of it can be rerun from the
repository.] [Hover over a mark to see its numbers.]
VeriQuote is a TypeScript library and CLI with no dependencies, MIT licensed.
[Code, benchmarks and raw results are on GitHub](https://github.com/rickintoplace/veriquote).
Icons by Lucide (ISC licence).