Last week I shipped a small feature to a QA pipeline I maintain: before a score is stored, the model has to quote the transcript it claims to be scoring.
That sounds trivial. It is not, and the reason is worth a full article, because "make the model cite its evidence" is advice that feels sufficient and is not.
The standard advice for LLM-as-judge systems is: require a literal quote alongside each score. The reasoning is sound — if every score comes with a quote, a human can check it in seconds instead of minutes.
But nobody checks whether the quote exists.
I reviewed the code of a pipeline I was about to adopt, and this is what the persistence layer looked like:
score = Score(
value=float(item["value"]),
confidence=float(item.get("confidence") or 0),
evidence=(item.get("evidence") or "")[:4000],
rationale=(item.get("rationale") or "")[:4000],
)
session.add(score)
The prompt said, in bold, that every score must cite exact evidence. The code stored whatever string came back. If the model paraphrased, if it summarized, if it hallucinated a sentence that was never said, the pipeline would have stored it and displayed it with a confidence of 0.91 next to a number a supervisor was expected to trust.
The prompt is an instruction. The string is data. Instructions are not controls.
Here is the kind of output you get from a good model asked for literal evidence:
{
"criterion_key": "plazos",
"value": 2.0,
"confidence": 0.90,
"evidence": "AGENTE: Y después te aviso.",
"rationale": "The closing statement has no concrete deadline."
}
And here is the same model, asked the same way, on a worse run:
{
"criterion_key": "plazos",
"value": 2.0,
"confidence": 0.88,
"evidence": "The agent did not provide a specific timeframe.",
"rationale": "The closing promise was vague."
}
Both look reasonable to a reader. The second one is worthless: it is a description of an absence, which is exactly what a model produces when it cannot find a quote. It reads like evidence but contains no words from the transcript.
A human reviewer skimming it might approve it. The whole point of the quote was to make review cheap. A paraphrase puts the review cost back where it started, while looking like it was solved.
The verification does not need an embedding model, a second LLM call, or a vector store. It is a substring check against the transcript you already have.
The naive version breaks immediately: models quote with different capitalization, smart quotes, and they join two fragments with an ellipsis. So normalize first, then compare every fragment:
import re
import unicodedata
ELISION = re.compile(r"\s*\.\.\.\s*|\s*…\s*")
def _normalizar(texto: str) -> str:
plano = unicodedata.normalize("NFKD", texto)
plano = "".join(c for c in plano if not unicodedata.combining(c))
plano = plano.lower().replace("“", '"').replace("”", '"')
plano = plano.replace("‘", "'").replace("’", "'")
return re.sub(r"\s+", " ", plano).strip()
def evidencia_sostenida(evidencia: str | None, transcripcion: str | None) -> bool:
if not evidencia or not transcripcion:
return False
base = _normalizar(transcripcion)
if not base:
return False
fragmentos = [f for f in ELISION.split(_normalizar(evidencia)) if f]
if not fragmentos:
return False
return all(fragmento in base for fragmento in fragmentos)
Note the last line: every fragment has to appear. A model that quotes one real sentence and appends a fabricated one fails the check, which is the correct outcome.
Stripping accents is deliberate. Transcript pipelines produce inconsistent diacritics — ASR output, diarization labels, and imported captions do not agree on spelling, and a strict comparison would flag correct quotes as false. If you strip accents for matching, you also have to strip them from the quote, which is what _normalizar does.
Now the interesting part. A check that only records a warning changes nothing. The check has to change what happens to the score.
Two decisions carried most of the weight:
1. Unsupported evidence drops the confidence to zero. Not to "low" — to zero. Confidence in that system means "how much does the evidence support this score". If the evidence does not exist, the evidence supports nothing, and the correct confidence is zero. This also makes the routing automatic, because a zero-confidence score is below any sane review threshold and goes to a human. No new code path, no new flag.
2. The evidence keeps the failure attached. When the check fails, the stored evidence is the model's string plus a marker:
if not evidencia_sostenida(evidencia, recording.transcript):
confianza = 0.0
evidencia = (evidencia + " " + NO_VERIFICADA).strip()
Do not silently discard the model's answer. You want to see what it claimed, because the pattern of the failures is diagnostic: a model that paraphrases on one criterion and quotes correctly on another is telling you something about that criterion's wording.
A check that only logs is a check nobody reads. The pipeline now reports the rate of unverifiable citations:
def calidad_evidencia(session) -> dict:
total = session.scalar(select(func.count(Score.id))) or 0
if not total:
return {"puntajes": 0, "no_verificadas": 0, "tasa_no_verificada": None}
no_verificadas = session.scalar(
select(func.count(Score.id)).where(Score.evidence.like(f"%{NO_VERIFICADA}%"))
) or 0
return {
"puntajes": int(total),
"no_verificadas": int(no_verificadas),
"tasa_no_verificada": round(no_verificadas / total, 3),
}
That single number tells you whether your evaluation system is auditable. If the rate is near zero, the model is quoting and the review layer is cheap. If it is high, the scores are prose, and the numbers in your dashboard are decoration.
In my experience this rate correlates with how vague the rubric is. A criterion that says "the agent explains the procedure properly" gives the model nothing to quote, so it describes. A criterion that says "the agent states the return window in days and the shipping cost" gives it a quote to find. Fixing the rubric fixes the evidence rate — which is the same lesson as always, arriving from a different direction.
Two things, so you do not over-engineer around them:
Add the check and the zero-confidence rule. It is about fifteen lines and one metric, and it converts an instruction into an enforcement.
The prompt can tell the model to quote. Only code can tell you whether it did.
This is part of the QA with AI series. The kit I use for this, with the judge prompt, the rubric guide and the deviation reports, is here: https://kit-qa-con-ia.surge.sh/ (MIT license, US$79). If you only want the idea, the check above is the whole thing.
One question for you: if you have an LLM scoring pipeline in production right now, what percentage of the stored evidence strings would survive a literal substring check against your transcripts? I would bet it is lower than you expect, and I would like to hear the real number.