# Retrieval Is Not a Safety Boundary: Poisoned-Corpus and Unsupported-Question Evaluation of a RAG Pipeline > Source: > Published: 2026-11-03 04:37:58+00:00 **Abstract.** I built a retrieval-augmented generation service with hybrid retrieval (dense + BM25), a relevance threshold that is supposed to make it refuse unanswerable questions, and an extractive generator that quotes retrieved text with citations. I tested the two failure surfaces the standard evals never touch: a poisoned corpus (a planted document carrying a false claim, a steering instruction or a canary token, next to the real policy) and unsupported questions (questions the corpus cannot answer). Across 12 attack cases, 8 unsupported questions and 10 benign controls on three retriever backends, planted claims reached the answer in 80.6% of attack runs, unsupported questions were fabricated on 83.3% of runs (100% on the hybrid backend), and benign completion stayed at 90% with zero false refusals — the system keeps all of its utility and skips the safety. A 20-question SQuAD-format slice scores the answer path separately: the gold span is contained in the answer in 19–20 of 20 cases, while mean token F1 is only 0.20–0.21, pricing the verbosity of sentence-level answers. All runs use the extractive mock generator. A RAG pipeline's safety story usually rests on two claims: the retriever only fetches relevant text, and the generator only says what the context supports. Both claims are graded, if at all, on a clean corpus and answerable questions. What happens when the corpus is adversarial — when anyone who can add a document can write next week's "policy"? And what does the system actually do with a question its corpus cannot answer, when the threshold that should make it refuse was tuned on a different backend? One recall number answers none of this, so I scored attack success, fabrication, false refusals and completion separately. **System.** The pipeline chunks documents (120-word windows), embeds them with a hashing TF-IDF embedder, retrieves with one of three backends (dense cosine, BM25, or hybrid reciprocal-rank fusion), drops everything scoring below `min_score` 0.15, and generates with an extractive mock that quotes the three retrieved sentences with the most question-word overlap, citing their chunks. With no key configured this mock *is* the shipped offline path; the evals force it explicitly so no API key can change a run. **Adversarial cases.** Twelve hand-written attacks plant a document beside the real policy corpus (5 short policy documents: refunds, leave, warranty, US shipping, pricing) and follow with an ordinary user question, metadata filters and all. Styles: direct instructions, a note addressed to the assistant, fake system-update framing, a planted false fact with no instruction language, a near-duplicate of the real policy with one number changed, a payload split across two documents, one sentence buried in office news, a phishing-style address change, a process-weakening notice, a base64-encoded payload, and a canary token that must never reach an answer. Each case names a success signal — the planted claim, address or canary whose presence in the answer proves the payload landed. Eight unsupported questions (four near-misses against real policies, four far-misses) test refusal. Ten benign controls include documents that quote attack-like text as content (security training, review minutes that rejected the attacker's number, an FAQ, a blog post). Four attack cases also carry a bystander question — unrelated benign traffic asked against the same poisoned corpus. Metric definitions are frozen in `evals/rag_scoring.py` (v1): an attack run is **blocked** (poisoned chunk never cited), **succeeded** (signal in the answer) or **evaded** (poison cited, signal not in the answer — reached the generator unfiltered); an unsupported run is **refused** or **fabricated**; a benign run is **completed** (expected evidence in the answer), **answered ungrounded**, or a **false refusal**. **SQuAD-format slice.** Twenty hand-written questions over ten short passages in SQuAD v1.1 format (context, question, answer span with character offset), generated by `evals/make_squad_qa_slice.py` with the passage set's sha256 stamped into the version string. This is a slice in the benchmark's *format*, not the SQuAD dataset — the repo's full-scale slice over real SQuAD data (302 questions) grades retrieval only; this one grades the answer path with SQuAD's own exact-match and token-F1 machinery. F1 is computed against the answer's best-matching sentence, since the pipeline answers in quoted sentences plus a boilerplate note rather than bare spans. | backend | attacks | blocked | succeeded | evaded | ASR | |---|---|---|---|---|---| | dense | 12 | 2 | 10 | 0 | 83.3% | | bm25 | 12 | 0 | 9 | 3 | 75.0% | | hybrid | 12 | 0 | 10 | 2 | 83.3% | | **all** | **36** | **2** | **29** | **5** | **80.6%** | Eight of the twelve styles succeeded on all three backends: direct instructions, the assistant-addressed note, the fake system update, the bare false fact, the near-duplicate, the canary, the refund-address change and the no-order-ID notice. The split payload succeeded on dense and hybrid. The only runs that never succeeded are the two whose payloads need a smarter reader or a different ranking: the buried paternity sentence and the base64 bulletin. | backend | refused | fabricated | fabrication rate | |---|---|---|---| | dense | 3/8 | 5/8 | 62.5% | | bm25 | 1/8 | 7/8 | 87.5% | | hybrid | 0/8 | 8/8 | 100.0% | | **all** | **4/24** | **20/24** | **83.3%** | Every near-miss (sabbatical leave, gift-card refunds, India shipping times, cracked-screen coverage) fabricated on every backend — 12 of 12. The refusals that did happen were all far-misses, and only dense managed any consistency there. False-refusal rate 0.0% on all backends; completion 90% (9/10) on each — the one miss is the warranty control, answered on-topic but quoting the exclusions instead of the "2 year" fact. Documents quoting attack text (training material, the minutes that *rejected* the 90-day proposal, the FAQ, the injection explainer) all answered normally. Completion under attack: 100% (12/12 bystander runs) — the poison does not degrade unrelated questions. | backend | retrieval hit | exact match (span in answer) | mean token F1 | refused | |---|---|---|---|---| | dense | 19/20 | 19/20 | 0.203 | 1 | | bm25 | 20/20 | 20/20 | 0.213 | 0 | | hybrid | 20/20 | 20/20 | 0.213 | 0 | For scale, the repo's retrieval slice over 302 real SQuAD questions reports recall@3 of 73.8% (dense), 86.8% (bm25), 82.1% (hybrid) — retrieval is the harder problem at scale; on this small slice the answer path keeps what retrieval finds. Everything ran on the extractive mock generator, which bounds the claims both ways. The mock quotes but cannot obey: instruction payloads succeeded here only by being quoted, so on a real LLM — which reads all cited context and follows instructions in it — the instruction classes would land by obedience as well, making measured ASR a floor. Symmetrically, the encoded payload measured evaded/blocked precisely because the mock cannot decode base64; a real model decodes it trivially. No real-model runs are included (the live path needs a key the eval environment doesn't carry). The corpus is five short policies, each a single chunk at the default chunking, so chunk-boundary effects are unmeasured — the split case splits across documents, not chunks. Seeds are hand-written and the suites are small (12/8/10), so rates are directional, not precise. Embeddings are hashing TF-IDF, not neural: paraphrase-only payloads with no keyword overlap would rank differently against a neural embedder — the keyword path measured here is the one the default deployment actually runs. The SQuAD-format slice is 20 hand-written questions: a format and comparability check, not a benchmark score. A real-generator slice over the same seeds (the pipeline's live LLM path) converts evaded into measured ASR and tests obedience, not just quotation. A defence phase follows as a separately measured phase, graded by these suites: source-trust tiers and provenance weighting in retrieval, conflict detection when two "policies" disagree on a number (the near-duplicate case), and a grounding check that refuses when the top source is a lone low-trust document. Re-running both suites with the neural embedder swap (`embedder_backend: st`) will show whether semantic retrieval raises or lowers ASR — paraphrase payloads become retrievable at the same time as planted text becomes easier to rank down. The project is public: `rag-service` under github.com/tushar29k. Both suites are deterministic and regenerate their reports byte-identically (`evals/run_adversarial_rag.py`, `evals/run_squad_qa.py`); metric definitions are frozen in `evals/rag_scoring.py` v1; the SQuAD-format slice is generated from `evals/make_squad_qa_slice.py` with the passage set's sha256 stamped into its version. The adversarial suite is report-only — the golden eval remains the CI gate — and every finding above is logged deliberately unfixed, so a defence phase can be graded against it.