Sentence-Window RAG for Better Context

A developer published a tutorial demonstrating sentence-window retrieval-augmented generation (RAG), a technique that indexes individual sentences for precise matching and then expands each hit into a surrounding window of nearby sentences to preserve context. The dependency-free Python prototype, built with only the standard library, avoids model API calls so users can inspect whether retrieval selects adequate evidence before adding answer generation. The writeup cites research showing a multi-abstraction retrieval approach improved AI-evaluated question-answer correctness by 25.739% on a Glycoscience-paper evaluation compared with a traditional single-level approach.

🚀 Technical Briefing: This tutorial is part of our deep-dive series on Agentic Workflows at Gate of AI https://gateofai.com . For the full technical breakdown, interactive code sandbox, and the native Arabic translation, visit the original article here https://gateofai.com/tutorial/sentence-window-rag-better-context/ . Build a local, dependency-free sentence-window retrieval prototype, understand why precise retrieval needs surrounding context, and evaluate the evidence before connecting the pattern to a production RAG stack. Retrieval-augmented generation, usually shortened to RAG, gives an answer system external text to consult at query time. A common implementation choice is to split every document into fixed-size chunks and retrieve the chunks most related to a question. That approach is useful, but it creates a persistent design trade-off. Small chunks can make retrieval precise while removing definitions, conditions, and exceptions. Large chunks can restore context while adding irrelevant material to the prompt. Sentence-window retrieval addresses that trade-off by separating the unit used for retrieval from the unit used for interpretation. The system indexes individual sentences. When a sentence is selected, the system expands it into a local window containing nearby sentences from the same document. The retrieval signal remains precise, while the reader receives a fuller passage. This tutorial intentionally uses only the Python standard library. The verified context does not establish current APIs, package versions, model availability, or persistence behavior for a particular RAG framework or model provider. A local prototype is therefore the most accurate way to demonstrate the technique without presenting unverified architecture as fact. Once the behavior is understood and tested, map the same concepts to the components that your organization has independently verified. A sentence often contains the words that best match a user question but not the complete meaning. Consider a policy passage with a rule, an exception, and a deadline. A query may match the sentence containing the deadline, while the preceding sentence says the policy applies only to a particular role. Returning the deadline alone can create a misleading answer. The verified research context identifies a related problem in conventional RAG: retrieving too much information can create token-limit pressure and the “lost in the middle” problem, where relevant details become less useful among excessive context. The same research proposes retrieving chunks at multiple abstraction levels, including multi-sentence, paragraph, section, and document levels. In its Glycoscience-paper evaluation, that approach improved AI-evaluated question-answer correctness by 25.739% compared with a traditional single-level approach. This is a research result for that evaluation, not a promise that every corpus or sentence-window configuration will improve by the same amount. A sentence window is one practical multi-sentence context pattern. It is especially appropriate when facts and their qualifications are usually located near each other. It is less suitable when the evidence required to answer a question is dispersed across distant sections or multiple documents. In those cases, a system may need broader retrieval, additional abstraction levels, or a document structure designed for the task. This prototype does not call a model API. It retrieves evidence and prints the selected context windows. That boundary is deliberate: it lets you inspect whether retrieval has selected adequate evidence before introducing answer generation. Create a project directory and two short Markdown documents. The sample corpus is fictional. Its purpose is to make it easy to see why a sentence match alone may not carry enough context. mkdir sentence-window-rag cd sentence-window-rag mkdir data cat data/travel policy.md <<'EOF' Travel Policy Employees must use the approved travel portal when inventory is available. Economy class is required for flights shorter than six hours. Premium economy may be booked for flights of six hours or longer. Business class requires written approval from a vice president before booking. A manager approval is not sufficient. The approval email must be attached to the expense report. EOF cat data/security policy.md <<'EOF' Security Policy Privileged production access requires multi-factor authentication and an approved access request. Shared user accounts are prohibited. Temporary production access expires automatically after eight hours unless an incident commander extends it during an active incident. Employees must report suspected security incidents immediately through the incident portal. If the portal is unavailable, employees must contact the on-call security engineer. EOF Keep documents that you index within the authorization boundary of the intended users. This local example has no authentication, filtering, or remote service. Do not treat it as a ready-made system for confidential documents. Create sentence window rag.py . The script reads Markdown and text files, separates text into simple sentence-like units, calculates a transparent lexical relevance score, and expands every result into neighboring sentences from the same source file. It is a learning implementation, not a linguistic sentence parser or a semantic-vector retrieval engine. python from future import annotations import argparse import math import re from collections import Counter from dataclasses import dataclass from pathlib import Path TOKEN PATTERN = re.compile r" a-z0-9 +" SENTENCE PATTERN = re.compile r" ?<= . ? \s+" @dataclass frozen=True class SentenceRecord: source file: str position: int text: str def tokenize text: str - list str : return TOKEN PATTERN.findall text.lower def split sentences text: str - list str : cleaned = re.sub r"^ +\s+. $", "", text, flags=re.MULTILINE cleaned = re.sub r"\s+", " ", cleaned .strip if not cleaned: return return part.strip for part in SENTENCE PATTERN.split cleaned if part.strip def load records data dir: Path - list SentenceRecord : records: list SentenceRecord = for path in sorted data dir.rglob " " : if not path.is file or path.suffix.lower not in {".md", ".txt"}: continue text = path.read text encoding="utf-8" for position, sentence in enumerate split sentences text : records.append SentenceRecord source file=path.name, position=position, text=sentence, if not records: raise ValueError "No non-empty .md or .txt sentences were found in the data directory." return records def inverse document frequency records: list SentenceRecord - dict str, float : document frequency: Counter str = Counter for record in records: document frequency.update set tokenize record.text total = len records return { token: math.log total + 1 / count + 1 + 1.0 for token, count in document frequency.items } def score question: str, sentence: str, idf: dict str, float - float: question terms = Counter tokenize question sentence terms = Counter tokenize sentence if not question terms or not sentence terms: return 0.0 numerator = sum question terms token sentence terms token idf.get token, 0.0 2 for token in question terms question norm = math.sqrt sum count idf.get token, 0.0 2 for token, count in question terms.items sentence norm = math.sqrt sum count idf.get token, 0.0 2 for token, count in sentence terms.items if question norm == 0.0 or sentence norm == 0.0: return 0.0 return numerator / question norm sentence norm def context window records: list SentenceRecord , record: SentenceRecord, radius: int - str: same file = item for item in records if item.source file == record.source file start = max 0, record.position - radius end = min len same file , record.position + radius + 1 return " ".join item.text for item in same file start:end def search records: list SentenceRecord , question: str, top k: int, radius: int : idf = inverse document frequency records ranked = sorted score question, record.text, idf , record for record in records , key=lambda item: item 0 , reverse=True, return relevance, record, context window records, record, radius for relevance, record in ranked :top k if relevance 0.0 def main - None: parser = argparse.ArgumentParser description="Inspect sentence-window retrieval." parser.add argument "question", help="Question to search for" parser.add argument "--data-dir", default="data" parser.add argument "--top-k", type=int, default=3 parser.add argument "--window", type=int, default=1 args = parser.parse args if args.top k < 1 or args.window < 0: raise SystemExit "--top-k must be at least 1 and --window must be zero or greater." records = load records Path args.data dir results = search records, args.question, args.top k, args.window if not results: print "No lexical overlap was found. This prototype should abstain rather than answer." return for number, relevance, record, window in enumerate results, start=1 : print f"Result {number}" print f"Source: {record.source file}" print f"Sentence position: {record.position}" print f"Lexical score: {relevance:.4f}" print f"Retrieved sentence: {record.text}" print f"Context window: {window}\n" if name == " main ": main The retrieved sentence is the narrow evidence unit. The context window is the expanded evidence unit. The --window value is a radius: a value of 1 includes the selected sentence plus up to one preceding and one following sentence. Document boundaries limit the window automatically. Run the following commands. Start with a one-sentence radius, then compare the output with a radius of zero. The difference demonstrates why a sentence may be a strong retrieval match but a weak standalone citation. python sentence window rag.py "Who can approve business class travel?" --window 1 python sentence window rag.py "Who can approve business class travel?" --window 0 python sentence window rag.py "What are the requirements for temporary production access?" --window 1 python sentence window rag.py "What is the parental leave policy?" --window 1 For business-class travel, inspect whether the window contains both the vice-president requirement and the statement that manager approval is insufficient. For temporary production access, inspect whether the selected window contains the authentication requirement, the approved access request, the eight-hour expiry, and the incident-commander exception. The exact ranking is not the lesson; this prototype uses lexical scoring rather than semantic embeddings. The lesson is that the answerer should see the surrounding conditions before producing an answer. The parental-leave query is an abstention test. The sample corpus has no relevant source. A trustworthy next stage should not convert unrelated travel or security passages into an invented policy. Retaining the retrieved evidence in the result makes this failure visible to a reviewer. Do not judge a RAG design only by whether it produces fluent prose. Evaluate it in layers. First, ask whether the correct source passage appears among the retrieved results. Second, ask whether the selected window includes the qualifications necessary to interpret that passage. Third, once an answer component is added, ask whether each material statement in the answer is supported by the selected evidence. Finally, test whether the system abstains when the corpus does not contain an answer. Create a compact evaluation file such as evaluation.jsonl . Each record can include a question, expected source file, required concepts, and whether abstention is expected. For example, the travel question should require the concepts “written approval,” “vice president,” and “manager approval is not sufficient.” The access question should require “multi-factor authentication,” “approved access request,” “eight hours,” and the active-incident exception. Run the same evaluation set when you adjust sentence splitting, window size, ranking method, document formatting, or the downstream answer prompt. This turns tuning into a comparison process rather than an anecdotal exercise. A larger window is not automatically better: it can add useful conditions, but it can also add unrelated language that distracts an answer system. Likewise, retrieving more sentences can improve recall while increasing context volume. The local script is not a production service. It does not provide semantic retrieval, access control, document-version management, API authentication, concurrency controls, or answer generation. Those are separate design decisions. When moving to a verified framework and provider stack, preserve the core sequence: parse trusted documents into sentence-level retrieval units; store a local window as associated context; retrieve narrow evidence; replace or supplement the retrieval unit with its window; present citations alongside any generated answer; and measure retrieval and answer support independently. Use the smallest context that reliably retains key conditions. If questions commonly require information distributed across paragraphs, sections, or documents, evaluate multiple retrieval abstraction levels rather than assuming a fixed sentence window will solve every case. This is consistent with the verified research context: information needs can occur at more than one level of abstraction, while excessive retrieved text can harm usefulness. For sensitive or regulated material, apply authorization before text is selected for an answer workflow. Maintain an evaluation corpus that reflects the documents and users your system actually serves. Avoid presenting research benchmarks as deployment guarantees, and avoid relying on a prompt alone to compensate for missing evidence or inappropriate retrieval.