Python Retrieval Pipelines for Support Content — Enforcing Crawl Boundaries

Short answer: a retrieval architecture for a customer-support knowledge base should treat crawl boundaries as a data contract and an incident boundary, not as a single allowlist. For an e-commerce support corpus, fetch only approved source classes, preserve document and section citations, and quarantine anything whose ownership or freshness is unclear before it reaches the vector index.

A support answer is only as trustworthy as the text behind its citation. The Retrieval-Augmented Generation (RAG) paper describes a generator that conditions on retrieved passages; that makes retrieval provenance part of the answer, not an afterthought. In practice, the expensive failure is rarely a failed HTTP request. It is a plausible paragraph assembled from an outdated return policy, a staging page, and an internal note that should never have been crawled.

I build email, SMS, and OTP systems, so I am biased toward explicit fences. A message that lands in the wrong inbox is an incident; a support citation that points to the wrong policy deserves the same treatment.

Start with four invariants. The crawler must know which hosts and URL prefixes are owned by the support team, which content states are publishable, which records contain customer or employee data, and which timestamp is authoritative. These are policy fields, not heuristics hidden in a parser.

For an e-commerce knowledge base, an allowlist might include published help-center articles, approved API reference pages, and versioned return-policy documents. It should exclude checkout sessions, account pages, search-result URLs, preview links, and ticket transcripts unless a separate data steward has approved them. robots.txt is useful crawler guidance, but it is not an authorization system; the application still needs its own scope and access checks.

The boundary should travel with every chunk. Store a source URI, canonical URI, document revision, section path, visibility class, and fetched-at time beside the embedding. A citation can then point to the exact revision that supplied the passage. If a policy is replaced, old chunks can be invalidated by revision rather than by guessing which text changed.

A useful mental model is a quarantine lane. New material is fetched into raw storage, normalized, and checked against policy. Only after those checks does it become eligible for chunking and indexing. This separates “we received bytes” from “we are willing to answer with them.”

The critical path is intentionally boring. It makes a bad boundary visible before an embedding call hides the evidence.

from dataclasses import dataclass
from urllib.parse import urlparse

@dataclass
class SourceRecord:
    url: str
    canonical_url: str
    state: str
    visibility: str
    revision: str
    section: str
    text: str

def in_scope(record: SourceRecord, allowed_hosts: set[str], allowed_prefixes: tuple[str, ...]) -> bool:
    parsed = urlparse(record.canonical_url)
    host_ok = parsed.hostname in allowed_hosts
    path_ok = parsed.path.startswith(allowed_prefixes)
    return (
        host_ok
        and path_ok
        and record.state == "published"
        and record.visibility == "public"
        and bool(record.revision)
    )

def make_chunks(record: SourceRecord, max_chars: int = 1800) -> list[dict]:
    if not record.text.strip():
        return []
    chunks = []
    for offset in range(0, len(record.text), max_chars):
        chunk_text = record.text[offset:offset + max_chars]
        chunks.append({
            "text": chunk_text,
            "citation": {
                "url": record.canonical_url,
                "revision": record.revision,
                "section": record.section,
                "offset": offset,
            },
        })
    return chunks

The example is not a crawler implementation. It is the contract I want to test. Notice that the index receives a revision and section, not just a vector and a blob of text. Chunk size is a tunable boundary for context, while host, path, state, and visibility are safety boundaries. Mixing those concerns makes emergency exclusion difficult.

One trap is canonicalization. Redirects and tracking parameters can turn one article into many apparent documents. Canonicalize before scope checks, retain the original URL for audit, and reject a redirect that leaves the approved host set. Another trap is recursive navigation: a “related articles” widget can lead from public help content to a logged-in account route. Depth limits help with load, but they do not replace a per-URL policy check.

Crawl jobs should emit a decision record for every candidate: accepted, rejected, or quarantined, with a reason code. That gives operators something concrete to inspect during a policy change. Metrics worth keeping include candidates by reason, chunks by source revision, citation miss rate, and age of the newest accepted document. I want an alert when accepted volume drops sharply, but I do not want an automatic “crawl everything” fallback.

When a source changes ownership, freeze its new chunks and keep the previous revision available only if policy permits it. When a parser sees a malformed page, quarantine the page and preserve the raw response for diagnosis. Do not silently index a partial article; partial text can produce a citation that looks precise while omitting the qualifying sentence.

I once assumed a path prefix was enough protection. It was not: a locale redirect returned an HTTP 302, moved /help/returns to a preview host, and the preview carried a draft exception. Ive learned to log the original URL, every 3xx hop, the final host, and the policy decision as one event. That event lets an on-call engineer replay the exact candidate without re-crawling the site, compare the published revision with the quarantined revision, and explain why a chunk did or did not enter the index. The fix was a redirect-aware scope check plus a published-state assertion. The lesson generalizes: every boundary needs a failure mode that is safe to observe.

Three words: quarantine, explain, replay.

That small vocabulary also helps during a rollback. If a source owner revokes a document, the indexer can mark its revision ineligible, leave the manifest intact, and regenerate answers against the last approved snapshot. No emergency delete script needs to infer document identity from vector similarity.

There is no universal fence. Choose the narrowest strategy that matches the content owners and their publishing discipline.

A broad traversal strategy is unsuitable when the site mixes public help with account-specific data. Stick with an export or metadata-driven feed when legal review requires a complete inventory of what entered the corpus. Use link traversal when discovery matters more than immediate coverage, and keep the runtime checks in place.

Chunking and embedding are downstream of governance. Keep a manifest that maps document revision to chunk identifiers, embedding model version, and index namespace. A re-crawl should be idempotent: the same revision produces the same chunk IDs, while a changed revision creates a new set and retires the old set according to retention policy.

At query time, filter by visibility and current revision before ranking. Return citation metadata with the passage so the answer layer cannot invent a source link after retrieval. If no eligible passage survives the filters, the correct response is an escalation or a clarifying question, not a confident guess.

Your mileage may vary on chunk length. Product catalogs with dense tables need different boundaries from prose return policies, and I am not sure one universal splitter survives both. Measure citation completeness on representative questions, then adjust the splitter without weakening the crawl contract.

The catch is operational overhead: decision logs, revision manifests, and quarantine storage take engineering time. This approach is not suitable for a throwaway internal search tool with disposable data. For customer-facing support, though, the overhead buys a defensible answer trail and a way to stop one bad source from poisoning every response.