Why Your Enterprise RAG Pipeline Is Failing Before the First Query Runs

A developer argues that most enterprise RAG pipeline failures originate at the document parsing stage of ingestion, before any query runs, rather than in the LLM or retrieval layers teams typically tune. The writeup documents a production-grade ingestion pipeline built with Docling, LangChain, and a structured metadata extraction layer to address layout, table, and multi-column parsing failures at their source.

Most teams building Retrieval-Augmented Generation RAG systems invest their engineering effort in two places: the LLM and the retrieval layer. They tune prompts, experiment with embedding models, compare vector databases, and benchmark retrieval precision. When the system underperforms in production, they go back to those same two layers and tune again. The failure, in the majority of enterprise RAG deployments, is not there. It is at the ingestion layer — specifically at the document parsing stage — and it happens before a single query is ever issued. This is not a problem that appears during proof-of-concept work. In a POC, teams typically use small, clean documents: a few well-structured PDFs, maybe a Markdown file or two. Basic parsers work fine on those inputs. The results look reasonable. The system gets approved for production. In production, the document corpus looks different. It contains 200-page regulatory policy documents where tables start on page 22 and continue on page 23. It contains scanned audit reports where the actual content lives in raster images. It contains financial disclosures where running headers and footers break into the body text every page. It contains multi-column layouts where two entirely separate sections sit side by side on the same page. A basic parser cannot handle any of these cases correctly. It does not fail loudly. It silently discards structural context, flattens layout into a text stream, and passes that degraded output downstream. Every component that follows — the chunker, the embedding model, the vector index — operates on corrupted input. The hallucinations and retrieval failures that appear at query time are traced back to the LLM or the retrieval logic. The actual root cause sits two stages upstream, in the parser, and is never touched. This article walks through the precise mechanics of why this happens, and documents a production-grade ingestion pipeline built with Docling https://github.com/docling-project/docling , LangChain, and a structured metadata extraction layer designed to address these failures at their actual source. Understanding why standard parsers fail requires understanding what a PDF actually is at the data level. A PDF file is not a document. It is a page description language. Its internal representation does not encode semantic structure — headings, paragraphs, tables, or sections. It encodes rendering instructions: draw this glyph at this coordinate using this font at this size. There is no native concept of "this text belongs to this table cell" or "this heading is the parent of this section." That structure is a human visual inference from the rendered page layout. When a library like PyPDF or PDFMiner processes a PDF, it extracts the character sequences and their approximate positions on the page. Without a layout model, it has no mechanism to determine reading order, column boundaries, or the relationship between spatially separated elements. Its default behaviour is to read the page sequentially — typically left to right, top to bottom — treating the full page width as a single text stream. The failure modes this produces in enterprise documents are specific and compounding: A two-column regulatory document has column A on the left and column B on the right. A sequential reader scans across the page at each y-coordinate, producing output that reads: left-column sentence fragment, right-column sentence fragment, left-column continuation, right-column continuation. The output is a mixed stream of two entirely unrelated content sections. Both sections are now unparseable as coherent text, and neither can be chunked into a semantically meaningful unit. Enterprise documents routinely contain tables that span page boundaries. The table header row is on page 11; the data rows continue on page 12. A page-scoped parser processes each page independently. The table on page 11 is extracted as a fragment with no data rows. The continuation on page 12 is extracted as rows with no header context. Neither fragment contains enough information to be retrievable as a meaningful unit. When these fragments are chunked and embedded, they produce vectors that cannot answer any query correctly about that table's content. Page numbers, document titles, section labels, and legal footers appear on every page of an enterprise document. A parser with no layout model injects these strings into the body text at the point where they appear spatially. A chunk that should contain a clean paragraph of policy text instead contains: policy sentence fragment, "Page 47 of 312 — CONFIDENTIAL", remainder of policy sentence. The embedding of that chunk is polluted by the header token sequence, shifting the vector away from its true semantic position. Charts, diagrams, process flow diagrams, and scanned pages exist in enterprise documents as embedded raster images. A text-extraction parser has no path to retrieve content from these images. It silently skips them. In regulated banking environments — financial disclosures, audit frameworks, risk matrices — a significant fraction of the information that actually matters lives in these images. It never enters the RAG index. The cumulative effect of these failures arrives at the embedding model. The embedding model has no knowledge of what the original document looked like. It encodes the text sequence it receives. A corrupted chunk — mixed columns, severed table rows, injected headers — produces a vector in a region of the embedding space that does not accurately represent the semantic content of the source material. That vector is then stored in the index and retrieved in response to queries. The retrieved context is wrong. The LLM generates a response based on wrong context. This is the hallucination. Its origin is the parser. The failure chain is deterministic: Bad parsing → Corrupted text sequences → Structurally invalid chunks → Semantically inaccurate vectors → Irrelevant retrieval → LLM hallucination No amount of prompt engineering, model selection, or retrieval tuning corrects a failure that originates at stage one. The production ingestion pipeline is composed of four distinct functional layers: DocumentConverter with a configured PdfPipelineOptions pipeline HybridChunker Document construction with structured metadata, ready for embedding and vector store ingestion The pipeline is designed to be format-agnostic at the discovery layer and format-specific at the parsing layer. PDF files pass through the full PdfPipelineOptions pipeline. DOCX and Markdown files pass through Docling's native handlers. This avoids the common mistake of running PDF-specific configuration against non-PDF inputs. chunks.py The parsing configuration is the most critical engineering decision in the entire pipeline. Every downstream component inherits the quality of this stage. python from pathlib import Path from docling.document converter import DocumentConverter, PdfFormatOption from docling.datamodel.base models import InputFormat from docling.datamodel.pipeline options import PdfPipelineOptions, TableFormerMode, HeadingHierarchyOptions from docling.chunking import HybridChunker def load docs dir path : """Recursively discover supported documents. Returns Path objects.""" ROOT = Path dir path EXTS = {".pdf", ".docx", ".md"} return p for p in ROOT.rglob " " if p.is file and p.suffix.lower in EXTS and not p.name.startswith "." def parse file file path : """ Parse a single document using layout-aware Docling pipeline. Returns a chunker, document tuple for downstream chunking. """ file path = Path file path Pipeline configuration: heading hierarchy and parsed pages enabled; table structure and OCR disabled for native-text PDFs. opts = PdfPipelineOptions do table structure=False opts.heading hierarchy options = HeadingHierarchyOptions enabled=True opts.generate parsed pages = True opts.do ocr = False converter = DocumentConverter format options={InputFormat.PDF: PdfFormatOption pipeline options=opts } doc = converter.convert file path .document chunker = HybridChunker return chunker, doc def meta data metadata : """ Extract structured metadata from a chunk's ChunkMeta object. Page number is derived from provenance data across all doc items in the chunk. """ page = sorted {p.page no for it in metadata.doc items for p in it.prov} 0 if metadata.doc items else 0 return { "page": page, "filetype": metadata.origin.mimetype, "filename": metadata.origin.filename, "heading": " ".join metadata.headings if metadata.headings else "" } index.py python import os from dotenv import load dotenv from llm import embed documents from chunks import load docs, meta data, parse file from langchain core.documents import Document load dotenv def process documents dir path : """ End-to-end ingestion: discover → parse → chunk → contextualize → index. """ docs = load docs dir path for doc path in docs: chunker, parsed doc = parse file doc path for i, chunk in enumerate chunker.chunk parsed doc : metadata = meta data chunk.meta contextualize prepends the heading breadcrumb path to chunk text. This is the text that goes to the embedding model — not chunk.text. text = chunker.contextualize chunk document = Document page content=text, metadata={ metadata, "chunk index": i} embed documents document — downstream call to vector store if name == " main ": process documents os.environ.get "docs path" Step 1 — Document Discovery load docs The discovery function uses Path.rglob " " to traverse the directory tree recursively. It filters on a defined extension set {".pdf", ".docx", ".md"} and excludes hidden files by checking p.name.startswith "." . This prevents macOS resource fork files .DS Store , . filename from entering the pipeline and causing parse errors. The function returns Path objects rather than string paths, which allows the downstream parse file function to leverage pathlib 's existence check and type-safe path composition. Step 2 — Layout-Aware Parsing parse file This is where the pipeline diverges from a naive implementation. Four specific pipeline options are set: do table structure=False — This disables the TableFormer model, which is Docling's ML-based table cell recognition model. For native-text PDFs PDFs with an embedded text layer , the structural relationships between cells can be inferred from the text positions without running the full vision model. Disabling it significantly reduces parse latency per document. For scanned documents or PDFs where tables are rendered as images, this flag must be set to True and the corresponding do ocr=True flag must be enabled. HeadingHierarchyOptions enabled=True — By default, Docling's layout model classifies page regions as SECTION HEADER without assigning a level. Every heading in the exported document is treated as level 1, producing a flat structure with no parent-child relationship between sections. Enabling HeadingHierarchyOptions activates a post-processing stage that assigns heading levels using three signals in precedence order: PDF bookmarks from the document outline, leading numbering patterns in the heading text e.g., 1. , 1.1 , 2.3.4 , and visual font styling. This recovered hierarchy is what makes the HybridChunker 's structural chunking meaningful. generate parsed pages=True — This option keeps the raw parsed PDF cell data in memory after the layout analysis stage. It is specifically required for the font-style heading inference signal. Without it, the HeadingHierarchyOptions stage can still use bookmarks and numbering patterns, but font weight, slant, and size comparisons are skipped silently. In enterprise documents that have no bookmarks and no numbered headings — dense policy documents, legal agreements, free-form reports — font styling is often the only reliable heading signal available. This flag must be enabled to use it. do ocr=False — OCR is disabled for native-text PDFs. Running Tesseract or another OCR engine on a PDF that already has an embedded text layer introduces character recognition errors and significantly increases processing time. OCR should only be enabled for scanned documents or for pages that contain image-only regions. Step 3 — Hybrid Chunking HybridChunker HybridChunker operates in four sequential stages on the parsed DoclingDocument : LineBasedTokenChunker that repeats the header row for each split segment. Text elements use the semchunk algorithm, which splits at the most semantically meaningful boundary available rather than at a fixed character or token count. chunker.contextualize chunk call prepends the full heading breadcrumb path to the chunk text before it is passed to the embedding model. A chunk from a section 3 3.2 Access Control Requirements produces an embedding that encodes both the section context and the content — significantly improving retrieval precision for queries that reference specific sections without quoting them verbatim. Step 4 — Metadata Extraction meta data The meta data function derives the page number from the provenance data attached to each doc item in the chunk's metadata. A chunk that spans multiple pages will have prov entries from each page; the function takes the minimum page number as the canonical page reference. This is more accurate than line-based page attribution because the HybridChunker can merge elements from adjacent pages into a single chunk. The heading field concatenates the heading path as a " " -delimited breadcrumb string. This is stored as vector store metadata and enables filtered retrieval: a query scoped to a specific section of a document can filter on the heading field rather than relying solely on vector similarity. Step 5 — LangChain Document Construction Each processed chunk is wrapped in a LangChain Document object with page content set to the contextualized text not chunk.text and metadata containing the structured fields plus the sequential chunk index . The chunk index enables re-ranking by document position as a secondary signal when multiple chunks from the same document are retrieved with similar cosine scores. The current implementation sets do ocr=False globally. In a production corpus that contains a mix of native-text PDFs and scanned documents, this requires a pre-processing routing stage before the converter is invoked. A practical approach is to probe the document's text layer density before routing: python import fitz PyMuPDF def is native text pdf file path: Path, sample pages: int = 3 - bool: """ Sample the first N pages to determine if the PDF has a native text layer. Returns True for native-text PDFs, False for scanned/image-only PDFs. """ doc = fitz.open str file path pages to check = min sample pages, len doc total chars = sum len doc i .get text "text" .strip for i in range pages to check doc.close return total chars 100 threshold: fewer than 100 chars implies image-only pages Documents that fail the native-text check should be routed to a configuration with do ocr=True and, depending on table density, do table structure=True with TableFormerMode.ACCURATE . The HeadingHierarchyOptions signals apply in precedence order. For enterprise document corpora, the appropriate configuration depends on the document type: | Document Type | Recommended Signal Configuration | |---|---| | Regulatory PDF with bookmarked ToC | use bookmarks=True default , others as fallback | | Technical specifications with numbered sections | use numbering=True , use bookmarks=False if ToC is absent or unreliable | | Legal agreements, free-form policy documents | use style=True , use font style=True , requires generate parsed pages=True | | Mixed corpus | All signals enabled default when enabled=True | For a banking regulatory corpus — MAS TRM guidelines, internal policy documents, audit frameworks — the mixed configuration is appropriate because the document types are heterogeneous. contextualize vs chunk.text Decision This is a consistently misunderstood detail in Docling-based RAG implementations. chunk.text contains only the raw text of the chunk's content elements. chunker.contextualize chunk prepends the heading breadcrumb path to that text. For a chunk with headings "3. Security Controls", "3.2 Access Management" and body text "All privileged access must be logged..." , the outputs are: chunk.text: "All privileged access must be logged..." chunker.contextualize chunk : "3. Security Controls 3.2 Access Management All privileged access must be logged..." The embedding model encodes both versions. The contextualized version places the vector in a region of the embedding space that reflects the section semantics, not just the sentence semantics. A query for "what are the access management logging requirements" retrieves the contextualized chunk with significantly higher cosine similarity than the raw text chunk, because the heading tokens anchor the embedding to the correct domain. Always pass chunker.contextualize chunk to the embedding model. Store chunk.text separately if you need the raw text for display purposes. The metadata schema produced by meta data — page , filetype , filename , heading , chunk index — serves a dual purpose. It is stored alongside each vector in the vector store and enables metadata-filtered retrieval. In regulated environments, this has a direct compliance application. A query scoped to a specific document version filename filter or a specific section heading filter can be constrained at the retrieval layer rather than relying on the LLM to perform that scoping from the retrieved context. This reduces the surface area for cross-document content bleed, where a retrieved chunk from document version N-1 is used to answer a query that should be answered from version N. generate parsed pages=True Keeping the parsed PDF cells in memory generate parsed pages=True increases per-document memory consumption. For large documents — 200+ pages — this can add 300–500 MB of in-process memory overhead per concurrent parse operation. In a batch ingestion workload running on a fixed instance, the concurrency ceiling must be set to account for this. The tradeoff is worth it for enterprise corpora where free-form documents without bookmarks are common. For corpora where all documents have reliable PDF outlines or numbered sections, generate parsed pages can be set to False to reduce memory pressure, with use style=False in the HeadingHierarchyOptions . do table structure=False Tradeoff Disabling the TableFormer model eliminates the ML inference cost for table structure recognition, which is the most expensive single operation in the Docling pipeline for table-dense documents. For native-text PDFs, this is a reasonable tradeoff because cell relationships can be preserved through text-position inference. The failure mode: if native-text PDFs contain tables that use borderless layouts or unconventional spacing, the text-position inference may still produce incorrect cell associations. A validation step that samples the parsed table output before indexing is strongly recommended for corpora with complex table structures. The current implementation processes documents sequentially in a single process. For initial corpus ingestion at enterprise scale — thousands of documents — this does not meet practical time constraints. A production deployment should parallelise at the document level using a worker pool: python from concurrent.futures import ProcessPoolExecutor import os def process documents parallel dir path: str, max workers: int = 4 : docs = load docs dir path with ProcessPoolExecutor max workers=max workers as executor: futures = {executor.submit parse and index, doc path : doc path for doc path in docs} for future in futures: try: future.result except Exception as e: print f"Failed: {futures future } — {e}" Worker count should be tuned based on available CPU cores and the memory overhead of generate parsed pages=True . On an m6i.4xlarge 16 vCPU, 64 GiB , a practical ceiling is 4–6 workers for large-document corpora. The combination of layout-aware parsing, structure-driven chunking, and heading-enriched contextualization produces an index where retrieved chunks carry their structural position in the source document as part of the vector encoding. This has a measurable effect on retrieval precision for the query types that dominate enterprise RAG use cases: policy lookups, section-specific compliance checks, and cross-document comparisons. The cost of this precision is paid once, at ingestion time. The alternative — fast parsing, naive chunking, no contextualization — produces an index that is cheap to build and unreliable to query. In a regulated environment, an unreliable RAG system is not a minor inconvenience. It is an audit risk. Fix the ingestion layer. Everything downstream inherits its quality.