{"slug": "why-your-enterprise-rag-pipeline-is-failing-before-the-first-query-runs", "title": "Why Your Enterprise RAG Pipeline Is Failing Before the First Query Runs", "summary": "A developer argues that most enterprise RAG pipeline failures originate at the document parsing stage of ingestion, before any query runs, rather than in the LLM or retrieval layers teams typically tune. The writeup documents a production-grade ingestion pipeline built with Docling, LangChain, and a structured metadata extraction layer to address layout, table, and multi-column parsing failures at their source.", "body_md": "Most teams building Retrieval-Augmented Generation (RAG) systems invest their engineering effort in two places: the LLM and the retrieval layer. They tune prompts, experiment with embedding models, compare vector databases, and benchmark retrieval precision. When the system underperforms in production, they go back to those same two layers and tune again.\n\nThe failure, in the majority of enterprise RAG deployments, is not there.\n\nIt is at the ingestion layer — specifically at the document parsing stage — and it happens before a single query is ever issued.\n\nThis is not a problem that appears during proof-of-concept work. In a POC, teams typically use small, clean documents: a few well-structured PDFs, maybe a Markdown file or two. Basic parsers work fine on those inputs. The results look reasonable. The system gets approved for production.\n\nIn production, the document corpus looks different. It contains 200-page regulatory policy documents where tables start on page 22 and continue on page 23. It contains scanned audit reports where the actual content lives in raster images. It contains financial disclosures where running headers and footers break into the body text every page. It contains multi-column layouts where two entirely separate sections sit side by side on the same page.\n\nA basic parser cannot handle any of these cases correctly. It does not fail loudly. It silently discards structural context, flattens layout into a text stream, and passes that degraded output downstream. Every component that follows — the chunker, the embedding model, the vector index — operates on corrupted input. The hallucinations and retrieval failures that appear at query time are traced back to the LLM or the retrieval logic. The actual root cause sits two stages upstream, in the parser, and is never touched.\n\nThis article walks through the precise mechanics of why this happens, and documents a production-grade ingestion pipeline built with [Docling](https://github.com/docling-project/docling), LangChain, and a structured metadata extraction layer designed to address these failures at their actual source.\n\nUnderstanding why standard parsers fail requires understanding what a PDF actually is at the data level.\n\nA PDF file is not a document. It is a page description language. Its internal representation does not encode semantic structure — headings, paragraphs, tables, or sections. It encodes rendering instructions: draw this glyph at this coordinate using this font at this size. There is no native concept of \"this text belongs to this table cell\" or \"this heading is the parent of this section.\" That structure is a human visual inference from the rendered page layout.\n\nWhen a library like PyPDF or PDFMiner processes a PDF, it extracts the character sequences and their approximate positions on the page. Without a layout model, it has no mechanism to determine reading order, column boundaries, or the relationship between spatially separated elements. Its default behaviour is to read the page sequentially — typically left to right, top to bottom — treating the full page width as a single text stream.\n\nThe failure modes this produces in enterprise documents are specific and compounding:\n\nA two-column regulatory document has column A on the left and column B on the right. A sequential reader scans across the page at each y-coordinate, producing output that reads: left-column sentence fragment, right-column sentence fragment, left-column continuation, right-column continuation. The output is a mixed stream of two entirely unrelated content sections. Both sections are now unparseable as coherent text, and neither can be chunked into a semantically meaningful unit.\n\nEnterprise documents routinely contain tables that span page boundaries. The table header row is on page 11; the data rows continue on page 12. A page-scoped parser processes each page independently. The table on page 11 is extracted as a fragment with no data rows. The continuation on page 12 is extracted as rows with no header context. Neither fragment contains enough information to be retrievable as a meaningful unit. When these fragments are chunked and embedded, they produce vectors that cannot answer any query correctly about that table's content.\n\nPage numbers, document titles, section labels, and legal footers appear on every page of an enterprise document. A parser with no layout model injects these strings into the body text at the point where they appear spatially. A chunk that should contain a clean paragraph of policy text instead contains: policy sentence fragment, \"Page 47 of 312 — CONFIDENTIAL\", remainder of policy sentence. The embedding of that chunk is polluted by the header token sequence, shifting the vector away from its true semantic position.\n\nCharts, diagrams, process flow diagrams, and scanned pages exist in enterprise documents as embedded raster images. A text-extraction parser has no path to retrieve content from these images. It silently skips them. In regulated banking environments — financial disclosures, audit frameworks, risk matrices — a significant fraction of the information that actually matters lives in these images. It never enters the RAG index.\n\nThe cumulative effect of these failures arrives at the embedding model. The embedding model has no knowledge of what the original document looked like. It encodes the text sequence it receives. A corrupted chunk — mixed columns, severed table rows, injected headers — produces a vector in a region of the embedding space that does not accurately represent the semantic content of the source material. That vector is then stored in the index and retrieved in response to queries. The retrieved context is wrong. The LLM generates a response based on wrong context. This is the hallucination. Its origin is the parser.\n\nThe failure chain is deterministic:\n\n```\nBad parsing\n  → Corrupted text sequences\n    → Structurally invalid chunks\n      → Semantically inaccurate vectors\n        → Irrelevant retrieval\n          → LLM hallucination\n```\n\nNo amount of prompt engineering, model selection, or retrieval tuning corrects a failure that originates at stage one.\n\nThe production ingestion pipeline is composed of four distinct functional layers:\n\n`DocumentConverter` with a configured `PdfPipelineOptions` pipeline`HybridChunker`\n`Document` construction with structured metadata, ready for embedding and vector store ingestion\nThe pipeline is designed to be format-agnostic at the discovery layer and format-specific at the parsing layer. PDF files pass through the full `PdfPipelineOptions` pipeline. DOCX and Markdown files pass through Docling's native handlers. This avoids the common mistake of running PDF-specific configuration against non-PDF inputs.\n\n`chunks.py`\nThe parsing configuration is the most critical engineering decision in the entire pipeline. Every downstream component inherits the quality of this stage.\n\n``` python\nfrom pathlib import Path\nfrom docling.document_converter import DocumentConverter, PdfFormatOption\nfrom docling.datamodel.base_models import InputFormat\nfrom docling.datamodel.pipeline_options import (\n    PdfPipelineOptions,\n    TableFormerMode,\n    HeadingHierarchyOptions\n)\nfrom docling.chunking import HybridChunker\n\ndef load_docs(dir_path):\n    \"\"\"Recursively discover supported documents. Returns Path objects.\"\"\"\n    ROOT = Path(dir_path)\n    EXTS = {\".pdf\", \".docx\", \".md\"}\n    return [\n        p for p in ROOT.rglob(\"*\")\n        if p.is_file() and p.suffix.lower() in EXTS and not p.name.startswith(\".\")\n    ]\n\ndef parse_file(file_path):\n    \"\"\"\n    Parse a single document using layout-aware Docling pipeline.\n    Returns a (chunker, document) tuple for downstream chunking.\n    \"\"\"\n    file_path = Path(file_path)\n\n    # Pipeline configuration: heading hierarchy and parsed pages enabled;\n    # table structure and OCR disabled for native-text PDFs.\n    opts = PdfPipelineOptions(do_table_structure=False)\n    opts.heading_hierarchy_options = HeadingHierarchyOptions(enabled=True)\n    opts.generate_parsed_pages = True\n    opts.do_ocr = False\n\n    converter = DocumentConverter(\n        format_options={InputFormat.PDF: PdfFormatOption(pipeline_options=opts)}\n    )\n\n    doc = converter.convert(file_path).document\n    chunker = HybridChunker()\n    return chunker, doc\n\ndef meta_data(metadata):\n    \"\"\"\n    Extract structured metadata from a chunk's ChunkMeta object.\n    Page number is derived from provenance data across all doc_items in the chunk.\n    \"\"\"\n    page = (\n        sorted({p.page_no for it in metadata.doc_items for p in it.prov})[0]\n        if metadata.doc_items else 0\n    )\n    return {\n        \"page\":     page,\n        \"filetype\": metadata.origin.mimetype,\n        \"filename\": metadata.origin.filename,\n        \"heading\":  \" > \".join(metadata.headings) if metadata.headings else \"\"\n    }\n```\n\n`index.py`\n\n``` python\nimport os\nfrom dotenv import load_dotenv\nfrom llm import embed_documents\nfrom chunks import load_docs, meta_data, parse_file\nfrom langchain_core.documents import Document\n\nload_dotenv()\n\ndef process_documents(dir_path):\n    \"\"\"\n    End-to-end ingestion: discover → parse → chunk → contextualize → index.\n    \"\"\"\n    docs = load_docs(dir_path)\n\n    for doc_path in docs:\n        chunker, parsed_doc = parse_file(doc_path)\n\n        for i, chunk in enumerate(chunker.chunk(parsed_doc)):\n            metadata = meta_data(chunk.meta)\n\n            # contextualize() prepends the heading breadcrumb path to chunk text.\n            # This is the text that goes to the embedding model — not chunk.text.\n            text = chunker.contextualize(chunk)\n\n            document = Document(\n                page_content=text,\n                metadata={**metadata, \"chunk_index\": i}\n            )\n\n            # embed_documents(document) — downstream call to vector store\n\nif __name__ == \"__main__\":\n    process_documents(os.environ.get(\"docs_path\"))\n```\n\n**Step 1 — Document Discovery (`load_docs`)**\n\nThe discovery function uses `Path.rglob(\"*\")` to traverse the directory tree recursively. It filters on a defined extension set `{\".pdf\", \".docx\", \".md\"}` and excludes hidden files by checking `p.name.startswith(\".\")`. This prevents macOS resource fork files (`.DS_Store`, `._filename`) from entering the pipeline and causing parse errors. The function returns `Path` objects rather than string paths, which allows the downstream `parse_file` function to leverage `pathlib`'s existence check and type-safe path composition.\n\n**Step 2 — Layout-Aware Parsing (`parse_file`)**\n\nThis is where the pipeline diverges from a naive implementation. Four specific pipeline options are set:\n\n`do_table_structure=False` — This disables the TableFormer model, which is Docling's ML-based table cell recognition model. For native-text PDFs (PDFs with an embedded text layer), the structural relationships between cells can be inferred from the text positions without running the full vision model. Disabling it significantly reduces parse latency per document. For scanned documents or PDFs where tables are rendered as images, this flag must be set to `True` and the corresponding `do_ocr=True` flag must be enabled.\n\n`HeadingHierarchyOptions(enabled=True)` — By default, Docling's layout model classifies page regions as `SECTION_HEADER` without assigning a level. Every heading in the exported document is treated as level 1, producing a flat structure with no parent-child relationship between sections. Enabling `HeadingHierarchyOptions` activates a post-processing stage that assigns heading levels using three signals in precedence order: PDF bookmarks from the document outline, leading numbering patterns in the heading text (e.g., `1.`, `1.1`, `2.3.4`), and visual font styling. This recovered hierarchy is what makes the `HybridChunker`'s structural chunking meaningful.\n\n`generate_parsed_pages=True` — This option keeps the raw parsed PDF cell data in memory after the layout analysis stage. It is specifically required for the font-style heading inference signal. Without it, the `HeadingHierarchyOptions` stage can still use bookmarks and numbering patterns, but font weight, slant, and size comparisons are skipped silently. In enterprise documents that have no bookmarks and no numbered headings — dense policy documents, legal agreements, free-form reports — font styling is often the only reliable heading signal available. This flag must be enabled to use it.\n\n`do_ocr=False` — OCR is disabled for native-text PDFs. Running Tesseract or another OCR engine on a PDF that already has an embedded text layer introduces character recognition errors and significantly increases processing time. OCR should only be enabled for scanned documents or for pages that contain image-only regions.\n\n**Step 3 — Hybrid Chunking (`HybridChunker`)**\n\n`HybridChunker` operates in four sequential stages on the parsed `DoclingDocument`:\n\n`LineBasedTokenChunker` that repeats the header row for each split segment. Text elements use the `semchunk` algorithm, which splits at the most semantically meaningful boundary available rather than at a fixed character or token count.`chunker.contextualize(chunk)` call prepends the full heading breadcrumb path to the chunk text before it is passed to the embedding model. A chunk from a section `3 > 3.2 > Access Control Requirements` produces an embedding that encodes both the section context and the content — significantly improving retrieval precision for queries that reference specific sections without quoting them verbatim.\n**Step 4 — Metadata Extraction (`meta_data`)**\n\nThe `meta_data` function derives the page number from the provenance data attached to each `doc_item` in the chunk's metadata. A chunk that spans multiple pages will have `prov` entries from each page; the function takes the minimum page number as the canonical page reference. This is more accurate than line-based page attribution because the `HybridChunker` can merge elements from adjacent pages into a single chunk.\n\nThe `heading` field concatenates the heading path as a `\" > \"`-delimited breadcrumb string. This is stored as vector store metadata and enables filtered retrieval: a query scoped to a specific section of a document can filter on the `heading` field rather than relying solely on vector similarity.\n\n**Step 5 — LangChain Document Construction**\n\nEach processed chunk is wrapped in a LangChain `Document` object with `page_content` set to the contextualized text (not `chunk.text`) and `metadata` containing the structured fields plus the sequential `chunk_index`. The `chunk_index` enables re-ranking by document position as a secondary signal when multiple chunks from the same document are retrieved with similar cosine scores.\n\nThe current implementation sets `do_ocr=False` globally. In a production corpus that contains a mix of native-text PDFs and scanned documents, this requires a pre-processing routing stage before the converter is invoked.\n\nA practical approach is to probe the document's text layer density before routing:\n\n``` python\nimport fitz  # PyMuPDF\n\ndef is_native_text_pdf(file_path: Path, sample_pages: int = 3) -> bool:\n    \"\"\"\n    Sample the first N pages to determine if the PDF has a native text layer.\n    Returns True for native-text PDFs, False for scanned/image-only PDFs.\n    \"\"\"\n    doc = fitz.open(str(file_path))\n    pages_to_check = min(sample_pages, len(doc))\n    total_chars = sum(\n        len(doc[i].get_text(\"text\").strip())\n        for i in range(pages_to_check)\n    )\n    doc.close()\n    return total_chars > 100  # threshold: fewer than 100 chars implies image-only pages\n```\n\nDocuments that fail the native-text check should be routed to a configuration with `do_ocr=True` and, depending on table density, `do_table_structure=True` with `TableFormerMode.ACCURATE`.\n\nThe `HeadingHierarchyOptions` signals apply in precedence order. For enterprise document corpora, the appropriate configuration depends on the document type:\n\n| Document Type | Recommended Signal Configuration | \n|---|---|\n| Regulatory PDF with bookmarked ToC | `use_bookmarks=True` (default), others as fallback | \n| Technical specifications with numbered sections | `use_numbering=True` ,`use_bookmarks=False` if ToC is absent or unreliable | \n| Legal agreements, free-form policy documents | `use_style=True` ,`use_font_style=True` , requires`generate_parsed_pages=True` | \n| Mixed corpus | All signals enabled (default when `enabled=True` ) | \n\nFor a banking regulatory corpus — MAS TRM guidelines, internal policy documents, audit frameworks — the mixed configuration is appropriate because the document types are heterogeneous.\n\n`contextualize()` vs `chunk.text` Decision\nThis is a consistently misunderstood detail in Docling-based RAG implementations. `chunk.text` contains only the raw text of the chunk's content elements. `chunker.contextualize(chunk)` prepends the heading breadcrumb path to that text.\n\nFor a chunk with headings `[\"3. Security Controls\", \"3.2 Access Management\"]` and body text `\"All privileged access must be logged...\"`, the outputs are:\n\n```\nchunk.text:\n\"All privileged access must be logged...\"\n\nchunker.contextualize(chunk):\n\"3. Security Controls\n3.2 Access Management\nAll privileged access must be logged...\"\n```\n\nThe embedding model encodes both versions. The contextualized version places the vector in a region of the embedding space that reflects the section semantics, not just the sentence semantics. A query for \"what are the access management logging requirements\" retrieves the contextualized chunk with significantly higher cosine similarity than the raw text chunk, because the heading tokens anchor the embedding to the correct domain.\n\nAlways pass `chunker.contextualize(chunk)` to the embedding model. Store `chunk.text` separately if you need the raw text for display purposes.\n\nThe metadata schema produced by `meta_data()` — `page`, `filetype`, `filename`, `heading`, `chunk_index` — serves a dual purpose. It is stored alongside each vector in the vector store and enables metadata-filtered retrieval.\n\nIn regulated environments, this has a direct compliance application. A query scoped to a specific document version (`filename` filter) or a specific section (`heading` filter) can be constrained at the retrieval layer rather than relying on the LLM to perform that scoping from the retrieved context. This reduces the surface area for cross-document content bleed, where a retrieved chunk from document version N-1 is used to answer a query that should be answered from version N.\n\n`generate_parsed_pages=True`\nKeeping the parsed PDF cells in memory (`generate_parsed_pages=True`) increases per-document memory consumption. For large documents — 200+ pages — this can add 300–500 MB of in-process memory overhead per concurrent parse operation. In a batch ingestion workload running on a fixed instance, the concurrency ceiling must be set to account for this.\n\nThe tradeoff is worth it for enterprise corpora where free-form documents without bookmarks are common. For corpora where all documents have reliable PDF outlines or numbered sections, `generate_parsed_pages` can be set to `False` to reduce memory pressure, with `use_style=False` in the `HeadingHierarchyOptions`.\n\n`do_table_structure=False` Tradeoff\nDisabling the TableFormer model eliminates the ML inference cost for table structure recognition, which is the most expensive single operation in the Docling pipeline for table-dense documents. For native-text PDFs, this is a reasonable tradeoff because cell relationships can be preserved through text-position inference.\n\nThe failure mode: if native-text PDFs contain tables that use borderless layouts or unconventional spacing, the text-position inference may still produce incorrect cell associations. A validation step that samples the parsed table output before indexing is strongly recommended for corpora with complex table structures.\n\nThe current implementation processes documents sequentially in a single process. For initial corpus ingestion at enterprise scale — thousands of documents — this does not meet practical time constraints. A production deployment should parallelise at the document level using a worker pool:\n\n``` python\nfrom concurrent.futures import ProcessPoolExecutor\nimport os\n\ndef process_documents_parallel(dir_path: str, max_workers: int = 4):\n    docs = load_docs(dir_path)\n    with ProcessPoolExecutor(max_workers=max_workers) as executor:\n        futures = {executor.submit(parse_and_index, doc_path): doc_path for doc_path in docs}\n        for future in futures:\n            try:\n                future.result()\n            except Exception as e:\n                print(f\"Failed: {futures[future]} — {e}\")\n```\n\nWorker count should be tuned based on available CPU cores and the memory overhead of `generate_parsed_pages=True`. On an `m6i.4xlarge` (16 vCPU, 64 GiB), a practical ceiling is 4–6 workers for large-document corpora.\n\nThe combination of layout-aware parsing, structure-driven chunking, and heading-enriched contextualization produces an index where retrieved chunks carry their structural position in the source document as part of the vector encoding. This has a measurable effect on retrieval precision for the query types that dominate enterprise RAG use cases: policy lookups, section-specific compliance checks, and cross-document comparisons.\n\nThe cost of this precision is paid once, at ingestion time. The alternative — fast parsing, naive chunking, no contextualization — produces an index that is cheap to build and unreliable to query. In a regulated environment, an unreliable RAG system is not a minor inconvenience. It is an audit risk.\n\nFix the ingestion layer. Everything downstream inherits its quality.", "url": "https://wpnews.pro/news/why-your-enterprise-rag-pipeline-is-failing-before-the-first-query-runs", "canonical_source": "https://dev.to/aws_sa_sg/why-your-enterprise-rag-pipeline-is-failing-before-the-first-query-runs-3lmb", "published_at": "2026-09-11 07:24:37+00:00", "updated_at": "2026-09-11 07:57:22.255541+00:00", "lang": "en", "topics": ["ai-tools", "natural-language-processing", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Docling", "LangChain", "PyPDF", "PDFMiner"], "alternates": {"html": "https://wpnews.pro/news/why-your-enterprise-rag-pipeline-is-failing-before-the-first-query-runs", "markdown": "https://wpnews.pro/news/why-your-enterprise-rag-pipeline-is-failing-before-the-first-query-runs.md", "text": "https://wpnews.pro/news/why-your-enterprise-rag-pipeline-is-failing-before-the-first-query-runs.txt", "jsonld": "https://wpnews.pro/news/why-your-enterprise-rag-pipeline-is-failing-before-the-first-query-runs.jsonld"}}