{"slug": "pdf-to-markdown-for-rag-the-4-problems-that-silently-ruin-your-retrieval", "title": "PDF to Markdown for RAG: the 4 problems that silently ruin your retrieval", "summary": "A developer at Extas Corp published a guide identifying four PDF parsing failures that degrade RAG retrieval: mixed two-column reading order, hyphenated line breaks that split keywords, tables flattened into unstructured text, and scanned pages with no text layer that silently vanish from the index. The writeup offers per-problem checks (searching output for '-\\n', comparing page counts to character counts) and a pymupdf4llm snippet for text-layer PDFs, alongside the company's hosted Apify converter that adds Tesseract OCR in seven languages and parallel URL processing, returning per-document status, pageCount, ocrPages, tableCount and charCount fields.", "body_md": "PDFs are the worst input format for a RAG pipeline. The text is stored as positioned glyphs, not as paragraphs, so a naive extraction gives you broken lines, columns mixed together and tables turned into word soup. Then your chunks are bad, your embeddings are bad, and the model answers from garbage.\n\nConverting each PDF to Markdown first fixes most of this: headings give you natural chunk borders, tables stay readable, and the text is clean enough to embed.\n\nHere are the four problems that hurt retrieval the most, how to check for each, and two ways to do the conversion.\n\nPapers and journals use two columns. A simple extractor reads straight across the page, so line 1 of the left column is glued to line 1 of the right column.\n\n**Check:** open the output of a two-column paper and read the second paragraph. If the sentences stop making sense halfway through a line, the columns are mixed.\n\nPDF lines end where the page ends, not where the sentence ends. You get `infor-` on one line and `mation` on the next, and a keyword search for \"information\" misses it.\n\n**Check:** search the output for `-\\n`. A good converter joins the lines of a paragraph and repairs the split words.\n\nA table extracted as plain text loses its columns: the model can no longer tell which number belongs to which header. Markdown tables keep that link and LLMs read them well.\n\n**Check:** find a table in the PDF and look for `|` rows in the output.\n\nA scanned page has no text layer at all. The extractor returns an empty string and that page silently disappears from your index. This is the most dangerous case, because nothing fails.\n\n**Check:** compare the number of pages with the number of characters. A 20-page document with 300 characters is a scan and needs OCR.\n\nFor a handful of files with a text layer, this is enough:\n\n``` python\nimport pymupdf4llm  # pip install pymupdf4llm\n\nmd = pymupdf4llm.to_markdown(\"paper.pdf\")\nopen(\"paper.md\", \"w\", encoding=\"utf-8\").write(md)\n```\n\nYou will need to add OCR yourself (Tesseract) for scanned pages, and a queue if you have thousands of files.\n\nDisclosure: this one is ours. We publish a hosted converter, [PDF to Markdown for LLM & RAG](https://apify.com/extas_corp/pdf-to-markdown), on Apify. It handles the four problems above: two-column reading order, re-joined paragraphs, Markdown tables, and OCR (Tesseract, 7 languages) for pages without a text layer. You give it URLs and it processes them in parallel.\n\n``` python\nfrom apify_client import ApifyClient  # pip install apify-client\n\nclient = ApifyClient(\"YOUR_APIFY_TOKEN\")\nrun = client.actor(\"extas_corp/pdf-to-markdown\").call(run_input={\n    \"pdfUrls\": [{\"url\": \"https://arxiv.org/pdf/1706.03762\"}],\n    \"extractTables\": True,\n    \"ocrScannedPages\": True,\n})\n\nfor doc in client.dataset(run[\"defaultDatasetId\"]).iterate_items():\n    print(doc[\"title\"], doc[\"pageCount\"], \"pages,\", doc[\"ocrPages\"], \"OCR pages\")\n    markdown = doc[\"markdown\"]\n```\n\nEach PDF comes back as one item with `status`, `title`, `pageCount`, `ocrPages`, `tableCount`, `charCount` and `markdown`. The `ocrPages` field answers problem 4 directly: you can see which documents were scans.\n\nIt is free during launch: you only pay Apify's own platform usage, which the free monthly credits cover for small jobs. Limits: 100 MB per PDF, no password-protected files, and very complex merged-cell tables may come out simplified.\n\nIf you only have one file, there is also a free browser version at [hackthehydra.com](https://hackthehydra.com/pdf-to-markdown/) with no account.\n\nOnce you have Markdown, split on headings so each chunk is one coherent section:\n\n``` python\nimport re\n\ndef chunk_by_heading(markdown, max_chars=4000):\n    sections = re.split(r\"\\n(?=#{1,3} )\", markdown)\n    for section in sections:\n        for i in range(0, len(section), max_chars):\n            yield section[i:i + max_chars]\n```\n\nKeep the heading inside the chunk: it gives the embedding the context that the body text often lacks.\n\nWhat breaks most often in your PDF pipeline? I am collecting hard cases to test against.", "url": "https://wpnews.pro/news/pdf-to-markdown-for-rag-the-4-problems-that-silently-ruin-your-retrieval", "canonical_source": "https://dev.to/extascorp/pdf-to-markdown-for-rag-the-4-problems-that-silently-ruin-your-retrieval-gid", "published_at": "2026-11-12 00:13:32+00:00", "updated_at": "2026-11-12 00:21:30.827183+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["Extas Corp", "Apify", "pymupdf4llm", "Tesseract", "hackthehydra.com", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/pdf-to-markdown-for-rag-the-4-problems-that-silently-ruin-your-retrieval", "markdown": "https://wpnews.pro/news/pdf-to-markdown-for-rag-the-4-problems-that-silently-ruin-your-retrieval.md", "text": "https://wpnews.pro/news/pdf-to-markdown-for-rag-the-4-problems-that-silently-ruin-your-retrieval.txt", "jsonld": "https://wpnews.pro/news/pdf-to-markdown-for-rag-the-4-problems-that-silently-ruin-your-retrieval.jsonld"}}