{"slug": "pdf-to-markdown-for-llm-pipelines-in-net-without-python-and-without-native", "title": "PDF to Markdown for LLM pipelines in .NET – without Python and without native libraries", "summary": "A developer released miPDFlib, an Apache-2.0 PDF library written in pure managed .NET code, which analyzes page layout to export structured Markdown and heading-based chunks for retrieval-augmented generation pipelines without Python or native dependencies. The library's PdfLayoutExport.ToMarkdown detects columns, headings, tables, headers and footers, avoiding the interleaved text and lost table structure that naive line-by-line extraction produces. The author notes that layout-aware extraction keeps tables and headings intact and yields cleaner chunks for embedding.", "body_md": "Most retrieval-augmented generation (RAG) pipelines start the same way: take a PDF, pull out the text, split it into\n\nchunks, embed the chunks. The first step looks trivial and is where many pipelines quietly lose quality. A language\n\nmodel can only answer from what the extractor hands it – and a typical extractor hands it text in the order the\n\ncharacters happen to sit on the page.\n\nThis article shows what goes wrong, and how to get structured Markdown and clean chunks out of PDFs in a .NET service\n\nwith [miPDFlib](https://github.com/Mibuw/miPDFlib), an Apache-2.0 PDF library in pure managed code. (Disclosure: I\n\nwrote it.)\n\nTake an ordinary report page: a heading, two columns of text, a table, a running head at the top and a page number at\n\nthe bottom. Read line by line, as most extractors do by default, it comes out like this:\n\n```\nWater Utility – Quarterly Report\n2 Operations\nPumps in the northern plant ran at Energy use per cubic metre fell\n92 percent of their rated capacity for the fourth quarter in a row. The\nduring the second quarter. The two largest saving came from the new\nolder units were replaced in May, variable-speed drives, which adjust\nwhich cut maintenance hours by the pump speed to the demand in\nalmost a third. the network.\nPlant Capacity Energy (kWh/m³)\nNorth 92 % 0.41\nSouth 87 % 0.45\nPage 1\n```\n\nEvery word is there, so a quick check passes. But the sentences of the two columns are interleaved (\"ran at Energy use\n\nper cubic metre fell\"), the table has lost its columns, and the running head and page number sit in the middle of the\n\ncontent – on every page. Chunk this and embed it, and the retriever finds chunks that mix two topics, the model reads\n\nnumbers without knowing which column they belong to, and \"Page 1\" ends up in the vector index a few hundred times.\n\nThe structure is there on the page; it just has to be read. `PdfLayoutExport.ToMarkdown` analyses the layout first –\n\ncolumns, blocks, headings, lists, tables, headers and footers – and writes what it finds:\n\n``` js\nusing MiPdf;\nusing MiPdf.Text;\n\nusing var document = PdfDocument.Open(\"report.pdf\");\nstring markdown = PdfLayoutExport.ToMarkdown(document);\n# 2 Operations\n\nPumps in the northern plant ran at 92 percent of their rated capacity during the second quarter. The two older units were replaced in May, which cut maintenance hours by almost a third.\n\nEnergy use per cubic metre fell for the fourth quarter in a row. The largest saving came from the new variable-speed drives, which adjust the pump speed to the demand in the network.\n\n| Plant | Capacity | Energy (kWh/m³) |\n| --- | --- | --- |\n| North | 92 % | 0.41 |\n| South | 87 % | 0.45 |\n\n## Appendix\n```\n\nWhat happened on the way:\n\nMarkdown is a good intermediate format for language models: headings and tables survive, and it costs far fewer\n\ntokens than HTML.\n\nFor retrieval you usually do not want one long Markdown string but chunks that each belong to one section. The layout\n\nanalysis behind the export is available directly, so chunking by heading takes a few lines:\n\n``` js\nvar chunks = new List<(string Heading, string Text)>();\nstring heading = \"\";\nvar text = new System.Text.StringBuilder();\n\nforeach (var page in PdfLayoutAnalyzer.Analyze(document))\n    foreach (var block in page.Blocks)\n    {\n        switch (block.Kind)\n        {\n            case LayoutBlockKind.Header or LayoutBlockKind.Footer:\n                continue;                                         // running heads and page numbers\n            case LayoutBlockKind.Heading:\n                if (text.Length > 0) chunks.Add((heading, text.ToString().Trim()));\n                heading = block.Text;\n                text.Clear();\n                break;\n            case LayoutBlockKind.Table:\n                foreach (var row in block.Table!.Cells.GroupBy(c => c.Row))\n                    text.AppendLine(string.Join(\" | \", row.Select(c => c.Text)));\n                break;\n            default:\n                text.AppendLine(block.Text.ReplaceLineEndings(\" \"));\n                break;\n        }\n    }\nif (text.Length > 0) chunks.Add((heading, text.ToString().Trim()));\n```\n\nFor the sample report this gives one chunk under \"2 Operations\" with both paragraphs and the table rows – the heading\n\ncan go into the chunk's metadata or be prepended to the text before embedding. Each block also has its `Bounds`, the\n\npage number and its lines with word positions, so an answer can point back to the exact place in the PDF.\n\nLayout analysis is heuristic, so I measured it rather than trusting a few examples:\n\nThe same analysis also exports to hOCR, ALTO and PAGE XML, if your pipeline or an archive expects those formats.\n\nBetter to know them before relying on it:\n\nIn Python, tools such as Docling or pymupdf4llm do this job, and they are good at it. For a .NET service, though, a\n\nPython sidecar means a second runtime, a second deployment and a process boundary – and PyMuPDF is AGPL or commercial.\n\nmiPDFlib runs in-process, has no native dependencies (so it works the same on Windows, Linux and in containers), and is Apache-2.0, so it can go into closed commercial products. The same library also creates, fills,\n\nsigns and archives PDFs, which helps when the documents that come out of the pipeline need to go back into a PDF.\n\n```\ndotnet add package miPDFlib\n```\n\nThe [text extraction guide](https://github.com/Mibuw/miPDFlib/blob/main/docs/guide/text-extraction.md) covers the\n\noptions (columns, Markdown export settings, letters with fonts and colors, vector paths). If a PDF of yours comes out\n\nwrong, an issue with the file – or a synthetic copy of its layout – is the fastest way to get it fixed.", "url": "https://wpnews.pro/news/pdf-to-markdown-for-llm-pipelines-in-net-without-python-and-without-native", "canonical_source": "https://dev.to/wolfgang_mitterbucher_dd5/pdf-to-markdown-for-llm-pipelines-in-net-without-python-and-without-native-libraries-436b", "published_at": "2026-11-16 14:38:29+00:00", "updated_at": "2026-11-16 14:51:35.091327+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "large-language-models", "ai-infrastructure"], "entities": ["miPDFlib", "Mibuw", ".NET"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/pdf-to-markdown-for-llm-pipelines-in-net-without-python-and-without-native", "markdown": "https://wpnews.pro/news/pdf-to-markdown-for-llm-pipelines-in-net-without-python-and-without-native.md", "text": "https://wpnews.pro/news/pdf-to-markdown-for-llm-pipelines-in-net-without-python-and-without-native.txt", "jsonld": "https://wpnews.pro/news/pdf-to-markdown-for-llm-pipelines-in-net-without-python-and-without-native.jsonld"}}