{"slug": "i-tried-to-prove-doclang-beats-markdown-for-pdf-llm-the-data-said-otherwise", "title": "I Tried to Prove DocLang Beats Markdown for PDF→LLM. The Data Said Otherwise.", "summary": "A developer's experiment comparing DocLang XML markup to Markdown for feeding PDFs to LLMs found that all three representations produced identical answers—100% accuracy on facts and 87.5% on structure questions—when the whole document was in context, with Markdown being fastest and cheapest per call. The test used a 15-page construction RFP parsed once into a DoclingDocument and serialized by docling-core's first-party exporters, with gpt-5.4-mini via Azure OpenAI as the model. DocLang's expected advantage in locating answers did not materialize, and its output lacked hierarchical structure due to the layout model's failure to recover exhibit groupings.", "body_md": "If you’re building anything that feeds PDFs to an LLM — RAG, document Q&A, contract review — you hit the same fork early: **what text representation do you hand the model?**\n\nThe default answer is Markdown. Parse the PDF, export_to_markdown(), stuff it in the context window. It's cheap, it's readable, every tokenizer likes it.\n\nThe newer answer is [**DocLang**](https://www.doclang.ai/) — an XML-based, “AI-native” markup for unstructured documents. It keeps headings, lists, tables, reading order, and even bounding-box coordinates, all in a schema designed to be read directly by a language model. On paper it should dominate Markdown: same content, but *structured*.\n\nI went in wanting DocLang to win. I built the experiment to show it. Here’s what actually happened.\n\n**Source document:** a real “General Conditions and Fee Request for Proposal” — a 15-page construction RFP with a project-info section and six exhibits (A–F). It’s a scanned-style PDF, the kind you actually get in the wild.\n\n**The critical design choice — one parse, many renderings.** My first attempt at this experiment had a fatal flaw: I compared *my hand-written* PDF→DocLang converter against Docling’s *mature, years-tuned* Markdown exporter. That’s not “Markdown vs DocLang.” That’s “my weekend project vs a library with hundreds of contributors.”\n\nSo I rebuilt it. The PDF is parsed **once** into a DoclingDocument. Then every representation is produced by **docling-core's own first-party serializers** on that single shared object:\n\nNo hand-written converter anywhere in the loop. Any difference between the arms is now attributable to **the notation**, not to converter maturity.\n\n**The model:** gpt-5.4-mini via Azure OpenAI, JSON-mode, whole document in the system prompt.\n\n**The questions:** two sets, each graded by an LLM judge against a gold answer *and* a gold location (“which section/exhibit is this in?”).\n\nEverything is in the repo: doclang-impl/experiment/ — prepare_inputs.py builds the arms, run_ci.py runs the comparison.\n\nWith the whole document in context, **all three representations produced the same answers** — 100% on facts, 87.5% on structure questions. The one structure-set miss (“is there a substantive Exhibit G?”) was shared by every arm: it’s a text-extraction gap, not a format gap.\n\nThe notation did not change *whether the model could find and state the answer*. Not for flat facts, not for “which exhibit is this in,” not for cross-references.\n\nHere’s where it gets expensive:\n\nThat’s a per-call cost you pay on every single query, for zero measured accuracy gain on this document.\n\nNo surprise, but worth showing: more input tokens → slower first token. Markdown was fastest in every cell. The scatter makes it plain — the three arms form three vertical bands by token count, and latency climbs with them.\n\nThis is the one I expected DocLang to win — “which section is this answer in?” is exactly what structure should help with.\n\nThe per-question grid shows where the misses land:\n\nThe answer squares are almost all green. The red dots — wrong cited location — **cluster on “DocLang (no locations).”**\n\nHere’s the thing I didn’t expect. When I looked at what export_to_doclang() actually produced:\n\n```\n Wind Point Partners (WPP) ... General Conditions and Fee Request for Proposal ... November 21, 2025 \n```\n\nEvery heading is level=\"2\". There are no containers wrapping the exhibits. The running page header (\"Wind Point Partners (WPP)\") is repeated ~12 times as a real element.\n\n**This isn’t a serializer bug.** Docling’s layout model parsed this scanned PDF and *didn’t recover the exhibit hierarchy* — so there’s no nested structure in the DoclingDocument for export_to_doclang() to serialize. The DocLang output faithfully reflects a flat parse.\n\nWhich means: **on this document, DocLang is Markdown’s exact information content, wrapped in XML that costs 1.5–2.6× the tokens.** The structural advantage I was counting on requires structure the parser never extracted.\n\nMy first write-up compared two hand-rolled DocLang converters against Docling’s Markdown exporter and concluded “flat DocLang is worse, my structured version closes the gap.” That comparison was confounded — I was measuring my converter, not the format.\n\nRemoving the confound, the conclusion held and got sharper: **the official DocLang serializer is also flat for this PDF, ties on answers, and loses on cost and latency.** The format didn’t fail. The *parser* didn’t give the format anything to work with.\n\nThis experiment stress-tests exactly one regime: **one small document, full context, single-doc Q&A.** In that regime, Markdown wins — cheaper, faster, equal accuracy. Full stop.\n\nDocLang’s value is in the regimes I *didn’t* test:\n\n**1. When the parser actually recovers structure.** A born-digital PDF with a real outline, or a pipeline that post-processes the DoclingDocument tree to rebuild sections before serializing. Then export_to_doclang() carries nesting that Markdown flattens away.\n\n**2. Retrieval, when the corpus doesn’t fit in context.** DocLang gives you addressable units — , label — as natural chunk boundaries. Split Markdown on ## and your running page headers bleed into every chunk. This is DocLang's home turf and it's completely untested here.\n\n**3. Round-trip fidelity.** Tables as , formulas, forms as , checkboxes, reading order, RTL — Markdown drops all of it. And if you need to cite *\"page 12, this bounding box\"* back to the source pixels, the elements are the entire point. That 2.6× token cost is buying you provenance.\n\n**4. Scale.** ~46–156% overhead per call here. Across a retrieval pipeline with clean unit boundaries, that math can flip.\n\n[I Tried to Prove DocLang Beats Markdown for PDF→LLM. The Data Said Otherwise.](https://pub.towardsai.net/i-tried-to-prove-doclang-beats-markdown-for-pdf-llm-the-data-said-otherwise-19d256cbaa00) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/i-tried-to-prove-doclang-beats-markdown-for-pdf-llm-the-data-said-otherwise", "canonical_source": "https://pub.towardsai.net/i-tried-to-prove-doclang-beats-markdown-for-pdf-llm-the-data-said-otherwise-19d256cbaa00?source=rss----98111c9905da---4", "published_at": "2026-09-08 22:01:01+00:00", "updated_at": "2026-09-08 22:28:54.939565+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-research"], "entities": ["DocLang", "Docling", "docling-core", "gpt-5.4-mini", "Azure OpenAI", "Wind Point Partners"], "alternates": {"html": "https://wpnews.pro/news/i-tried-to-prove-doclang-beats-markdown-for-pdf-llm-the-data-said-otherwise", "markdown": "https://wpnews.pro/news/i-tried-to-prove-doclang-beats-markdown-for-pdf-llm-the-data-said-otherwise.md", "text": "https://wpnews.pro/news/i-tried-to-prove-doclang-beats-markdown-for-pdf-llm-the-data-said-otherwise.txt", "jsonld": "https://wpnews.pro/news/i-tried-to-prove-doclang-beats-markdown-for-pdf-llm-the-data-said-otherwise.jsonld"}}