Building RAG pipelines, the step that kept letting me down was the very first one: turning a PDF into text. Two runs of the same file could give slightly different output, tables collapsed into garbled lines, and Chinese PDFs were mostly a coin flip. When your chunks change between runs, so do your embeddings and your eval numbers — and you stop trusting the index.
So I wrote pdfmuse: a PDF/DOCX parser with one goal — be a precise,
This post is partly about how it works, and partly about the humbling bug I only found by dogfooding it on real data. That second part is the one worth your time.
NeedsOcr
warning, and OCR/layout inference are a pluggable backend.You can try it in your browser (WASM, nothing gets uploaded — your file never leaves the tab): ** https://casperkwok.github.io/pdfmuse/** — drag a PDF and watch it come apart into text-with-coordinates, tables, and Markdown.
pip install pdfmuse # or: npm i @pdfmuse/node / cargo add pdfmuse-core
python
import pdfmuse
data = open("report.pdf", "rb").read()
text = pdfmuse.to_text(data) # plain reading-order text
md = pdfmuse.to_markdown(data) # headings + tables
doc = pdfmuse.parse(data) # full IR: chars/blocks with exact bboxes
I had a test suite. Snapshot tests, a fuzzer, cross-binding parity, a CJK suite — all green. I ran it on a handful of sample resumes and the output looked great. I was ready to call the extraction layer done.
Then I ran it against 80 real resumes pulled from a product database (I build a resume tool on the side). The plan was a boring "shadow comparison" vs PyMuPDF to confirm parity before switching anything.
8 of the 80 came back with zero characters. Not garbled — empty. And it failed silently: no warning, no error, just an empty document. PyMuPDF read all 8 fine.
That's a 10% total-failure rate, invisible to my entire test suite, because my test files happened to be clean.
The 8 failing files all shared traits:
ToUnicode
maps. So it wasn't a missing-CMap problem.There it was. Canva (and Chrome's PDFium, and plenty of design tools) draw the page's text inside a form XObject, invoked by a Do
operator. My content-stream interpreter didn't handle Do
at all — and, worse, a lazy- optimization I'd added skipped /XObject
bodies for speed. So the actual page text was never even parsed. On a clean PDF where text lives in the page content stream directly, everything worked. On a Canva export, I silently got nothing.
The fix was to make the interpreter recurse into form XObjects (decode the stream, apply its /Matrix
, build its own font map, recurse with a depth guard) and stop skipping them in the . After that:
zero-char failures:
8/80 → 0/80. The 10 form-XObject files now match PyMuPDF ~100%.
The lesson isn't "I wrote a bug." It's that a green test suite on synthetic fixtures told me nothing about the real distribution. Canva is one of the most popular resume builders on earth; "10% of resumes" is not an edge case. I only found it because I ran it on real data before trusting it. If you're swapping any parser into a pipeline, do the shadow-comparison on your corpus first — the demo files lie.
While I was in there: I'd been claiming "4× faster than PyMuPDF." For a Python user, that turned out to be… not quite true. The Rust core parses in ~1.3 ms, but if you json.loads
the full intermediate representation (every char with its bbox) back into Python, that deserialization costs ~3.5 ms and eats the win — you end up roughly on par with PyMuPDF.
The fix was obvious once measured: if you only want text, don't materialize the whole IR. to_text()
/ to_markdown()
now return a string straight from Rust, and the Python text path is back to full core speed. Measure the path your users actually take, not the one that flatters your benchmark. (Profiling that same corpus later turned up two more hotspots — a table detector that mistook a dense figure for a 1000-cell table, and a form XObject re-parsed 18,000 times on a single page — and fixing those is what got it to winning every file.)
There are drop-in s for the three big frameworks, all emitting chunks with heading_path
bbox
metadata (which section, where on the page):
from langchain_pdfmuse import Pdfmuse
docs = Pdfmuse("report.pdf", mode="elements").load()
from llama_index.readers.pdfmuse import PdfmuseReader
docs = PdfmuseReader(mode="elements").load_data("report.pdf")
from pdfmuse_haystack import PdfmuseConverter
docs = PdfmuseConverter(mode="markdown").run(sources=["report.pdf"])["documents"]
NeedsOcr
; OCR is a backend, kept out of the deterministic core.If you try it, I'd genuinely love the files it gets wrong — that feedback is worth more than stars. Thanks for reading.