# pdfium and pypdf return different text from the same PDF — four mismatches to know before you feed PDFs to an LLM > Source: > Published: 2026-09-23 13:14:33+00:00 I assumed that "extract the text from this PDF" gives you the same characters no matter which library you use. It does not. **Feed the same PDF to pdfium and to pypdf, and you get characters that only one of them returns, and characters that one of them silently replaces with a different sentence.** Worse, neither output necessarily matches what a human sees on screen. The page says Hello world; the extracted text says something else. The page shows nothing; the extracted text has an extra line. All of it is within the PDF specification, and neither library is wrong. Most pipelines that hand PDFs to an LLM take the output of one extraction library and treat it as "the text of the document". This article shows four cases where that assumption breaks, each with a minimal PDF you can build yourself and the actual output of both libraries. The second half is about how I turned the mismatch into a detection signal for hidden text. **TL;DR** `0 Tz` (zero horizontal scaling) does not exist as far as pdfium's text extraction is concerned. pypdf reads it.`/ActualText` makes pdfium return the replacement string `/ToUnicode` table makes "A" on screen come out as "B" in the text. Mappings to invisible Unicode tag characters pass through both libraries untouched.`Do` is read by neither. pypdf's `extract_xform_text` gets it. My previous post ([I built a tool that finds "invisible text" hidden in PDFs](https://dev.to/okinawasoftware/i-built-a-tool-that-finds-invisible-text-hidden-in-pdfs-white-text-tiny-fonts-and-the-5982)) was about text that *is* extracted but cannot be seen, found by rendering the page with and without its text. This one is about a step earlier: the extracted text itself being different. You do not need to have read the first one. Code in this article was run with Python 3.12 / pypdfium2 5.12.1 (pdfium build 7947) / pypdf 6.15.0. The full reproduction script is at the end. If you generate test PDFs with a library, you lose control over *how* the text is hidden. So I write raw PDF syntax, byte by byte. The builder is about 60 lines: stack objects, write the xref table by hand. ``` python class PDF: def __init__(self): self.objs = [None] # object numbers start at 1 def add(self, body): self.objs.append(body.encode("latin-1") if isinstance(body, str) else body) return len(self.objs) - 1 def stream(self, dic, data): data = data.encode("latin-1") if isinstance(data, str) else data return self.add(b"<< " + dic.encode() + b" /Length " + str(len(data)).encode() + b" >>\nstream\n" + data + b"\nendstream") def build(self, root): out = bytearray(b"%PDF-1.7\n") offsets = [0] * len(self.objs) for i in range(1, len(self.objs)): offsets[i] = len(out) out += f"{i} 0 obj\n".encode() + self.objs[i] + b"\nendobj\n" xref = len(out) out += f"xref\n0 {len(self.objs)}\n".encode() + b"0000000000 65535 f \n" for i in range(1, len(self.objs)): out += f"{offsets[i]:010d} 00000 n \n".encode() out += (f"trailer\n<< /Size {len(self.objs)} /Root {root} 0 R >>\n" f"startxref\n{xref}\n%%EOF\n").encode() return bytes(out) ``` Every page has one font (Helvetica, WinAnsiEncoding) and one visible line. Each experiment adds one more line. ``` VISIBLE = "BT /F1 14 Tf 0 0 0 rg 60 780 Td (This is the visible body text.) Tj ET\n" ``` Two readers, both using the most ordinary "give me all the text on this page" API: ``` python import pypdfium2 as pdfium import pypdf def pdfium_text(path): return pdfium.PdfDocument(path)[0].get_textpage().get_text_range() def pypdf_text(path): return pypdf.PdfReader(path).pages[0].extract_text() ``` `Tz` is horizontal scaling. At `0 Tz` every glyph has zero width and nothing appears on screen. The height is normal, so a "tiny font" check does not fire either. ``` BT /F1 12 Tf 0 Tz 0 0 0 rg 60 700 Td (SECRET-1 zero width) Tj ET pdfium : 'This is the visible body text.' pypdf : 'This is the visible body text.\nSECRET-1 zero width' ``` Not one character of that line makes it into pdfium's text page (`FPDFText`). It is not even counted by `count_chars()`. pdfium builds its text page from layout, and a character with zero width apparently does not exist. pypdf just walks the `Tj` operators in the content stream and returns what it finds. For what it's worth, at `1 Tz` (1%) pdfium does return the text. The character box is then 0.08 pt wide, and a 12 pt letter is drawn as a hairline. Unreadable to a person, present in both extractors. `/ActualText` is a marked-content property that says "for text extraction, use this string instead of the glyphs in this span". It has legitimate uses: turning the ligature fi into "fi", rejoining a hyphenated word. Word uses it when exporting PDFs. ``` /Span << /ActualText (SECRET-2 replaced text) >> BDC BT /F1 12 Tf 0 0 0 rg 60 700 Td (Hello world) Tj ET EMC ``` The page draws Hello world. ``` pdfium : 'This is the visible body text.\r\nSECRET-2 replaced text' pypdf : 'This is the visible body text.\nHello world' ``` pdfium follows the spec and returns the replacement. Hello world is nowhere in its output. pypdf ignores `/ActualText` and returns the glyphs as drawn. This is not a question of which one is right. The spec intends the replacement, and pdfium honours it. The point is that **the string a person verified on screen and the string a pdfium-based consumer receives (copy from Chrome's viewer, a pypdfium2 pipeline) can be two different things.** And pdfium assigns the replacement characters the coordinates of the original glyphs. My "render with and without text" check from the previous post removes text at those coordinates, sees Hello world disappear, and concludes the text is visible. The replacement string sailed straight through as "visible text". A font's `/ToUnicode` entry is a table from character codes to Unicode. Extractors trust it. Rewrite the table and the glyphs stay the same while the extracted characters change. I put three tricks into one table: ``` php 3 beginbfchar <41> % 'A' -> U+E0041 (TAG LATIN CAPITAL LETTER A) <42> <00490067006E006F00720065> % 'B' -> "Ignore" (one glyph, six characters) <43> <200B> % 'C' -> U+200B (ZERO WIDTH SPACE) endbfchar BT /F1 12 Tf 0 0 0 rg 60 700 Td (ABC abc) Tj ET ``` The page draws ABC abc. ``` pdfium : 'This is the visible body text.\r\n\U000e0041Ignore\u200b abc' pypdf : 'This is the visible body text.\n\U000e0041Ignore\u200b abc' ``` This time both agree. A became a tag character, B became "Ignore", C became a zero-width space. If you `print()` the result it looks like `Ignore abc`, because tag characters and zero-width spaces are invisible in terminals and editors alike. Unicode tag characters (U+E0000 to U+E007F) are an invisible copy of ASCII. Humans and editors do not see them, but an LLM tokenizer receives them as letters, which is why they are used to smuggle instructions (the technique is usually called ASCII smuggling). Being able to plant them through a font table means **a PDF whose page shows nothing but ordinary English can hand the extractor a different English sentence.** The one-glyph-to-six-characters expansion deserves a look too. Ligature expansion (fi to fi, ffi to ffi) is legitimate, so libraries accept multi-character mappings. The PDF spec sets no length limit; pypdf caps a single mapping at 512 bytes (256 UTF-16 characters) as a safety valve, which is still enough for a sentence or two per glyph. One detail: pdfium's `FPDFText_GetUnicode` returns UTF-16 code units, so U+E0041 comes back as two values, `0xDB40, 0xDC41`. If you classify characters one at a time, you have to combine surrogate pairs yourself. pypdf returns a Python `str`, already combined. ``` pdfium chars (code points >0x7e): ['0xdb40', '0xdc41', '0x200b'] pypdf chars (code points >0x7e): ['0xe0041', '0x200b'] ``` A Form XObject is a bundle of drawing operators, drawn when the page content invokes it with `/X0 Do`. If nothing invokes it, nothing is drawn. You can register it in the page's resource dictionary and simply never call it. ``` %% inside the XObject BT /F1 12 Tf 0 0 0 rg 60 500 Td (SECRET-4 inside unused xobject) Tj ET %% registered in the page resources, but the content never says /X0 Do /Resources << /Font << /F1 5 0 R >> /XObject << /X0 6 0 R >> >> pdfium : 'This is the visible body text.' pypdf : 'This is the visible body text.' ``` Neither reads it. Both follow "the text that appears when the page is drawn", and an object that is never drawn is never visited. That is correct behaviour, and since nothing appears on screen it is not even hidden text in the naive sense. But it is still in the file, and it can be pulled out with a different tool. pypdf has an API that reads a stream directly: ``` r = pypdf.PdfReader(path) page = r.pages[0] xo = page["/Resources"]["/XObject"]["/X0"].get_object() page.extract_xform_text(xo) # -> 'SECRET-4 inside unused xobject' ``` "Invisible to the extractor but still in the file" is the classic shape of leftover data. Editing tools can leave old objects in the resource dictionary and remove only the reference. | Trick | Human eye | pdfium | pypdf | Note | |---|---|---|---|---| | `0 Tz` | nothing | no characters | reads it | zero-width chars never enter pdfium's text page | | `/ActualText` | the glyphs | replacement string | the glyphs | the spec intends the replacement; coordinates are those of the original glyphs | | rewritten `/ToUnicode` | the glyphs | per the table | per the table | tag characters, zero-width, one-to-many mappings all pass through | | unused XObject | nothing | not read | not read | `extract_xform_text` reads it individually | Four rows, and not a single one where the human, pdfium and pypdf all see the same characters. **I did not test PyMuPDF, pdfminer.six or pdf.js.** This article covers exactly two readers. Swapping the two reader functions in the script at the end is all it takes to check another library; if you do, please leave a comment. The one-sentence explanation: **text extraction is a by-product of rendering, and rendering is optimized for showing things, not for reporting them.** pdfium lays out the page, drops zero-width characters and honours replacement strings. pypdf reads the operator stream. Both are valid choices under the spec, and nothing anywhere guarantees "this is the text you should give the model". The rest is about what I did with this. I build a tool that finds invisible text in PDFs (the previous post), and the four cases above were found as things that slipped past it. The fix was not to replace pdfium with pypdf. pypdf ignores `/ActualText` and cannot do the render comparison. **The right use is to run both and look at the difference.** The basic rule is "report words that are in pypdf's output and not in pdfium's". Compared naively, that produces nothing but false positives. Here is what I had to do after running it over 29 real Japanese business PDFs (quotes, delivery notes, vehicle registrations). **Normalize.** Strip whitespace, format characters, punctuation and symbols, and apply compatibility normalization (NFKC) so full-width/half-width forms and ligatures line up. pypdf and pdfium break lines in different places and treat punctuation differently; none of that should count. ``` python import unicodedata def norm(s): s = unicodedata.normalize("NFKC", s or "") return "".join(c for c in s if not unicodedata.category(c).startswith(("Z", "C", "P", "S"))) ``` **Split words into CJK runs and non-CJK runs, and match them loosely.** pypdf tends to concatenate adjacent table cells. Two Japanese headers, "vehicle type" and "registration number", come back as one run with the first character missing; "13th" and "issue date" become one word. Judging those as "missing from pdfium" is wrong every time. For Japanese I accept a run if at least half of its 2-character fragments appear in the pdfium text; for everything else, half of its 4-character fragments. ``` python def segment_present(seg, target): cjk = is_cjk(seg[0]) # kana / kanji / half-width katakana range check k = 2 if cjk else 4 if len(seg) < (3 if cjk else 4) or seg in target: return True if not cjk and seg.isdigit(): return True # digit runs get glued to the neighbouring cell's number if len(seg) < k + 1: return False grams = [seg[i:i + k] for i in range(len(seg) - k + 1)] return sum(g in target for g in grams) >= len(grams) * 0.5 ``` This leniency has a price: a short Japanese hidden word of 3 or 4 characters (say "confidential") can be missed if its two-character pieces occur in the visible text. I documented that as a known limit. Long text does not slip through, and hidden instructions are usually sentences. **Skip `/ActualText` that matches what is drawn.** Word attaches `/ActualText` to ligatures and soft hyphens. If the replacement string is contained in the glyph-side text (pypdf's output), the replacement is harmless. Only the ones that differ get reported. **Inspect `/ToUnicode` tables directly.** The difference cannot see this case (both libraries return the same thing), so I read the font dictionary. Three checks: a code that expands to 4 or more characters (ligatures stop at 3); any mapping to an invisible character (tags, zero-width, bidi controls); and, for simple fonts with a standard encoding such as WinAnsi, an alphanumeric mismatch between what the encoding says the glyph is and what the table says the character is. Embedded subset fonts with private glyph names (`g123` and the like) have no ground truth for "what glyph is drawn", so the third check is skipped for them. **Decode tag characters and show the message.** U+E0020 to U+E007E map back to ASCII with `chr(cp - 0xE0000)`. "9 invisible characters" is a fact; "decoded, they say SECRET-44" is something the reader can act on. With all of that in place I ran the check over 29 real PDFs (mostly Japanese business forms) plus 17 synthetic samples. Exactly one finding came out of the new checks. A delivery note produced by Excel had, in its structure tree (the `/Alt` of a tagged figure), the string `C:\Documents and Settings\\My Documents\My Pictures\untitled.gif`. The alt text of an image contained a path from the author's PC. That is not a false positive; that is exactly the kind of thing the tool exists to find. The previous post's render comparison removed text objects from the page and re-rendered. pdfium's `FPDFPage_RemoveObject` only works on top-level page objects, so text inside a Form XObject could not be removed and its area was excluded from the comparison. Text inside an XObject, covered with a white rectangle, walked straight through that gap. The new version switches the text rendering mode to 3 (invisible) instead of deleting: ``` for o in page.get_objects(max_depth=16): # descends into XObjects if o.type == R.FPDF_PAGEOBJ_TEXT: mode = R.FPDFTextObj_GetTextRenderMode(o.raw) want = (R.FPDF_TEXTRENDERMODE_CLIP # modes 4-7: keep only the clip if mode >= R.FPDF_TEXTRENDERMODE_FILL_CLIP else R.FPDF_TEXTRENDERMODE_INVISIBLE) R.FPDFTextObj_SetTextRenderMode(o.raw, want) img_without_text = page.render(scale=2).to_pil() # no gen_content() needed ``` pdfium renders from the in-memory object state, so the change takes effect without regenerating the content stream (no `gen_content()` call). It works at any depth, and hidden text inside XObjects is now part of the comparison. For the record: Whether or not you build tools, if you own a pipeline that hands PDFs to a model: `unicodedata.category(c) == "Cf"` and the U+E0000 to U+E007F range catches tag characters and zero-width injection. It is a few lines. Builds the four PDFs, reads them with both libraries and prints the outputs side by side. About 150 lines including the builder; the only dependencies are pypdfium2 and pypdf. article7_experiments.py (full) ``` python # -*- coding: utf-8 -*- import os import tempfile import pypdf import pypdfium2 as pdfium import pypdfium2.raw as R OUT = tempfile.mkdtemp(prefix="article7_") def _b(x): return x.encode("latin-1") if isinstance(x, str) else x class PDF: def __init__(self): self.objs = [None] def add(self, body): self.objs.append(_b(body)) return len(self.objs) - 1 def reserve(self): self.objs.append(b"") return len(self.objs) - 1 def set(self, num, body): self.objs[num] = _b(body) def stream(self, dic, data): data = _b(data) return self.add(b"<< " + _b(dic).strip() + b" /Length " + str(len(data)).encode() + b" >>\nstream\n" + data + b"\nendstream") def build(self, root): out = bytearray(b"%PDF-1.7\n%\xe2\xe3\xcf\xd3\n") offsets = [0] * len(self.objs) for i in range(1, len(self.objs)): offsets[i] = len(out) out += f"{i} 0 obj\n".encode() + self.objs[i] + b"\nendobj\n" xref = len(out) n = len(self.objs) out += f"xref\n0 {n}\n".encode() + b"0000000000 65535 f \n" for i in range(1, n): out += f"{offsets[i]:010d} 00000 n \n".encode() out += (f"trailer\n<< /Size {n} /Root {root} 0 R >>\n" f"startxref\n{xref}\n%%EOF\n").encode() return bytes(out) def make_pdf(name, content, font_extra="", resources_extra="", hook=None): p = PDF() extra = hook(p) if hook else {} font = p.add("<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica " "/Encoding /WinAnsiEncoding " + font_extra.format(**extra) + " >>") cs = p.stream("", content) pages = p.reserve() page = p.add(f"<< /Type /Page /Parent {pages} 0 R /MediaBox [0 0 595 842] " f"/Resources << /Font << /F1 {font} 0 R >> " f"{resources_extra.format(**extra)} >> /Contents {cs} 0 R >>") p.set(pages, f"<< /Type /Pages /Kids [{page} 0 R] /Count 1 >>") cat = p.add(f"<< /Type /Catalog /Pages {pages} 0 R >>") path = os.path.join(OUT, name) with open(path, "wb") as f: f.write(p.build(cat)) return path VISIBLE = "BT /F1 14 Tf 0 0 0 rg 60 780 Td (This is the visible body text.) Tj ET\n" def pdfium_text(path): doc = pdfium.PdfDocument(path) return doc[0].get_textpage().get_text_range() def pdfium_chars(path): doc = pdfium.PdfDocument(path) tp = doc[0].get_textpage() return "".join(chr(R.FPDFText_GetUnicode(tp.raw, i) or 0) for i in range(tp.count_chars())) def pypdf_text(path): return pypdf.PdfReader(path).pages[0].extract_text() def show(title, path): print(f"\n=== {title} ===") print("pdfium :", repr(pdfium_text(path))) print("pypdf :", repr(pypdf_text(path))) # 1. zero-width text (0 Tz) p1 = make_pdf("1_tz_zero.pdf", VISIBLE + "BT /F1 12 Tf 0 Tz 0 0 0 rg 60 700 Td (SECRET-1 zero width) Tj ET\n") show("1. 0 Tz (zero-width text)", p1) # 2. ActualText p2 = make_pdf("2_actualtext.pdf", VISIBLE + "/Span << /ActualText (SECRET-2 replaced text) >> BDC " "BT /F1 12 Tf 0 0 0 rg 60 700 Td (Hello world) Tj ET EMC\n") show("2. /ActualText", p2) # 3. rewritten ToUnicode cmap = ("/CIDInit /ProcSet findresource begin 12 dict begin begincmap " "/CMapName /Custom def 1 begincodespacerange <00> endcodespacerange\n" "3 beginbfchar\n" "<41> \n" "<42> <00490067006E006F00720065>\n" "<43> <200B>\n" "endbfchar\nendcmap CMapName currentdict /CMap defineresource pop end end") p3 = make_pdf("3_tounicode.pdf", VISIBLE + "BT /F1 12 Tf 0 0 0 rg 60 700 Td (ABC abc) Tj ET\n", font_extra="/ToUnicode {tu} 0 R", hook=lambda p: {"tu": p.stream("", cmap)}) show("3. /ToUnicode remap", p3) print("pdfium chars (code points >0x7e):", [hex(ord(c)) for c in pdfium_chars(p3) if ord(c) > 0x7e]) print("pypdf chars (code points >0x7e):", [hex(ord(c)) for c in pypdf_text(p3) if ord(c) > 0x7e]) # 4. unused Form XObject def _xobj(p): font = p.add("<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica " "/Encoding /WinAnsiEncoding >>") xo = p.stream("/Type /XObject /Subtype /Form /BBox [0 0 595 842] " f"/Resources << /Font << /F1 {font} 0 R >> >>", "BT /F1 12 Tf 0 0 0 rg 60 500 Td (SECRET-4 inside unused xobject) Tj ET\n") return {"xo": xo} p4 = make_pdf("4_unused_xobject.pdf", VISIBLE, resources_extra="/XObject << /X0 {xo} 0 R >>", hook=_xobj) show("4. unused Form XObject (never drawn with Do)", p4) r = pypdf.PdfReader(p4) pg = r.pages[0] xo = pg["/Resources"]["/XObject"]["/X0"].get_object() print("pypdf extract_xform_text:", repr(pg.extract_xform_text(xo))) print("\nPDFs written to:", OUT) ``` Output, verbatim: ``` === 1. 0 Tz (zero-width text) === pdfium : 'This is the visible body text.' pypdf : 'This is the visible body text.\nSECRET-1 zero width' === 2. /ActualText === pdfium : 'This is the visible body text.\r\nSECRET-2 replaced text' pypdf : 'This is the visible body text.\nHello world' === 3. /ToUnicode remap === pdfium : 'This is the visible body text.\r\n\U000e0041Ignore\u200b abc' pypdf : 'This is the visible body text.\n\U000e0041Ignore\u200b abc' pdfium chars (code points >0x7e): ['0xdb40', '0xdc41', '0x200b'] pypdf chars (code points >0x7e): ['0xe0041', '0x200b'] === 4. unused Form XObject (never drawn with Do) === pdfium : 'This is the visible body text.' pypdf : 'This is the visible body text.' pypdf extract_xform_text: 'SECRET-4 inside unused xobject' ``` These checks shipped in the next version of the tool from the previous post, **PDF Privacy Checker**, on the Microsoft Store (detection is free; it runs fully offline, and not a single byte of your file leaves your PC). [https://apps.microsoft.com/detail/9PLRJHFTPS53?hl=en-us&gl=US](https://apps.microsoft.com/detail/9PLRJHFTPS53?hl=en-us&gl=US) If you have a trick you want to see run through this, leave a comment. Adding a case to the script above takes one function call. **Okinawa Software Lab.** I lead in-house digital transformation at a small company in Okinawa, Japan. I build the tools we need ourselves, and I publish PDF apps on the Microsoft Store that follow the same principle: everything happens on your own PC.