Which Line Breaks in Extracted PDF Text Are Real? Rejoining Wrapped Lines from Character Coordinates

A developer behind the Windows app PDF Privacy Checker documented how its "Export text for AI" feature decides which line breaks in pdfium-extracted PDF text are soft wraps inside paragraphs, using only character coordinates. After measuring simple heuristics on real PDFs, the developer found that removing all breaks merges bullets and tables, while joining only after periods fails in both directions, leading to three fixes in the app. The goal is to distinguish breaks that should be joined from those that should be kept so LLMs, search indexes and summarizers receive correctly structured text.

Extract text from a PDF and you get far too many line breaks. pdfium the engine behind Chrome's PDF viewer returns a break after every visual line, so sentences are cut in the middle of paragraphs. Feed that to an LLM, a search index or a summarizer, and the broken line structure can hurt the results. Remove every break instead, and bullet lists and tables melt into one long sentence. This post is about how the "Export text for AI" feature of PDF Privacy Checker a Windows app that finds hidden text in PDFs decides which line breaks are soft wraps inside a paragraph, using only character coordinates . It is the follow-up I promised at the end of the previous post I rebuilt my "export PDF text for AI" feature three times in one day https://dev.to/okinawasoftware/i-rebuilt-my-export-pdf-text-for-ai-feature-three-times-in-one-day-heres-what-each-version-got-3dge . This is not a write-up of rules that worked the first time. I measured them on real PDFs for this post, found they were broken, fixed them and measured again. The measurements led to three fixes in the app. TL;DR Here is a made-up store closing checklist I built for this post the script at the end generates the PDF . This is what pdfium returns for page 1: Closing Checklist This sheet lists the checks to do in the last five minutes before closing. If you follow them in order, you will not forget to lock up, and the sales total will match the cash in the register. When you are done, sign the back. 1 Count the cash in the register and compare it with the total on the Daily Sales Sheet. If it does not match, call the manager. 2 Record the temperature of the fridge and the freezer. If one is too warm, circle it in red for the morning shift. - Manager: extension 11 - Security company: number on the receipt - When in doubt, use the paper checklist Denomination Count Amount $20 bills 12 240.00 $5 bills 9 45.00 Quarters 40 10.00 rest omitted The sample contains a title, a four-line paragraph, two numbered items with a hanging indent, three bullets, and a small cash-count table denomination, count, amount . The breaks inside the paragraph and inside the numbered items should be joined. The breaks after the title, between the bullets and between the table rows should stay. The goal is to tell these two kinds apart. A PDF has no notion of "this is the middle of a paragraph". It only says where each glyph is drawn. The meaning of a line break has to be inferred from positions. These are real outputs of simple rules, run on the sample PDF that the companion script generates. 1. Remove every line break Closing Checklist This sheet lists the checks … sign the back. 1 Count the cash in the register … call the manager. 2 Record the temperature … morning shift. - Manager: extension 11 - Security company: number on the receipt - When in doubt, use the paper checklist Denomination Count Amount $20 bills 12 240.00 $5 bills 9 45.00 Quarters 40 10.00 Pull the shutter … The title merges into the text, the bullets become one sentence and the whole table becomes one line. Is 45.00 the amount for the $5 bills, or the start of the next row? Neither a person nor an LLM can tell any more. 2. Join unless the line ends with a period or ? : Closing Checklist This sheet lists the checks to do in the last five minutes before closing. … - Manager: extension 11 - Security company: number on the receipt - When in doubt, use the paper checklist Denomination Count Amount $20 bills 12 240.00 $5 bills 9 45.00 Quarters 40 10.00 Pull the shutter down last … It fails in both directions. Titles, bullets and table rows have no period, so they all merge. And whenever a sentence happens to end exactly at the right margin inside a paragraph, the paragraph is cut there. The first failure is dangerous; the second is merely ugly. Japanese, with 。 as its full stop, fails the same way. 3. Join when the next line starts with a lowercase letter 1 Count the cash in the register and compare it with the total on the Daily Sales Sheet. If it does not match, call the manager. This works well on this page, and its failures are mostly misses rather than false merges: it cannot join a wrap that is followed by a capital letter or a digit, such as a document name, a person's name, an amount or a date. Business text is full of those. It also has no equivalent for Japanese, which has no letter case; the closest thing there, joining when both sides are CJK characters, merged the title into the text and the table header into its first row on my Japanese version of this page. Letter case and character class tell you how to join with or without a space , not whether to join. 4. Same baseline means same line This one is about the earlier step of merging fragments. PDFs from Word's character spacing or from CAD tools often draw one visual line as many small strings, and pdfium inserts a break between them. Matching baselines sounds like the fix, but a line where the number "5" is set in a different size from the text has different baselines. I use vertical overlap instead. All four look only at the string. The meaning of a line break lives in where the lines are, not in the text. pdfium gives a box for every character: FPDFText GetCharBox the tight glyph bounds and a "loose" box built from the advance width and the font height. I use the union of both, because with some fonts one of them collapses an example appears later . The pipeline: PDF ↓ pdfium characters and boxes; a break after every visual line character boxes ↓ step 1: merge fragments of the same visual line visual lines left, right, top, bottom, median character height h ↓ step 2: check each pair of neighbouring lines against 8 conditions each break: soft wrap or real break ↓ join no space between CJK characters, one space in English text for the LLM the report's full-text appendix keeps the original breaks Every threshold is expressed in units of h, the character height, so the same rules work for any font size. h is about 1.1 times the font size. | Threshold | Step | Meaning | |---|---|---| | vertical overlap ≥ 0.5h | 1 | on the same height | | next fragment within 2.5h to the right | 1 | continuation of the same line | | gap < 0.35h | 1 | touching join without a space | | line pitch 0.5–1.8h | 2 | normal line spacing, not a paragraph gap | | height difference ≤ 0.3h | 2 | same text size | | a gap ≥ 3h inside the line | 2 | column gap a table row | | ≥ 90% of the longest line with the same left edge | 2 | the line is full | | line width ≥ 12h | 2 | avoid short lines matching by accident | | left-edge offset within 0.5h / −1.3 to −0.5h / +0.5 to 4h | 2 | aligned / 2nd line of an indented paragraph / hanging indent | None of these come from the PDF specification. I set them once, then changed only what broke on real PDFs; the next sections say what. Also note that h is a height, so in terms of width it depends on the script: a full-width CJK character is about 0.9h wide, a Latin letter about 0.5h on average. "12h" is about 13 CJK characters or roughly 25 Latin ones. pdfium's breaks are not always visual line boundaries. If the next piece overlaps the current line by at least half its height and starts within 2.5h to the right , it is a fragment of the same line: joined with no space if touching, otherwise with one space. After this step, step 2 only has to compare whole visual lines. For two neighbouring visual lines a above and b below , the break after a is a soft wrap only if all of the following hold. For each condition, the right column says what would be wrongly joined without it. | | Condition | Without it, this gets joined | |---|---|---| | 1 | not vertical text | vertical CJK text horizontal only | | 2 | line pitch 0.5–1.8h | across a paragraph gap, or to a distant note | | 3 | nearly the same text height | a heading and its text, text and a small note | | 4 | the two lines overlap horizontally | different columns or cells | | 5 | neither line has a gap of 3h or more | table rows a condition I added later | | 6 | a is full ≥ 90% of the longest line with the same left edge and at least 12h wide | the short last line of a paragraph and the next paragraph; short bullets of equal width | | 7 | b does not start with a bullet or a number | the next list item | | 8 | the left edges are aligned, or form the 2nd line of an indented paragraph CJK only , or a hanging indent of a numbered item | any other indentation quotes, nesting | In code from the companion script, a shortened version of the app's logic : python def is wrap a, b, lines, rules : h = a "h" if a "T" - a "B" 2.5 h and a "R" - a "L" < 2 h: return False 1. vertical text out of scope if not 0.5 h < a "B" - b "B" <= 1.8 h : return False 2. unusual line pitch paragraph gap if abs a "h" - b "h" 0.3 h: return False 3. different text size heading, note if not b "R" a "L" and a "R" b "L" : return False 4. no horizontal overlap column, cell if "gap" in rules and max gap a = 3 or max gap b = 3 : return False 5. column gap a table row block = x "R" for x in lines if abs x "L" - a "L" < 1.5 h if a "R" - a "L" < 12 h or a "R" < 0.9 max block : return False 6. not a full line paragraph end first, head = b "text" .lstrip , a "text" .lstrip if first :1 in LIST HEADS or LIST NUM.match first : return False 7. next line starts a new item dl = b "L" - a "L" wide = is wide a "text" .rstrip -1 and is wide first 0 listy = head :1 in LIST HEADS or bool LIST NUM.match head return abs dl < 0.5 h same left edge or -1.3 h <= dl < -0.5 h and wide or "cjk" not in rules 2nd line of an indented paragraph or 0.5 h < dl <= 4 h and listy hanging indent of a numbered item Condition 6 does most of the work. Lines inside a paragraph run to the right margin; the last line of a paragraph is short. It uses the typesetting itself. The "longest line" is taken only among lines with the same left edge, so that a long line elsewhere on the page a table, another column does not distort it. The "2nd line of an indented paragraph" in condition 8 is a Japanese convention: when the first line is indented by one character, the second line starts one character further left. It originally applied to any script. Why it is now CJK-only comes below. Recap. The whole method is four steps: - Look at character coordinates, not at the string. - Rebuild the visual lines from those coordinates merging fragments of the same line . - Check whether two neighbouring lines are a soft wrap inside a paragraph, using eight conditions. - If any condition fails, do not join keep the break . A break that is a soft wrap gets replaced in one of three ways: The third one almost never fires, because pdfium handles line-end hyphens itself . When a line ends with "-", pdfium does not insert a break there; it glues the two lines into one word. That hyphen comes back as code 0x02 from FPDFText GetUnicode and as U+FFFE from get text range FPDFText IsHyphen returns 1 for it . In the sample, "con-" + "nected" on the English page came back as con\ufffenected . This is where I found a bug in my own app. PDF Privacy Checker has a check for glyphs that extract as control characters, a sign of a tampered ToUnicode map. That check caught 0x02 and raised a red warning on ordinary English PDFs that simply had a hyphen at the end of a line . All five of my English articles printed from Chrome triggered it. Japanese business documents almost never end a line with "-", which is why my earlier tests had missed it. The fix maps 0x02 with FPDFText IsHyphen back to "-". The hyphen is kept, not removed: every case I found was a compound word such as off-page or macro-enabled, and removing it gives "offpage". The "2nd line of an indented paragraph" rule used to apply to any script. On a publicly available English sample invoice a "Sample Invoice Template" , it joined a table header to the next row: first rules Date Specific Service/Task Time by Task Cost of Task 7/14/2023 Met with individual to draft … The next row started a little to the left of the header, exactly the shape of an indented paragraph's second line. That layout is common in English tables. A rule written for Japanese typesetting had fired on an English table, so I restricted it to CJK on both sides. The companion script reproduces the same shape on page 2 the heading of the example entries on the back of the sheet, and the first entry : first rules Examples of entries on the back of this sheet one line per day 7/14/2023 Closed at 22:10, cash matched, alarm set by M. Tanaka current rules Examples of entries on the back of this sheet one line per day 7/14/2023 Closed at 22:10, cash matched, alarm set by M. Tanaka The cost: in an English paragraph with an indented first line, the wrap after the first line is now missed. In the measurement below, that was a difference of two line breaks. For this post I measured the rules on two kinds of PDF. Prose PDFs : my own 10 articles 5 Japanese, 5 English , converted to minimal HTML and printed from Chrome. Because I have the source, I can label every break automatically: paragraphs, headings, list items, table cells and single code lines are "units", a break inside a unit is a soft wrap and a break between units is a real break . Business PDFs : 29 real invoices, quotes, delivery notes and vehicle documents 49 pages from my own work. There is no ground truth, so I judged every join the rules made by eye. Both were measured with heading detection turned off. The app v1.15.0 also uses the same coordinates to detect heading lines and keeps them out of line joining, but here I wanted to compare the wrap rules alone. Heading detection is the topic of the next post. Going through the joins in the business PDFs, I kept seeing lines like this one, from a publicly available Canada Post sample invoice: before Parcels 97.98 97.98 Commercial/Smartmail Marketing 216.90 216.90 Specialized services 1,240.00 1,240.00 Shipments 168.13 168.13 Adjustments -50.00 -50.00 Five invoice rows on one line. Table rows are stacked at the same width, so they pass condition 6, "the line is full." Normal line pitch, same text size, same left edge: they meet every condition for a paragraph. Japanese quotes had the same problem "parts 1.00 1,210 1,210" glued to the next row . What separates a paragraph line from a table row is the gaps inside the line. A paragraph only has word spaces; a table row has wide gaps between columns. So I added condition 5: neither line may contain a gap of three character heights or more. after Parcels 97.98 97.98 Commercial/Smartmail Marketing 216.90 216.90 Specialized services 1,240.00 1,240.00 Shipments 168.13 168.13 Adjustments -50.00 -50.00 The results. The two percentages point in different directions. Recall asks: of every 100 breaks that really are soft wraps, how many did it join? Precision asks: of every 100 joins it made, how many were really soft wraps? With the current rules, it joins about 90 of every 100 real wraps, and about 95 of every 100 joins are correct. Prose PDFs 10 articles, 3,751 labelled line breaks | Rules | Correct joins | Missed wraps | Wrong joins | Recall | Precision | |---|---|---|---|---|---| | First rules | 1,366 | 153 | 93 | 89.9% | 93.6% | | Indent rule limited to CJK | 1,364 | 155 | 93 | 89.8% | 93.6% | | + column gap current | 1,364 | 155 | 69 | 89.8% | 95.2% | Business PDFs 29 real documents, 3,603 line breaks | Rules | Joined as soft wraps | Correct by eye | Wrong | |---|---|---|---| | First rules | 125 | ~13 | ~112 | | Indent rule limited to CJK | 121 | ~13 | ~108 | | + column gap current | 56 | ~13 | ~43 | Missed wraps in the business PDFs were not measured : there is no ground truth, and judging all 3,603 breaks by eye was not realistic. The correct/wrong split is also by eye. Step 1 additionally merged 385 fragments; 379 of them are the fine background pattern of digits on two insurance certificates. What the numbers say: The "current" row also includes a small effect from a separate fix found along the way: in PDFs saved from Chrome, the kanji "一" and the em dash "—" were wrongly flagged as tiny text. The tiny-text check now uses the size the text is actually drawn at. The results show more missed wraps 155 than wrong joins 69 . That is intended. The two mistakes do different amounts of harm: For text that goes to an LLM, the second is much worse. Every condition leans towards keeping the break. In other words, I deliberately prefer false negatives to false positives: leaving one soft wrap untouched is less harmful than gluing two unrelated blocks together. Rebuilding lines and rebuilding reading order are separate problems. This only decides whether two neighbouring lines, in the order pdfium returns them, should be joined. In a two-column layout, the order of the columns is still the PDF's drawing order. Other things the current rules cannot do: Pull the shutter down last and push it by hand to check that it is fully closed; if it moves at all, pull it down again and test it. Set the alarm and leave. If it beeps, turn it off and try again. It only needs pypdfium2 about 200 lines, half of which build the test PDF . The script generates its own test PDF with the standard Helvetica font, so no font file is needed. Tested with Python 3.12 and pypdfium2 5.12.1. Run python article10 rejoin en.py your.pdf to try it on your own file. First, what it produces. Page 1 with the current rules: Closing Checklist This sheet lists the checks to do in the last five minutes before closing. If you follow them in order, you will not forget to lock up, and the sales total will match the cash in the register. When you are done, sign the back. 1 Count the cash in the register and compare it with the total on the Daily Sales Sheet. If it does not match, call the manager. 2 Record the temperature of the fridge and the freezer. If one is too warm, circle it in red for the morning shift. - Manager: extension 11 - Security company: number on the receipt - When in doubt, use the paper checklist Denomination Count Amount $20 bills 12 240.00 $5 bills 9 45.00 Quarters 40 10.00 Pull the shutter down last and push it by hand to check that it is fully closed; if it moves at all, pull it down again and test it. Set the alarm and leave. If it beeps, turn it off and try again. The paragraph and the numbered items are joined, including the wrap before "Daily Sales Sheet"; the bullets and the table keep their breaks. The last line is the limitation described above two paragraphs merged . With the first rules, the four table rows become one line: Denomination Count Amount $20 bills 12 240.00 $5 bills 9 45.00 Quarters 40 10.00 - - coding: utf-8 - - """article10 rejoin en.py - rejoin only the line breaks that are soft wraps inside a paragraph, using character coordinates from pdfium companion script for the article; needs only pypdfium2 . python article10 rejoin en.py - builds a test PDF and prints 3 versions python article10 rejoin en.py file.pdf - runs the current rules on your own PDF A shortened version of the logic in PDF Privacy Checker hidden core 1.11.1 . Every threshold is written in units of h, the character height. They are not derived from the PDF spec; they are rules of thumb tuned on real documents. """ import re import sys import pypdfium2 as pdfium import pypdfium2.raw as R LIST HEADS = "・•●○■□◆◇-–— ※①②③④⑤⑥⑦⑧⑨⑩" LIST NUM = re.compile r"^ （ ?\d{1,3} ?: ）.．、:： |\s|$ " def is wide ch : full-width kana, kanji, full-width punctuation : join without a space o = ord ch return 0x3000 <= o <= 0x30FF or 0x4E00 <= o <= 0x9FFF or 0xFF00 <= o <= 0xFFEF def page lines page : """Rebuild visual lines from pdfium's characters and boxes step 1: merge fragments .""" tp = page.get textpage n = tp.count chars chars = chr R.FPDFText GetUnicode tp.raw, i for i in range n for i in range n : pdfium returns a line-end "-" as 0x02 and joins the lines itself if chars i == "\x02" and R.FPDFText IsHyphen tp.raw, i : chars i = "-" boxes = None n for i, ch in enumerate chars : if ch.strip and not R.FPDFText IsGenerated tp.raw, i : t, lo = tp.get charbox i , tp.get charbox i, loose=True boxes i = min t 0 , lo 0 , min t 1 , lo 1 , max t 2 , lo 2 , max t 3 , lo 3 text = "".join chars lines = for m in re.finditer r" ^\r\n +", text : bx = boxes i for i in range m.start , m.end if boxes i if not bx: continue hs = sorted b 3 - b 1 for b in bx g = {"text": m.group , "bx": bx, "h": hs len hs // 2 , "L": min b 0 for b in bx , "R": max b 2 for b in bx , "T": max b 3 for b in bx , "B": min b 1 for b in bx } cur = lines -1 if lines else None if cur: step 1: overlaps vertically by half and continues within 2.5h to the right hmin = min cur "h" , g "h" overlap = min cur "T" , g "T" - max cur "B" , g "B" gap = g "L" - cur "R" if overlap = 0.5 hmin and -0.3 hmin < gap < 2.5 hmin: cur "text" += "" if gap < 0.35 hmin else " " + g "text" cur "bx" += g "bx" cur.update R=max cur "R" , g "R" , T=max cur "T" , g "T" , B=min cur "B" , g "B" continue lines.append g return lines def max gap line : """Largest horizontal gap between neighbouring character boxes, in units of h.""" bx = sorted line "bx" return max bx k + 1 0 - bx k 2 / line "h" for k in range len bx - 1 or 0 def is wrap a, b, lines, rules : """Is the break after line a a soft wrap? True only if none of 1-7 applies and the left edges match one of three shapes when in doubt, keep the break .""" h = a "h" if a "T" - a "B" 2.5 h and a "R" - a "L" < 2 h: return False 1. vertical text out of scope if not 0.5 h < a "B" - b "B" <= 1.8 h : return False 2. unusual line pitch paragraph gap if abs a "h" - b "h" 0.3 h: return False 3. different text size heading, note if not b "R" a "L" and a "R" b "L" : return False 4. no horizontal overlap column, cell if "gap" in rules and max gap a = 3 or max gap b = 3 : return False 5. column gap a table row block = x "R" for x in lines if abs x "L" - a "L" < 1.5 h if a "R" - a "L" < 12 h or a "R" < 0.9 max block : return False 6. not a full line paragraph end first, head = b "text" .lstrip , a "text" .lstrip if first :1 in LIST HEADS or LIST NUM.match first : return False 7. next line starts a new item dl = b "L" - a "L" wide = is wide a "text" .rstrip -1 and is wide first 0 listy = head :1 in LIST HEADS or bool LIST NUM.match head return abs dl < 0.5 h same left edge or -1.3 h <= dl < -0.5 h and wide or "cjk" not in rules 2nd line of an indented paragraph or 0.5 h < dl <= 4 h and listy hanging indent of a numbered item def rejoin lines, rules : out = lines 0 "text" if lines else "" for a, b in zip lines, lines 1: : if not is wrap a, b, lines, rules : out += "\n" + b "text" elif is wide out.rstrip -1 and is wide b "text" .lstrip 0 : out = out.rstrip + b "text" .lstrip Japanese: no space elif out.endswith "-" and b "text" :1 .islower : out = out :-1 + b "text" undo hyphenation a line-end "-" pdfium did not mark else: out = out.rstrip + " " + b "text" .lstrip English: one space return out ---- test PDF raw PDF, standard Helvetica; nothing is embedded ---- def build pdf path : en, en2 = , def E x, y, s, size=10, page=None : s = s.replace "\\", "\\\\" .replace " ", "\\ " .replace " ", "\\ " en if page is None else page .append f"BT /F1 {size} Tf {x} {y} Td {s} Tj ET" E 72, 780, "Closing Checklist", 16 for k, s in enumerate "This sheet lists the checks to do in the last five minutes before", "closing. If you follow them in order, you will not forget to lock", "up, and the sales total will match the cash in the register. When", "you are done, sign the back." : E 72, 750 - 14 k, s for y, no, a, b in 685, "1", "Count the cash in the register and compare it with the total on the", "Daily Sales Sheet. If it does not match, call the manager." , 651, "2", "Record the temperature of the fridge and the freezer. If one is", "too warm, circle it in red for the morning shift." : E 72, y, no ; E 86, y, a ; E 86, y - 14, b hanging indent for k, s in enumerate "- Manager: extension 11", "- Security company: number on the receipt", "- When in doubt, use the paper checklist" : E 72, 610 - 14 k, s for k, a, b, c in enumerate "Denomination", "Count", "Amount" , cash count "$20 bills", "12", "240.00" , "$5 bills", "9", "45.00" , "Quarters", "40", "10.00" : E 72, 555 - 14 k, a ; E 230, 555 - 14 k, b ; E 330, 555 - 14 k, c limitation: a paragraph whose last line happens to be full, followed by a paragraph with no indent and no extra space E 72, 480, "Pull the shutter down last and push it by hand to check that it" E 72, 466, "is fully closed; if it moves at all, pull it down again and test it." E 72, 452, "Set the alarm and leave. If it beeps, turn it off and try again." E 72, 780, "Notes", 16, en2 for k, s in enumerate "Deliveries that arrive after closing must not be left at the back", "door or in front of the shop. The store cannot take any", "responsibility for goods that are left outside overnight, so ask", "the driver to take them back." : E 72, 750 - 14 k, s, page=en2 E 72, 680, "The alarm panel shows a green light only when every door is con-", page=en2 E 72, 666, "nected to the system; a red light means that one is still open.", page=en2 E 82, 630, "Examples of entries on the back of this sheet one line per day ", page=en2 E 72, 616, "7/14/2023 Closed at 22:10, cash matched, alarm set by M. Tanaka", page=en2 objs = "<< /Type /Catalog /Pages 2 0 R ", "", "<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding " kids = for ops in en, en2 : data = "\n".join ops .encode "latin-1" objs.append f"<< /Length {len data } \nstream\n".encode + data + b"\nendstream" objs.append f"<< /Type /Page /Parent 2 0 R /MediaBox 0 0 595 842 /Contents {len objs } 0 R " "/Resources << /Font << /F1 3 0 R " kids.append len objs objs 1 = f"<< /Type /Pages /Kids {' '.join f'{k} 0 R' for k in kids } /Count {len kids } " out, offs = b"%PDF-1.7\n", for k, o in enumerate objs, 1 : offs.append len out out += f"{k} 0 obj\n".encode + o if isinstance o, bytes else o.encode + b"\nendobj\n" xref = len out out += f"xref\n0 {len objs + 1}\n0000000000 65535 f \n".encode out += b"".join f"{o:010d} 00000 n \n".encode for o in offs out += f"trailer << /Size {len objs + 1} /Root 1 0 R \nstartxref\n{xref}\n%%EOF\n".encode open path, "wb" .write out if name == " main ": sys.stdout.reconfigure encoding="utf-8" if len sys.argv 1: for page in pdfium.PdfDocument sys.argv 1 : print rejoin page lines page , {"cjk", "gap"} , end="\n\n" sys.exit build pdf "article10 sample en.pdf" doc = pdfium.PdfDocument "article10 sample en.pdf" for no, page in enumerate doc, 1 : lines = page lines page print f"===== page {no}: pdfium line breaks as-is =====" print page.get textpage .get text range .replace "\r\n", "\n" print f"===== page {no}: first rules indent rule for any script, no column-gap check =====" print rejoin lines, set print f"===== page {no}: current rules =====" print rejoin lines, {"cjk", "gap"} print Measuring for this post led to three fixes in PDF Privacy Checker: table rows are no longer glued together, a line-end hyphen no longer triggers a false control-character warning, and "一" and "—" in PDFs saved from Chrome are no longer flagged as tiny text. All three are in v1.15.0, which is now on the Store. PDF Privacy Checker is on the Microsoft Store detection is free; the text export for AI is part of a paid add-on . Files never leave your PC; it works fully offline. The source code is not public. https://apps.microsoft.com/detail/9PLRJHFTPS53?hl=en-us&gl=US https://apps.microsoft.com/detail/9PLRJHFTPS53?hl=en-us&gl=US Next, I plan to write about using the same coordinates to detect heading lines, so the text for AI keeps the document's structure. About this article — The implementation and the writing were done together with Claude Anthropic's AI . Most of the prose was drafted by Claude and checked by me. The decisions about which errors to fix in the product and which to leave as limitations, and the real business documents, are mine. Okinawa Software Lab. I lead in-house digital transformation at a small company in Okinawa, Japan. I build the tools we need ourselves, and I publish PDF apps on the Microsoft Store that follow the same principle: everything happens on your own PC.