Extract text from a PDF and you get far too many line breaks. pdfium (the engine behind Chrome's PDF viewer) returns a break after every visual line, so sentences are cut in the middle of paragraphs. Feed that to an LLM, a search index or a summarizer, and the broken line structure can hurt the results. Remove every break instead, and bullet lists and tables melt into one long sentence.
This post is about how the "Export text for AI" feature of PDF Privacy Checker (a Windows app that finds hidden text in PDFs) decides which line breaks are soft wraps inside a paragraph, using only character coordinates. It is the follow-up I promised at the end of the previous post (I rebuilt my "export PDF text for AI" feature three times in one day).
This is not a write-up of rules that worked the first time. I measured them on real PDFs for this post, found they were broken, fixed them and measured again. The measurements led to three fixes in the app.
TL;DR
Here is a made-up store closing checklist I built for this post (the script at the end generates the PDF). This is what pdfium returns for page 1:
Closing Checklist
This sheet lists the checks to do in the last five minutes before
closing. If you follow them in order, you will not forget to lock
up, and the sales total will match the cash in the register. When
you are done, sign the back.
1 Count the cash in the register and compare it with the total on the
Daily Sales Sheet. If it does not match, call the manager.
2 Record the temperature of the fridge and the freezer. If one is
too warm, circle it in red for the morning shift.
- Manager: extension 11
- Security company: number on the receipt
- When in doubt, use the paper checklist
Denomination Count Amount
$20 bills 12 240.00
$5 bills 9 45.00
Quarters 40 10.00
(rest omitted)
The sample contains a title, a four-line paragraph, two numbered items with a hanging indent, three bullets, and a small cash-count table (denomination, count, amount). The breaks inside the paragraph and inside the numbered items should be joined. The breaks after the title, between the bullets and between the table rows should stay. The goal is to tell these two kinds apart.
A PDF has no notion of "this is the middle of a paragraph". It only says where each glyph is drawn. The meaning of a line break has to be inferred from positions.
These are real outputs of simple rules, run on the sample PDF that the companion script generates.
1. Remove every line break
Closing Checklist This sheet lists the checks β¦ sign the back. 1 Count the cash in the register β¦ call the manager. 2 Record the temperature β¦ morning shift. - Manager: extension 11 - Security company: number on the receipt - When in doubt, use the paper checklist Denomination Count Amount $20 bills 12 240.00 $5 bills 9 45.00 Quarters 40 10.00 Pull the shutter β¦
The title merges into the text, the bullets become one sentence and the whole table becomes one line. Is 45.00 the amount for the $5 bills, or the start of the next row? Neither a person nor an LLM can tell any more.
2. Join unless the line ends with a period (or ! ? :)
Closing Checklist This sheet lists the checks to do in the last five minutes before closing. β¦
- Manager: extension 11 - Security company: number on the receipt - When in doubt, use the paper checklist Denomination Count Amount $20 bills 12 240.00 $5 bills 9 45.00 Quarters 40 10.00 Pull the shutter down last β¦
It fails in both directions. Titles, bullets and table rows have no period, so they all merge. And whenever a sentence happens to end exactly at the right margin inside a paragraph, the paragraph is cut there. The first failure is dangerous; the second is merely ugly. Japanese, with γ as its full stop, fails the same way.
3. Join when the next line starts with a lowercase letter
1 Count the cash in the register and compare it with the total on the
Daily Sales Sheet. If it does not match, call the manager.
This works well on this page, and its failures are mostly misses rather than false merges: it cannot join a wrap that is followed by a capital letter or a digit, such as a document name, a person's name, an amount or a date. Business text is full of those. It also has no equivalent for Japanese, which has no letter case; the closest thing there, joining when both sides are CJK characters, merged the title into the text and the table header into its first row on my Japanese version of this page. Letter case and character class tell you how to join (with or without a space), not whether to join.
4. Same baseline means same line
This one is about the earlier step of merging fragments. PDFs from Word's character spacing or from CAD tools often draw one visual line as many small strings, and pdfium inserts a break between them. Matching baselines sounds like the fix, but a line where the number "5" is set in a different size from the text has different baselines. I use vertical overlap instead.
All four look only at the string. The meaning of a line break lives in where the lines are, not in the text.
pdfium gives a box for every character: FPDFText_GetCharBox (the tight glyph bounds) and a "loose" box built from the advance width and the font height. I use the union of both, because with some fonts one of them collapses (an example appears later).
The pipeline:
PDF
β pdfium (characters and boxes; a break after every visual line)
character boxes
β step 1: merge fragments of the same visual line
visual lines (left, right, top, bottom, median character height h)
β step 2: check each pair of neighbouring lines against 8 conditions
each break: soft wrap or real break
β join (no space between CJK characters, one space in English)
text for the LLM (the report's full-text appendix keeps the original breaks)
Every threshold is expressed in units of h, the character height, so the same rules work for any font size. h is about 1.1 times the font size.
| Threshold | Step | Meaning |
|---|---|---|
| vertical overlap β₯ 0.5h | 1 | on the same height |
| next fragment within 2.5h to the right | 1 | continuation of the same line |
| gap < 0.35h | 1 | touching (join without a space) |
| line pitch 0.5β1.8h | 2 | normal line spacing, not a paragraph gap |
| height difference β€ 0.3h | 2 | same text size |
| a gap β₯ 3h inside the line | 2 | column gap (a table row) |
| β₯ 90% of the longest line with the same left edge | 2 | the line is full |
| line width β₯ 12h | 2 | avoid short lines matching by accident |
| left-edge offset within 0.5h / β1.3 to β0.5h / +0.5 to 4h | 2 | aligned / 2nd line of an indented paragraph / hanging indent |
None of these come from the PDF specification. I set them once, then changed only what broke on real PDFs; the next sections say what. Also note that h is a height, so in terms of width it depends on the script: a full-width CJK character is about 0.9h wide, a Latin letter about 0.5h on average. "12h" is about 13 CJK characters or roughly 25 Latin ones.
pdfium's breaks are not always visual line boundaries. If the next piece overlaps the current line by at least half its height and starts within 2.5h to the right, it is a fragment of the same line: joined with no space if touching, otherwise with one space. After this step, step 2 only has to compare whole visual lines.
For two neighbouring visual lines a (above) and b (below), the break after a is a soft wrap only if all of the following hold. For each condition, the right column says what would be wrongly joined without it.
| # | Condition | Without it, this gets joined |
|---|---|---|
| 1 | not vertical text | vertical CJK text (horizontal only) |
| 2 | line pitch 0.5β1.8h | across a paragraph gap, or to a distant note |
| 3 | nearly the same text height | a heading and its text, text and a small note |
| 4 | the two lines overlap horizontally | different columns or cells |
| 5 | neither line has a gap of 3h or more | table rows (a condition I added later) |
| 6 | a is full (β₯ 90% of the longest line with the same left edge) and at least 12h wide | the short last line of a paragraph and the next paragraph; short bullets of equal width |
| 7 | b does not start with a bullet or a number | the next list item |
| 8 | the left edges are aligned, or form the 2nd line of an indented paragraph (CJK only), or a hanging indent of a numbered item | any other indentation (quotes, nesting) |
In code (from the companion script, a shortened version of the app's logic):
def is_wrap(a, b, lines, rules):
h = a["h"]
if (a["T"] - a["B"]) > 2.5 * h and (a["R"] - a["L"]) < 2 * h:
return False # 1. vertical text (out of scope)
if not (0.5 * h < a["B"] - b["B"] <= 1.8 * h):
return False # 2. unusual line pitch (paragraph gap)
if abs(a["h"] - b["h"]) > 0.3 * h:
return False # 3. different text size (heading, note)
if not (b["R"] > a["L"] and a["R"] > b["L"]):
return False # 4. no horizontal overlap (column, cell)
if "gap" in rules and (max_gap(a) >= 3 or max_gap(b) >= 3):
return False # 5. column gap (a table row)
block = [x["R"] for x in lines if abs(x["L"] - a["L"]) < 1.5 * h]
if a["R"] - a["L"] < 12 * h or a["R"] < 0.9 * max(block):
return False # 6. not a full line (paragraph end)
first, head = b["text"].lstrip(), a["text"].lstrip()
if first[:1] in LIST_HEADS or LIST_NUM.match(first):
return False # 7. next line starts a new item
dl = b["L"] - a["L"]
wide = is_wide(a["text"].rstrip()[-1]) and is_wide(first[0])
listy = head[:1] in LIST_HEADS or bool(LIST_NUM.match(head))
return (abs(dl) < 0.5 * h # same left edge
or (-1.3 * h <= dl < -0.5 * h and (wide or "cjk" not in rules)) # 2nd line of an indented paragraph
or (0.5 * h < dl <= 4 * h and listy)) # hanging indent of a numbered item
Condition 6 does most of the work. Lines inside a paragraph run to the right margin; the last line of a paragraph is short. It uses the typesetting itself. The "longest line" is taken only among lines with the same left edge, so that a long line elsewhere on the page (a table, another column) does not distort it.
The "2nd line of an indented paragraph" in condition 8 is a Japanese convention: when the first line is indented by one character, the second line starts one character further left. It originally applied to any script. Why it is now CJK-only comes below.
Recap. The whole method is four steps:
- Look at character coordinates, not at the string.
- Rebuild the visual lines from those coordinates (merging fragments of the same line).
- Check whether two neighbouring lines are a soft wrap inside a paragraph, using eight conditions.
- If any condition fails, do not join (keep the break).
A break that is a soft wrap gets replaced in one of three ways:
The third one almost never fires, because pdfium handles line-end hyphens itself. When a line ends with "-", pdfium does not insert a break there; it glues the two lines into one word. That hyphen comes back as code 0x02 from FPDFText_GetUnicode and as U+FFFE from get_text_range (FPDFText_IsHyphen returns 1 for it). In the sample, "con-" + "nected" on the English page came back as con\ufffenected.
This is where I found a bug in my own app. PDF Privacy Checker has a check for glyphs that extract as control characters, a sign of a tampered ToUnicode map. That check caught 0x02 and raised a red warning on ordinary English PDFs that simply had a hyphen at the end of a line. All five of my English articles printed from Chrome triggered it. Japanese business documents almost never end a line with "-", which is why my earlier tests had missed it.
The fix maps 0x02 with FPDFText_IsHyphen back to "-". The hyphen is kept, not removed: every case I found was a compound word such as off-page or macro-enabled, and removing it gives "offpage".
The "2nd line of an indented paragraph" rule used to apply to any script. On a publicly available English sample invoice (a "Sample Invoice Template"), it joined a table header to the next row:
(first rules)
Date Specific Service/Task Time by Task Cost of Task 7/14/2023 Met with individual to draft β¦
The next row started a little to the left of the header, exactly the shape of an indented paragraph's second line. That layout is common in English tables. A rule written for Japanese typesetting had fired on an English table, so I restricted it to CJK on both sides.
The companion script reproduces the same shape on page 2 (the heading of the example entries on the back of the sheet, and the first entry):
(first rules)
Examples of entries on the back of this sheet (one line per day) 7/14/2023 Closed at 22:10, cash matched, alarm set by M. Tanaka
(current rules)
Examples of entries on the back of this sheet (one line per day)
7/14/2023 Closed at 22:10, cash matched, alarm set by M. Tanaka
The cost: in an English paragraph with an indented first line, the wrap after the first line is now missed. In the measurement below, that was a difference of two line breaks.
For this post I measured the rules on two kinds of PDF.
Prose PDFs: my own 10 articles (5 Japanese, 5 English), converted to minimal HTML and printed from Chrome. Because I have the source, I can label every break automatically: paragraphs, headings, list items, table cells and single code lines are "units", a break inside a unit is a soft wrap and a break between units is a real break.
Business PDFs: 29 real invoices, quotes, delivery notes and vehicle documents (49 pages) from my own work. There is no ground truth, so I judged every join the rules made by eye.
Both were measured with heading detection turned off. The app (v1.15.0) also uses the same coordinates to detect heading lines and keeps them out of line joining, but here I wanted to compare the wrap rules alone. Heading detection is the topic of the next post.
Going through the joins in the business PDFs, I kept seeing lines like this one, from a publicly available Canada Post sample invoice:
(before)
Parcels 97.98 97.98 Commercial/Smartmail Marketing 216.90 216.90 Specialized services 1,240.00 1,240.00 Shipments 168.13 168.13 Adjustments -50.00 -50.00
Five invoice rows on one line. Table rows are stacked at the same width, so they pass condition 6, "the line is full." Normal line pitch, same text size, same left edge: they meet every condition for a paragraph. Japanese quotes had the same problem ("parts 1.00 1,210 1,210" glued to the next row).
What separates a paragraph line from a table row is the gaps inside the line. A paragraph only has word spaces; a table row has wide gaps between columns. So I added condition 5: neither line may contain a gap of three character heights or more.
(after)
Parcels 97.98 97.98
Commercial/Smartmail Marketing 216.90 216.90
Specialized services 1,240.00 1,240.00
Shipments 168.13 168.13
Adjustments -50.00 -50.00
The results. The two percentages point in different directions. Recall asks: of every 100 breaks that really are soft wraps, how many did it join? Precision asks: of every 100 joins it made, how many were really soft wraps? With the current rules, it joins about 90 of every 100 real wraps, and about 95 of every 100 joins are correct.
Prose PDFs (10 articles, 3,751 labelled line breaks)
| Rules | Correct joins | Missed wraps | Wrong joins | Recall | Precision |
|---|---|---|---|---|---|
| First rules | 1,366 | 153 | 93 | 89.9% | 93.6% |
| Indent rule limited to CJK | 1,364 | 155 | 93 | 89.8% | 93.6% |
| + column gap (current) | 1,364 | 155 | 69 | 89.8% | 95.2% |
Business PDFs (29 real documents, 3,603 line breaks)
| Rules | Joined as soft wraps | Correct (by eye) | Wrong |
|---|---|---|---|
| First rules | 125 | ~13 | ~112 |
| Indent rule limited to CJK | 121 | ~13 | ~108 |
| + column gap (current) | 56 | ~13 | ~43 |
Missed wraps in the business PDFs were not measured: there is no ground truth, and judging all 3,603 breaks by eye was not realistic. The correct/wrong split is also by eye. Step 1 additionally merged 385 fragments; 379 of them are the fine background pattern of digits on two insurance certificates.
What the numbers say:
The "current" row also includes a small effect from a separate fix found along the way: in PDFs saved from Chrome, the kanji "δΈ" and the em dash "β" were wrongly flagged as tiny text. The tiny-text check now uses the size the text is actually drawn at.
The results show more missed wraps (155) than wrong joins (69). That is intended.
The two mistakes do different amounts of harm:
For text that goes to an LLM, the second is much worse. Every condition leans towards keeping the break. In other words, I deliberately prefer false negatives to false positives: leaving one soft wrap untouched is less harmful than gluing two unrelated blocks together.
Rebuilding lines and rebuilding reading order are separate problems. This only decides whether two neighbouring lines, in the order pdfium returns them, should be joined. In a two-column layout, the order of the columns is still the PDF's drawing order.
Other things the current rules cannot do:
Pull the shutter down last and push it by hand to check that it is fully closed; if it moves at all, pull it down again and test it. Set the alarm and leave. If it beeps, turn it off and try again.
It only needs pypdfium2 (about 200 lines, half of which build the test PDF). The script generates its own test PDF with the standard Helvetica font, so no font file is needed. Tested with Python 3.12 and pypdfium2 5.12.1. Run python article10_rejoin_en.py your.pdf to try it on your own file.
First, what it produces. Page 1 with the current rules:
Closing Checklist
This sheet lists the checks to do in the last five minutes before closing. If you follow them in order, you will not forget to lock up, and the sales total will match the cash in the register. When you are done, sign the back.
1 Count the cash in the register and compare it with the total on the Daily Sales Sheet. If it does not match, call the manager.
2 Record the temperature of the fridge and the freezer. If one is too warm, circle it in red for the morning shift.
- Manager: extension 11
- Security company: number on the receipt
- When in doubt, use the paper checklist
Denomination Count Amount
$20 bills 12 240.00
$5 bills 9 45.00
Quarters 40 10.00
Pull the shutter down last and push it by hand to check that it is fully closed; if it moves at all, pull it down again and test it. Set the alarm and leave. If it beeps, turn it off and try again.
The paragraph and the numbered items are joined, including the wrap before "Daily Sales Sheet"; the bullets and the table keep their breaks. The last line is the limitation described above (two paragraphs merged). With the first rules, the four table rows become one line:
Denomination Count Amount $20 bills 12 240.00 $5 bills 9 45.00 Quarters 40 10.00
"""article10_rejoin_en.py - rejoin only the line breaks that are soft wraps inside a
paragraph, using character coordinates from pdfium (companion script for the article;
needs only pypdfium2).
python article10_rejoin_en.py - builds a test PDF and prints 3 versions
python article10_rejoin_en.py file.pdf - runs the current rules on your own PDF
A shortened version of the logic in PDF Privacy Checker (hidden_core 1.11.1).
Every threshold is written in units of h, the character height. They are not
derived from the PDF spec; they are rules of thumb tuned on real documents.
"""
import re
import sys
import pypdfium2 as pdfium
import pypdfium2.raw as R
LIST_HEADS = "γ»β’βββ β‘ββ-ββ*β»β β‘β’β£β€β₯β¦β§β¨β©"
LIST_NUM = re.compile(r"^[(οΌ]?\d{1,3}(?:[)οΌ.οΌγ:οΌ]|\s|$)")
def is_wide(ch): # full-width (kana, kanji, full-width punctuation): join without a space
o = ord(ch)
return 0x3000 <= o <= 0x30FF or 0x4E00 <= o <= 0x9FFF or 0xFF00 <= o <= 0xFFEF
def page_lines(page):
"""Rebuild visual lines from pdfium's characters and boxes (step 1: merge fragments)."""
tp = page.get_textpage()
n = tp.count_chars()
chars = [chr(R.FPDFText_GetUnicode(tp.raw, i)) for i in range(n)]
for i in range(n): # pdfium returns a line-end "-" as 0x02 and joins the lines itself
if chars[i] == "\x02" and R.FPDFText_IsHyphen(tp.raw, i):
chars[i] = "-"
boxes = [None] * n
for i, ch in enumerate(chars):
if ch.strip() and not R.FPDFText_IsGenerated(tp.raw, i):
t, lo = tp.get_charbox(i), tp.get_charbox(i, loose=True)
boxes[i] = (min(t[0], lo[0]), min(t[1], lo[1]), max(t[2], lo[2]), max(t[3], lo[3]))
text = "".join(chars)
lines = []
for m in re.finditer(r"[^\r\n]+", text):
bx = [boxes[i] for i in range(m.start(), m.end()) if boxes[i]]
if not bx:
continue
hs = sorted(b[3] - b[1] for b in bx)
g = {"text": m.group(), "bx": bx, "h": hs[len(hs) // 2],
"L": min(b[0] for b in bx), "R": max(b[2] for b in bx),
"T": max(b[3] for b in bx), "B": min(b[1] for b in bx)}
cur = lines[-1] if lines else None
if cur: # step 1: overlaps vertically by half and continues within 2.5h to the right
hmin = min(cur["h"], g["h"])
overlap = min(cur["T"], g["T"]) - max(cur["B"], g["B"])
gap = g["L"] - cur["R"]
if overlap >= 0.5 * hmin and -0.3 * hmin < gap < 2.5 * hmin:
cur["text"] += ("" if gap < 0.35 * hmin else " ") + g["text"]
cur["bx"] += g["bx"]
cur.update(R=max(cur["R"], g["R"]), T=max(cur["T"], g["T"]),
B=min(cur["B"], g["B"]))
continue
lines.append(g)
return lines
def max_gap(line):
"""Largest horizontal gap between neighbouring character boxes, in units of h."""
bx = sorted(line["bx"])
return max([(bx[k + 1][0] - bx[k][2]) / line["h"] for k in range(len(bx) - 1)] or [0])
def is_wrap(a, b, lines, rules):
"""Is the break after line a a soft wrap? True only if none of 1-7 applies and the
left edges match one of three shapes (when in doubt, keep the break)."""
h = a["h"]
if (a["T"] - a["B"]) > 2.5 * h and (a["R"] - a["L"]) < 2 * h:
return False # 1. vertical text (out of scope)
if not (0.5 * h < a["B"] - b["B"] <= 1.8 * h):
return False # 2. unusual line pitch (paragraph gap)
if abs(a["h"] - b["h"]) > 0.3 * h:
return False # 3. different text size (heading, note)
if not (b["R"] > a["L"] and a["R"] > b["L"]):
return False # 4. no horizontal overlap (column, cell)
if "gap" in rules and (max_gap(a) >= 3 or max_gap(b) >= 3):
return False # 5. column gap (a table row)
block = [x["R"] for x in lines if abs(x["L"] - a["L"]) < 1.5 * h]
if a["R"] - a["L"] < 12 * h or a["R"] < 0.9 * max(block):
return False # 6. not a full line (paragraph end)
first, head = b["text"].lstrip(), a["text"].lstrip()
if first[:1] in LIST_HEADS or LIST_NUM.match(first):
return False # 7. next line starts a new item
dl = b["L"] - a["L"]
wide = is_wide(a["text"].rstrip()[-1]) and is_wide(first[0])
listy = head[:1] in LIST_HEADS or bool(LIST_NUM.match(head))
return (abs(dl) < 0.5 * h # same left edge
or (-1.3 * h <= dl < -0.5 * h and (wide or "cjk" not in rules)) # 2nd line of an indented paragraph
or (0.5 * h < dl <= 4 * h and listy)) # hanging indent of a numbered item
def rejoin(lines, rules):
out = lines[0]["text"] if lines else ""
for a, b in zip(lines, lines[1:]):
if not is_wrap(a, b, lines, rules):
out += "\n" + b["text"]
elif is_wide(out.rstrip()[-1]) and is_wide(b["text"].lstrip()[0]):
out = out.rstrip() + b["text"].lstrip() # Japanese: no space
elif out.endswith("-") and b["text"][:1].islower():
out = out[:-1] + b["text"] # undo hyphenation (a line-end "-" pdfium did not mark)
else:
out = out.rstrip() + " " + b["text"].lstrip() # English: one space
return out
def build_pdf(path):
en, en2 = [], []
def E(x, y, s, size=10, page=None):
s = s.replace("\\", "\\\\").replace("(", "\\(").replace(")", "\\)")
(en if page is None else page).append(f"BT /F1 {size} Tf {x} {y} Td ({s}) Tj ET")
E(72, 780, "Closing Checklist", 16)
for k, s in enumerate(["This sheet lists the checks to do in the last five minutes before",
"closing. If you follow them in order, you will not forget to lock",
"up, and the sales total will match the cash in the register. When",
"you are done, sign the back."]):
E(72, 750 - 14 * k, s)
for y, no, a, b in [(685, "1", "Count the cash in the register and compare it with the total on the",
"Daily Sales Sheet. If it does not match, call the manager."),
(651, "2", "Record the temperature of the fridge and the freezer. If one is",
"too warm, circle it in red for the morning shift.")]:
E(72, y, no); E(86, y, a); E(86, y - 14, b) # hanging indent
for k, s in enumerate(["- Manager: extension 11", "- Security company: number on the receipt",
"- When in doubt, use the paper checklist"]):
E(72, 610 - 14 * k, s)
for k, (a, b, c) in enumerate([("Denomination", "Count", "Amount"), # cash count
("$20 bills", "12", "240.00"),
("$5 bills", "9", "45.00"),
("Quarters", "40", "10.00")]):
E(72, 555 - 14 * k, a); E(230, 555 - 14 * k, b); E(330, 555 - 14 * k, c)
E(72, 480, "Pull the shutter down last and push it by hand to check that it")
E(72, 466, "is fully closed; if it moves at all, pull it down again and test it.")
E(72, 452, "Set the alarm and leave. If it beeps, turn it off and try again.")
E(72, 780, "Notes", 16, en2)
for k, s in enumerate(["Deliveries that arrive after closing must not be left at the back",
"door or in front of the shop. The store cannot take any",
"responsibility for goods that are left outside overnight, so ask",
"the driver to take them back."]):
E(72, 750 - 14 * k, s, page=en2)
E(72, 680, "The alarm panel shows a green light only when every door is con-", page=en2)
E(72, 666, "nected to the system; a red light means that one is still open.", page=en2)
E(82, 630, "Examples of entries on the back of this sheet (one line per day)", page=en2)
E(72, 616, "7/14/2023 Closed at 22:10, cash matched, alarm set by M. Tanaka", page=en2)
objs = ["<< /Type /Catalog /Pages 2 0 R >>", "",
"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>"]
kids = []
for ops in (en, en2):
data = "\n".join(ops).encode("latin-1")
objs.append(f"<< /Length {len(data)} >>\nstream\n".encode() + data + b"\nendstream")
objs.append(f"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 595 842] /Contents {len(objs)} 0 R "
"/Resources << /Font << /F1 3 0 R >> >> >>")
kids.append(len(objs))
objs[1] = f"<< /Type /Pages /Kids [{' '.join(f'{k} 0 R' for k in kids)}] /Count {len(kids)} >>"
out, offs = b"%PDF-1.7\n", []
for k, o in enumerate(objs, 1):
offs.append(len(out))
out += f"{k} 0 obj\n".encode() + (o if isinstance(o, bytes) else o.encode()) + b"\nendobj\n"
xref = len(out)
out += f"xref\n0 {len(objs) + 1}\n0000000000 65535 f \n".encode()
out += b"".join(f"{o:010d} 00000 n \n".encode() for o in offs)
out += f"trailer << /Size {len(objs) + 1} /Root 1 0 R >>\nstartxref\n{xref}\n%%EOF\n".encode()
open(path, "wb").write(out)
if __name__ == "__main__":
sys.stdout.reconfigure(encoding="utf-8")
if len(sys.argv) > 1:
for page in pdfium.PdfDocument(sys.argv[1]):
print(rejoin(page_lines(page), {"cjk", "gap"}), end="\n\n")
sys.exit()
build_pdf("article10_sample_en.pdf")
doc = pdfium.PdfDocument("article10_sample_en.pdf")
for no, page in enumerate(doc, 1):
lines = page_lines(page)
print(f"===== page {no}: pdfium line breaks as-is =====")
print(page.get_textpage().get_text_range().replace("\r\n", "\n"))
print(f"===== page {no}: first rules (indent rule for any script, no column-gap check) =====")
print(rejoin(lines, set()))
print(f"===== page {no}: current rules =====")
print(rejoin(lines, {"cjk", "gap"}))
print()
Measuring for this post led to three fixes in PDF Privacy Checker: table rows are no longer glued together, a line-end hyphen no longer triggers a false control-character warning, and "δΈ" and "β" in PDFs saved from Chrome are no longer flagged as tiny text. All three are in v1.15.0, which is now on the Store.
PDF Privacy Checker is on the Microsoft Store (detection is free; the text export for AI is part of a paid add-on). Files never leave your PC; it works fully offline. The source code is not public.
https://apps.microsoft.com/detail/9PLRJHFTPS53?hl=en-us&gl=US
Next, I plan to write about using the same coordinates to detect heading lines, so the text for AI keeps the document's structure.
About this article β The implementation and the writing were done together with Claude (Anthropic's AI). Most of the prose was drafted by Claude and checked by me. The decisions about which errors to fix in the product and which to leave as limitations, and the real business documents, are mine.
Okinawa Software Lab. I lead in-house digital transformation at a small company in Okinawa, Japan. I build the tools we need ourselves, and I publish PDF apps on the Microsoft Store that follow the same principle: everything happens on your own PC.