{"slug": "which-line-breaks-in-extracted-pdf-text-are-real-rejoining-wrapped-lines-from", "title": "Which Line Breaks in Extracted PDF Text Are Real? Rejoining Wrapped Lines from Character Coordinates", "summary": "A developer behind the Windows app PDF Privacy Checker documented how its \"Export text for AI\" feature decides which line breaks in pdfium-extracted PDF text are soft wraps inside paragraphs, using only character coordinates. After measuring simple heuristics on real PDFs, the developer found that removing all breaks merges bullets and tables, while joining only after periods fails in both directions, leading to three fixes in the app. The goal is to distinguish breaks that should be joined from those that should be kept so LLMs, search indexes and summarizers receive correctly structured text.", "body_md": "Extract text from a PDF and you get far too many line breaks. pdfium (the engine behind Chrome's PDF viewer) returns a break after every visual line, so sentences are cut in the middle of paragraphs. Feed that to an LLM, a search index or a summarizer, and the broken line structure can hurt the results. Remove every break instead, and bullet lists and tables melt into one long sentence.\n\nThis post is about how the \"Export text for AI\" feature of PDF Privacy Checker (a Windows app that finds hidden text in PDFs) decides **which line breaks are soft wraps inside a paragraph, using only character coordinates**. It is the follow-up I promised at the end of the previous post ([I rebuilt my \"export PDF text for AI\" feature three times in one day](https://dev.to/okinawasoftware/i-rebuilt-my-export-pdf-text-for-ai-feature-three-times-in-one-day-heres-what-each-version-got-3dge)).\n\nThis is not a write-up of rules that worked the first time. I measured them on real PDFs for this post, found they were broken, fixed them and measured again. The measurements led to three fixes in the app.\n\n**TL;DR**\n\nHere is a made-up store closing checklist I built for this post (the script at the end generates the PDF). This is what pdfium returns for page 1:\n\n```\nClosing Checklist\nThis sheet lists the checks to do in the last five minutes before\nclosing. If you follow them in order, you will not forget to lock\nup, and the sales total will match the cash in the register. When\nyou are done, sign the back.\n1 Count the cash in the register and compare it with the total on the\nDaily Sales Sheet. If it does not match, call the manager.\n2 Record the temperature of the fridge and the freezer. If one is\ntoo warm, circle it in red for the morning shift.\n- Manager: extension 11\n- Security company: number on the receipt\n- When in doubt, use the paper checklist\nDenomination Count Amount\n$20 bills 12 240.00\n$5 bills 9 45.00\nQuarters 40 10.00\n(rest omitted)\n```\n\nThe sample contains a title, a four-line paragraph, two numbered items with a hanging indent, three bullets, and a small cash-count table (denomination, count, amount). The breaks inside the paragraph and inside the numbered items should be joined. The breaks after the title, between the bullets and between the table rows should stay. **The goal is to tell these two kinds apart.**\n\nA PDF has no notion of \"this is the middle of a paragraph\". It only says where each glyph is drawn. The meaning of a line break has to be inferred from positions.\n\nThese are real outputs of simple rules, run on the sample PDF that the companion script generates.\n\n**1. Remove every line break**\n\n```\nClosing Checklist This sheet lists the checks … sign the back. 1 Count the cash in the register … call the manager. 2 Record the temperature … morning shift. - Manager: extension 11 - Security company: number on the receipt - When in doubt, use the paper checklist Denomination Count Amount $20 bills 12 240.00 $5 bills 9 45.00 Quarters 40 10.00 Pull the shutter …\n```\n\nThe title merges into the text, the bullets become one sentence and the whole table becomes one line. Is 45.00 the amount for the $5 bills, or the start of the next row? Neither a person nor an LLM can tell any more.\n\n**2. Join unless the line ends with a period (or ! ? :)**\n\n```\nClosing Checklist This sheet lists the checks to do in the last five minutes before closing. …\n- Manager: extension 11 - Security company: number on the receipt - When in doubt, use the paper checklist Denomination Count Amount $20 bills 12 240.00 $5 bills 9 45.00 Quarters 40 10.00 Pull the shutter down last …\n```\n\nIt fails in both directions. Titles, bullets and table rows have no period, so they all merge. And whenever a sentence happens to end exactly at the right margin inside a paragraph, the paragraph is cut there. The first failure is dangerous; the second is merely ugly. Japanese, with 。 as its full stop, fails the same way.\n\n**3. Join when the next line starts with a lowercase letter**\n\n```\n1 Count the cash in the register and compare it with the total on the\nDaily Sales Sheet. If it does not match, call the manager.\n```\n\nThis works well on this page, and its failures are mostly misses rather than false merges: it cannot join a wrap that is followed by a capital letter or a digit, such as a document name, a person's name, an amount or a date. Business text is full of those. It also has no equivalent for Japanese, which has no letter case; the closest thing there, joining when both sides are CJK characters, merged the title into the text and the table header into its first row on my Japanese version of this page. Letter case and character class tell you *how* to join (with or without a space), not *whether* to join.\n\n**4. Same baseline means same line**\n\nThis one is about the earlier step of merging fragments. PDFs from Word's character spacing or from CAD tools often draw one visual line as many small strings, and pdfium inserts a break between them. Matching baselines sounds like the fix, but a line where the number \"5\" is set in a different size from the text has different baselines. I use **vertical overlap** instead.\n\nAll four look only at the string. The meaning of a line break lives in where the lines are, not in the text.\n\npdfium gives a box for every character: `FPDFText_GetCharBox` (the tight glyph bounds) and a \"loose\" box built from the advance width and the font height. I use the union of both, because with some fonts one of them collapses (an example appears later).\n\nThe pipeline:\n\n```\nPDF\n ↓ pdfium (characters and boxes; a break after every visual line)\ncharacter boxes\n ↓ step 1: merge fragments of the same visual line\nvisual lines (left, right, top, bottom, median character height h)\n ↓ step 2: check each pair of neighbouring lines against 8 conditions\neach break: soft wrap or real break\n ↓ join (no space between CJK characters, one space in English)\ntext for the LLM (the report's full-text appendix keeps the original breaks)\n```\n\nEvery threshold is expressed in units of h, the character height, so the same rules work for any font size. h is about 1.1 times the font size.\n\n| Threshold | Step | Meaning | \n|---|---|---|\n| vertical overlap ≥ 0.5h | 1 | on the same height | \n| next fragment within 2.5h to the right | 1 | continuation of the same line | \n| gap < 0.35h | 1 | touching (join without a space) | \n| line pitch 0.5–1.8h | 2 | normal line spacing, not a paragraph gap | \n| height difference ≤ 0.3h | 2 | same text size | \n| a gap ≥ 3h inside the line | 2 | column gap (a table row) | \n| ≥ 90% of the longest line with the same left edge | 2 | the line is full | \n| line width ≥ 12h | 2 | avoid short lines matching by accident | \n| left-edge offset within 0.5h / −1.3 to −0.5h / +0.5 to 4h | 2 | aligned / 2nd line of an indented paragraph / hanging indent | \n\nNone of these come from the PDF specification. I set them once, then changed only what broke on real PDFs; the next sections say what. Also note that h is a height, so in terms of width it depends on the script: a full-width CJK character is about 0.9h wide, a Latin letter about 0.5h on average. \"12h\" is about 13 CJK characters or roughly 25 Latin ones.\n\npdfium's breaks are not always visual line boundaries. If the next piece **overlaps the current line by at least half its height and starts within 2.5h to the right**, it is a fragment of the same line: joined with no space if touching, otherwise with one space. After this step, step 2 only has to compare whole visual lines.\n\nFor two neighbouring visual lines a (above) and b (below), the break after a is a soft wrap only if **all** of the following hold. For each condition, the right column says what would be wrongly joined without it.\n\n| # | Condition | Without it, this gets joined | \n|---|---|---|\n| 1 | not vertical text | vertical CJK text (horizontal only) | \n| 2 | line pitch 0.5–1.8h | across a paragraph gap, or to a distant note | \n| 3 | nearly the same text height | a heading and its text, text and a small note | \n| 4 | the two lines overlap horizontally | different columns or cells | \n| 5 | neither line has a gap of 3h or more | table rows (a condition I added later) | \n| 6 | a is full (≥ 90% of the longest line with the same left edge) and at least 12h wide | the short last line of a paragraph and the next paragraph; short bullets of equal width | \n| 7 | b does not start with a bullet or a number | the next list item | \n| 8 | the left edges are aligned, or form the 2nd line of an indented paragraph (CJK only), or a hanging indent of a numbered item | any other indentation (quotes, nesting) | \n\nIn code (from the companion script, a shortened version of the app's logic):\n\n``` python\ndef is_wrap(a, b, lines, rules):\n    h = a[\"h\"]\n    if (a[\"T\"] - a[\"B\"]) > 2.5 * h and (a[\"R\"] - a[\"L\"]) < 2 * h:\n        return False                                      # 1. vertical text (out of scope)\n    if not (0.5 * h < a[\"B\"] - b[\"B\"] <= 1.8 * h):\n        return False                                      # 2. unusual line pitch (paragraph gap)\n    if abs(a[\"h\"] - b[\"h\"]) > 0.3 * h:\n        return False                                      # 3. different text size (heading, note)\n    if not (b[\"R\"] > a[\"L\"] and a[\"R\"] > b[\"L\"]):\n        return False                                      # 4. no horizontal overlap (column, cell)\n    if \"gap\" in rules and (max_gap(a) >= 3 or max_gap(b) >= 3):\n        return False                                      # 5. column gap (a table row)\n    block = [x[\"R\"] for x in lines if abs(x[\"L\"] - a[\"L\"]) < 1.5 * h]\n    if a[\"R\"] - a[\"L\"] < 12 * h or a[\"R\"] < 0.9 * max(block):\n        return False                                      # 6. not a full line (paragraph end)\n    first, head = b[\"text\"].lstrip(), a[\"text\"].lstrip()\n    if first[:1] in LIST_HEADS or LIST_NUM.match(first):\n        return False                                      # 7. next line starts a new item\n    dl = b[\"L\"] - a[\"L\"]\n    wide = is_wide(a[\"text\"].rstrip()[-1]) and is_wide(first[0])\n    listy = head[:1] in LIST_HEADS or bool(LIST_NUM.match(head))\n    return (abs(dl) < 0.5 * h                                         # same left edge\n            or (-1.3 * h <= dl < -0.5 * h and (wide or \"cjk\" not in rules))  # 2nd line of an indented paragraph\n            or (0.5 * h < dl <= 4 * h and listy))                     # hanging indent of a numbered item\n```\n\nCondition 6 does most of the work. Lines inside a paragraph run to the right margin; the last line of a paragraph is short. It uses the typesetting itself. The \"longest line\" is taken only among lines with the same left edge, so that a long line elsewhere on the page (a table, another column) does not distort it.\n\nThe \"2nd line of an indented paragraph\" in condition 8 is a Japanese convention: when the first line is indented by one character, the second line starts one character further left. It originally applied to any script. Why it is now CJK-only comes below.\n\n**Recap.** The whole method is four steps:\n\n- Look at character coordinates, not at the string.\n- Rebuild the visual lines from those coordinates (merging fragments of the same line).\n- Check whether two neighbouring lines are a soft wrap inside a paragraph, using eight conditions.\n- If any condition fails, do not join (keep the break).\n\nA break that is a soft wrap gets replaced in one of three ways:\n\nThe third one almost never fires, because **pdfium handles line-end hyphens itself**. When a line ends with \"-\", pdfium does not insert a break there; it glues the two lines into one word. That hyphen comes back as code 0x02 from `FPDFText_GetUnicode` and as U+FFFE from `get_text_range` (`FPDFText_IsHyphen` returns 1 for it). In the sample, \"con-\" + \"nected\" on the English page came back as `con\\ufffenected`.\n\nThis is where I found a bug in my own app. PDF Privacy Checker has a check for glyphs that extract as control characters, a sign of a tampered ToUnicode map. That check caught 0x02 and **raised a red warning on ordinary English PDFs that simply had a hyphen at the end of a line**. All five of my English articles printed from Chrome triggered it. Japanese business documents almost never end a line with \"-\", which is why my earlier tests had missed it.\n\nThe fix maps 0x02 with `FPDFText_IsHyphen` back to \"-\". The hyphen is kept, not removed: every case I found was a compound word such as off-page or macro-enabled, and removing it gives \"offpage\".\n\nThe \"2nd line of an indented paragraph\" rule used to apply to any script. On a publicly available English sample invoice (a \"Sample Invoice Template\"), it joined a table header to the next row:\n\n```\n(first rules)\nDate Specific Service/Task Time by Task Cost of Task 7/14/2023 Met with individual to draft …\n```\n\nThe next row started a little to the left of the header, exactly the shape of an indented paragraph's second line. That layout is common in English tables. A rule written for Japanese typesetting had fired on an English table, so I restricted it to CJK on both sides.\n\nThe companion script reproduces the same shape on page 2 (the heading of the example entries on the back of the sheet, and the first entry):\n\n```\n(first rules)\nExamples of entries on the back of this sheet (one line per day) 7/14/2023 Closed at 22:10, cash matched, alarm set by M. Tanaka\n(current rules)\nExamples of entries on the back of this sheet (one line per day)\n7/14/2023 Closed at 22:10, cash matched, alarm set by M. Tanaka\n```\n\nThe cost: in an English paragraph with an indented first line, the wrap after the first line is now missed. In the measurement below, that was a difference of two line breaks.\n\nFor this post I measured the rules on two kinds of PDF.\n\n**Prose PDFs**: my own 10 articles (5 Japanese, 5 English), converted to minimal HTML and printed from Chrome. Because I have the source, I can label every break automatically: paragraphs, headings, list items, table cells and single code lines are \"units\", **a break inside a unit is a soft wrap and a break between units is a real break**.\n\n**Business PDFs**: 29 real invoices, quotes, delivery notes and vehicle documents (49 pages) from my own work. There is no ground truth, so I judged every join the rules made by eye.\n\nBoth were measured with heading detection turned off. The app (v1.15.0) also uses the same coordinates to detect heading lines and keeps them out of line joining, but here I wanted to compare the wrap rules alone. Heading detection is the topic of the next post.\n\nGoing through the joins in the business PDFs, I kept seeing lines like this one, from a publicly available Canada Post sample invoice:\n\n```\n(before)\nParcels 97.98 97.98 Commercial/Smartmail Marketing 216.90 216.90 Specialized services 1,240.00 1,240.00 Shipments 168.13 168.13 Adjustments -50.00 -50.00\n```\n\nFive invoice rows on one line. **Table rows are stacked at the same width, so they pass condition 6, \"the line is full.\"** Normal line pitch, same text size, same left edge: they meet every condition for a paragraph. Japanese quotes had the same problem (\"parts 1.00 1,210 1,210\" glued to the next row).\n\nWhat separates a paragraph line from a table row is the gaps inside the line. A paragraph only has word spaces; a table row has wide gaps between columns. So I added condition 5: neither line may contain a gap of three character heights or more.\n\n```\n(after)\nParcels 97.98 97.98\nCommercial/Smartmail Marketing 216.90 216.90\nSpecialized services 1,240.00 1,240.00\nShipments 168.13 168.13\nAdjustments -50.00 -50.00\n```\n\nThe results. The two percentages point in different directions. Recall asks: of every 100 breaks that really are soft wraps, how many did it join? Precision asks: of every 100 joins it made, how many were really soft wraps? With the current rules, it joins about 90 of every 100 real wraps, and about 95 of every 100 joins are correct.\n\n**Prose PDFs (10 articles, 3,751 labelled line breaks)**\n\n| Rules | Correct joins | Missed wraps | Wrong joins | Recall | Precision | \n|---|---|---|---|---|---|\n| First rules | 1,366 | 153 | 93 | 89.9% | 93.6% | \n| Indent rule limited to CJK | 1,364 | 155 | 93 | 89.8% | 93.6% | \n| + column gap (current) | 1,364 | 155 | 69 | 89.8% | 95.2% | \n\n**Business PDFs (29 real documents, 3,603 line breaks)**\n\n| Rules | Joined as soft wraps | Correct (by eye) | Wrong | \n|---|---|---|---|\n| First rules | 125 | ~13 | ~112 | \n| Indent rule limited to CJK | 121 | ~13 | ~108 | \n| + column gap (current) | 56 | ~13 | ~43 | \n\nMissed wraps in the business PDFs were **not measured**: there is no ground truth, and judging all 3,603 breaks by eye was not realistic. The correct/wrong split is also by eye. Step 1 additionally merged 385 fragments; 379 of them are the fine background pattern of digits on two insurance certificates.\n\nWhat the numbers say:\n\nThe \"current\" row also includes a small effect from a separate fix found along the way: in PDFs saved from Chrome, the kanji \"一\" and the em dash \"—\" were wrongly flagged as tiny text. The tiny-text check now uses the size the text is actually drawn at.\n\nThe results show more missed wraps (155) than wrong joins (69). That is intended.\n\nThe two mistakes do different amounts of harm:\n\nFor text that goes to an LLM, the second is much worse. Every condition leans towards keeping the break. In other words, I deliberately prefer false negatives to false positives: leaving one soft wrap untouched is less harmful than gluing two unrelated blocks together.\n\n**Rebuilding lines and rebuilding reading order are separate problems.** This only decides whether two neighbouring lines, in the order pdfium returns them, should be joined. In a two-column layout, the order of the columns is still the PDF's drawing order.\n\nOther things the current rules cannot do:\n\n```\nPull the shutter down last and push it by hand to check that it is fully closed; if it moves at all, pull it down again and test it. Set the alarm and leave. If it beeps, turn it off and try again.\n```\n\nIt only needs pypdfium2 (about 200 lines, half of which build the test PDF). The script generates its own test PDF with the standard Helvetica font, so no font file is needed. Tested with Python 3.12 and pypdfium2 5.12.1. Run `python article10_rejoin_en.py your.pdf` to try it on your own file.\n\nFirst, what it produces. Page 1 with the current rules:\n\n```\nClosing Checklist\nThis sheet lists the checks to do in the last five minutes before closing. If you follow them in order, you will not forget to lock up, and the sales total will match the cash in the register. When you are done, sign the back.\n1 Count the cash in the register and compare it with the total on the Daily Sales Sheet. If it does not match, call the manager.\n2 Record the temperature of the fridge and the freezer. If one is too warm, circle it in red for the morning shift.\n- Manager: extension 11\n- Security company: number on the receipt\n- When in doubt, use the paper checklist\nDenomination Count Amount\n$20 bills 12 240.00\n$5 bills 9 45.00\nQuarters 40 10.00\nPull the shutter down last and push it by hand to check that it is fully closed; if it moves at all, pull it down again and test it. Set the alarm and leave. If it beeps, turn it off and try again.\n```\n\nThe paragraph and the numbered items are joined, including the wrap before \"Daily Sales Sheet\"; the bullets and the table keep their breaks. The last line is the limitation described above (two paragraphs merged). With the first rules, the four table rows become one line:\n\n```\nDenomination Count Amount $20 bills 12 240.00 $5 bills 9 45.00 Quarters 40 10.00\n# -*- coding: utf-8 -*-\n\"\"\"article10_rejoin_en.py - rejoin only the line breaks that are soft wraps inside a\nparagraph, using character coordinates from pdfium (companion script for the article;\nneeds only pypdfium2).\n\n  python article10_rejoin_en.py            - builds a test PDF and prints 3 versions\n  python article10_rejoin_en.py file.pdf   - runs the current rules on your own PDF\n\nA shortened version of the logic in PDF Privacy Checker (hidden_core 1.11.1).\nEvery threshold is written in units of h, the character height. They are not\nderived from the PDF spec; they are rules of thumb tuned on real documents.\n\"\"\"\nimport re\nimport sys\n\nimport pypdfium2 as pdfium\nimport pypdfium2.raw as R\n\nLIST_HEADS = \"・•●○■□◆◇-–—*※①②③④⑤⑥⑦⑧⑨⑩\"\nLIST_NUM = re.compile(r\"^[(（]?\\d{1,3}(?:[)）.．、:：]|\\s|$)\")\n\ndef is_wide(ch):   # full-width (kana, kanji, full-width punctuation): join without a space\n    o = ord(ch)\n    return 0x3000 <= o <= 0x30FF or 0x4E00 <= o <= 0x9FFF or 0xFF00 <= o <= 0xFFEF\n\ndef page_lines(page):\n    \"\"\"Rebuild visual lines from pdfium's characters and boxes (step 1: merge fragments).\"\"\"\n    tp = page.get_textpage()\n    n = tp.count_chars()\n    chars = [chr(R.FPDFText_GetUnicode(tp.raw, i)) for i in range(n)]\n    for i in range(n):   # pdfium returns a line-end \"-\" as 0x02 and joins the lines itself\n        if chars[i] == \"\\x02\" and R.FPDFText_IsHyphen(tp.raw, i):\n            chars[i] = \"-\"\n    boxes = [None] * n\n    for i, ch in enumerate(chars):\n        if ch.strip() and not R.FPDFText_IsGenerated(tp.raw, i):\n            t, lo = tp.get_charbox(i), tp.get_charbox(i, loose=True)\n            boxes[i] = (min(t[0], lo[0]), min(t[1], lo[1]), max(t[2], lo[2]), max(t[3], lo[3]))\n    text = \"\".join(chars)\n    lines = []\n    for m in re.finditer(r\"[^\\r\\n]+\", text):\n        bx = [boxes[i] for i in range(m.start(), m.end()) if boxes[i]]\n        if not bx:\n            continue\n        hs = sorted(b[3] - b[1] for b in bx)\n        g = {\"text\": m.group(), \"bx\": bx, \"h\": hs[len(hs) // 2],\n             \"L\": min(b[0] for b in bx), \"R\": max(b[2] for b in bx),\n             \"T\": max(b[3] for b in bx), \"B\": min(b[1] for b in bx)}\n        cur = lines[-1] if lines else None\n        if cur:   # step 1: overlaps vertically by half and continues within 2.5h to the right\n            hmin = min(cur[\"h\"], g[\"h\"])\n            overlap = min(cur[\"T\"], g[\"T\"]) - max(cur[\"B\"], g[\"B\"])\n            gap = g[\"L\"] - cur[\"R\"]\n            if overlap >= 0.5 * hmin and -0.3 * hmin < gap < 2.5 * hmin:\n                cur[\"text\"] += (\"\" if gap < 0.35 * hmin else \" \") + g[\"text\"]\n                cur[\"bx\"] += g[\"bx\"]\n                cur.update(R=max(cur[\"R\"], g[\"R\"]), T=max(cur[\"T\"], g[\"T\"]),\n                           B=min(cur[\"B\"], g[\"B\"]))\n                continue\n        lines.append(g)\n    return lines\n\ndef max_gap(line):\n    \"\"\"Largest horizontal gap between neighbouring character boxes, in units of h.\"\"\"\n    bx = sorted(line[\"bx\"])\n    return max([(bx[k + 1][0] - bx[k][2]) / line[\"h\"] for k in range(len(bx) - 1)] or [0])\n\ndef is_wrap(a, b, lines, rules):\n    \"\"\"Is the break after line a a soft wrap? True only if none of 1-7 applies and the\n    left edges match one of three shapes (when in doubt, keep the break).\"\"\"\n    h = a[\"h\"]\n    if (a[\"T\"] - a[\"B\"]) > 2.5 * h and (a[\"R\"] - a[\"L\"]) < 2 * h:\n        return False                                      # 1. vertical text (out of scope)\n    if not (0.5 * h < a[\"B\"] - b[\"B\"] <= 1.8 * h):\n        return False                                      # 2. unusual line pitch (paragraph gap)\n    if abs(a[\"h\"] - b[\"h\"]) > 0.3 * h:\n        return False                                      # 3. different text size (heading, note)\n    if not (b[\"R\"] > a[\"L\"] and a[\"R\"] > b[\"L\"]):\n        return False                                      # 4. no horizontal overlap (column, cell)\n    if \"gap\" in rules and (max_gap(a) >= 3 or max_gap(b) >= 3):\n        return False                                      # 5. column gap (a table row)\n    block = [x[\"R\"] for x in lines if abs(x[\"L\"] - a[\"L\"]) < 1.5 * h]\n    if a[\"R\"] - a[\"L\"] < 12 * h or a[\"R\"] < 0.9 * max(block):\n        return False                                      # 6. not a full line (paragraph end)\n    first, head = b[\"text\"].lstrip(), a[\"text\"].lstrip()\n    if first[:1] in LIST_HEADS or LIST_NUM.match(first):\n        return False                                      # 7. next line starts a new item\n    dl = b[\"L\"] - a[\"L\"]\n    wide = is_wide(a[\"text\"].rstrip()[-1]) and is_wide(first[0])\n    listy = head[:1] in LIST_HEADS or bool(LIST_NUM.match(head))\n    return (abs(dl) < 0.5 * h                                         # same left edge\n            or (-1.3 * h <= dl < -0.5 * h and (wide or \"cjk\" not in rules))  # 2nd line of an indented paragraph\n            or (0.5 * h < dl <= 4 * h and listy))                     # hanging indent of a numbered item\n\ndef rejoin(lines, rules):\n    out = lines[0][\"text\"] if lines else \"\"\n    for a, b in zip(lines, lines[1:]):\n        if not is_wrap(a, b, lines, rules):\n            out += \"\\n\" + b[\"text\"]\n        elif is_wide(out.rstrip()[-1]) and is_wide(b[\"text\"].lstrip()[0]):\n            out = out.rstrip() + b[\"text\"].lstrip()          # Japanese: no space\n        elif out.endswith(\"-\") and b[\"text\"][:1].islower():\n            out = out[:-1] + b[\"text\"]   # undo hyphenation (a line-end \"-\" pdfium did not mark)\n        else:\n            out = out.rstrip() + \" \" + b[\"text\"].lstrip()     # English: one space\n    return out\n\n# ---- test PDF (raw PDF, standard Helvetica; nothing is embedded) ----\ndef build_pdf(path):\n    en, en2 = [], []\n\n    def E(x, y, s, size=10, page=None):\n        s = s.replace(\"\\\\\", \"\\\\\\\\\").replace(\"(\", \"\\\\(\").replace(\")\", \"\\\\)\")\n        (en if page is None else page).append(f\"BT /F1 {size} Tf {x} {y} Td ({s}) Tj ET\")\n\n    E(72, 780, \"Closing Checklist\", 16)\n    for k, s in enumerate([\"This sheet lists the checks to do in the last five minutes before\",\n                           \"closing. If you follow them in order, you will not forget to lock\",\n                           \"up, and the sales total will match the cash in the register. When\",\n                           \"you are done, sign the back.\"]):\n        E(72, 750 - 14 * k, s)\n    for y, no, a, b in [(685, \"1\", \"Count the cash in the register and compare it with the total on the\",\n                         \"Daily Sales Sheet. If it does not match, call the manager.\"),\n                        (651, \"2\", \"Record the temperature of the fridge and the freezer. If one is\",\n                         \"too warm, circle it in red for the morning shift.\")]:\n        E(72, y, no); E(86, y, a); E(86, y - 14, b)   # hanging indent\n    for k, s in enumerate([\"- Manager: extension 11\", \"- Security company: number on the receipt\",\n                           \"- When in doubt, use the paper checklist\"]):\n        E(72, 610 - 14 * k, s)\n    for k, (a, b, c) in enumerate([(\"Denomination\", \"Count\", \"Amount\"),     # cash count\n                                   (\"$20 bills\", \"12\", \"240.00\"),\n                                   (\"$5 bills\", \"9\", \"45.00\"),\n                                   (\"Quarters\", \"40\", \"10.00\")]):\n        E(72, 555 - 14 * k, a); E(230, 555 - 14 * k, b); E(330, 555 - 14 * k, c)\n    # limitation: a paragraph whose last line happens to be full, followed by a paragraph\n    # with no indent and no extra space\n    E(72, 480, \"Pull the shutter down last and push it by hand to check that it\")\n    E(72, 466, \"is fully closed; if it moves at all, pull it down again and test it.\")\n    E(72, 452, \"Set the alarm and leave. If it beeps, turn it off and try again.\")\n\n    E(72, 780, \"Notes\", 16, en2)\n    for k, s in enumerate([\"Deliveries that arrive after closing must not be left at the back\",\n                           \"door or in front of the shop. The store cannot take any\",\n                           \"responsibility for goods that are left outside overnight, so ask\",\n                           \"the driver to take them back.\"]):\n        E(72, 750 - 14 * k, s, page=en2)\n    E(72, 680, \"The alarm panel shows a green light only when every door is con-\", page=en2)\n    E(72, 666, \"nected to the system; a red light means that one is still open.\", page=en2)\n    E(82, 630, \"Examples of entries on the back of this sheet (one line per day)\", page=en2)\n    E(72, 616, \"7/14/2023 Closed at 22:10, cash matched, alarm set by M. Tanaka\", page=en2)\n\n    objs = [\"<< /Type /Catalog /Pages 2 0 R >>\", \"\",\n            \"<< /Type /Font /Subtype /Type1 /BaseFont /Helvetica /Encoding /WinAnsiEncoding >>\"]\n    kids = []\n    for ops in (en, en2):\n        data = \"\\n\".join(ops).encode(\"latin-1\")\n        objs.append(f\"<< /Length {len(data)} >>\\nstream\\n\".encode() + data + b\"\\nendstream\")\n        objs.append(f\"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 595 842] /Contents {len(objs)} 0 R \"\n                    \"/Resources << /Font << /F1 3 0 R >> >> >>\")\n        kids.append(len(objs))\n    objs[1] = f\"<< /Type /Pages /Kids [{' '.join(f'{k} 0 R' for k in kids)}] /Count {len(kids)} >>\"\n    out, offs = b\"%PDF-1.7\\n\", []\n    for k, o in enumerate(objs, 1):\n        offs.append(len(out))\n        out += f\"{k} 0 obj\\n\".encode() + (o if isinstance(o, bytes) else o.encode()) + b\"\\nendobj\\n\"\n    xref = len(out)\n    out += f\"xref\\n0 {len(objs) + 1}\\n0000000000 65535 f \\n\".encode()\n    out += b\"\".join(f\"{o:010d} 00000 n \\n\".encode() for o in offs)\n    out += f\"trailer << /Size {len(objs) + 1} /Root 1 0 R >>\\nstartxref\\n{xref}\\n%%EOF\\n\".encode()\n    open(path, \"wb\").write(out)\n\nif __name__ == \"__main__\":\n    sys.stdout.reconfigure(encoding=\"utf-8\")\n    if len(sys.argv) > 1:\n        for page in pdfium.PdfDocument(sys.argv[1]):\n            print(rejoin(page_lines(page), {\"cjk\", \"gap\"}), end=\"\\n\\n\")\n        sys.exit()\n    build_pdf(\"article10_sample_en.pdf\")\n    doc = pdfium.PdfDocument(\"article10_sample_en.pdf\")\n    for no, page in enumerate(doc, 1):\n        lines = page_lines(page)\n        print(f\"===== page {no}: pdfium line breaks as-is =====\")\n        print(page.get_textpage().get_text_range().replace(\"\\r\\n\", \"\\n\"))\n        print(f\"===== page {no}: first rules (indent rule for any script, no column-gap check) =====\")\n        print(rejoin(lines, set()))\n        print(f\"===== page {no}: current rules =====\")\n        print(rejoin(lines, {\"cjk\", \"gap\"}))\n        print()\n```\n\nMeasuring for this post led to three fixes in PDF Privacy Checker: table rows are no longer glued together, a line-end hyphen no longer triggers a false control-character warning, and \"一\" and \"—\" in PDFs saved from Chrome are no longer flagged as tiny text. All three are in v1.15.0, which is now on the Store.\n\nPDF Privacy Checker is on the Microsoft Store (detection is free; the text export for AI is part of a paid add-on). Files never leave your PC; it works fully offline. The source code is not public.\n\n[https://apps.microsoft.com/detail/9PLRJHFTPS53?hl=en-us&gl=US](https://apps.microsoft.com/detail/9PLRJHFTPS53?hl=en-us&gl=US)\n\nNext, I plan to write about using the same coordinates to detect heading lines, so the text for AI keeps the document's structure.\n\n**About this article** — The implementation and the writing were done together with Claude (Anthropic's AI). Most of the prose was drafted by Claude and checked by me. The decisions about which errors to fix in the product and which to leave as limitations, and the real business documents, are mine.\n\n**Okinawa Software Lab.** I lead in-house digital transformation at a small company in Okinawa, Japan. I build the tools we need ourselves, and I publish PDF apps on the Microsoft Store that follow the same principle: everything happens on your own PC.", "url": "https://wpnews.pro/news/which-line-breaks-in-extracted-pdf-text-are-real-rejoining-wrapped-lines-from", "canonical_source": "https://dev.to/okinawasoftware/which-line-breaks-in-extracted-pdf-text-are-real-rejoining-wrapped-lines-from-character-182p", "published_at": "2026-09-27 09:38:55+00:00", "updated_at": "2026-09-27 10:01:06.401594+00:00", "lang": "en", "topics": ["ai-tools", "natural-language-processing", "developer-tools", "ai-products"], "entities": ["PDF Privacy Checker", "pdfium", "Chrome", "Okinawa Software"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/which-line-breaks-in-extracted-pdf-text-are-real-rejoining-wrapped-lines-from", "markdown": "https://wpnews.pro/news/which-line-breaks-in-extracted-pdf-text-are-real-rejoining-wrapped-lines-from.md", "text": "https://wpnews.pro/news/which-line-breaks-in-extracted-pdf-text-are-real-rejoining-wrapped-lines-from.txt", "jsonld": "https://wpnews.pro/news/which-line-breaks-in-extracted-pdf-text-are-real-rejoining-wrapped-lines-from.jsonld"}}