Page headers and footers in RAG chunks: what I found in 25 government PDFs

When you pull the text out of a PDF, the running header, the footer and the page number come out as ordinary lines, mixed in with the body. Nothing marks them as different. A RAG pipeline then cuts that text into chunks, and two things happen. The same footer lands in chunk after chunk, which makes those chunks look alike to the embedding model. And any chunk that crosses a page break gets a header in the middle of a sentence.

I wanted to know how much of a real document this is, so I measured it on 25 public PDFs from US federal agencies. Disclosure up front: I make Agnos, a Windows app from Gidion that cleans documents before they go into AI tools, and the study measured what it removed. What it got wrong is in here too. That turned out to be the most useful part.

page.extract_text() with its default options. That's 64,682 non-empty lines and 3,509,754 characters.rejected.jsonl) of Agnos 2.3.2, run with its RAG / Retrieval Corpus purpose and default settings. Agnos removed 2,063 lines as page furniture, from 24 of the 25 documents. That's 50,928 characters, 1.45% of the plain text. Per document, the median was 1.32%, and the range ran from 0% to 3.17%.

By volume, that's small. The problem is where it sits. The median was 1.18 lines of furniture per page. Furniture was removed from every text page of 3 documents, and from at least 90% of the text pages in 15. It's at every page break, so it lands in the middle of any chunk that crosses one.

The most common kinds (one line can carry more than one label):

Some real ones: "Page 1 of 89" from NASA's software engineering requirements. "PAGE 3 OF 34 1333ND19QNB680025" from a NIST solicitation. And a line from an IRS printer's proof that starts "Page 2 of 59 Fileid: …" and runs on into a file path and a timestamp. None of that helps anyone answer a question.

You don't need any app to see this. The idea is simple. Compare lines with their digits masked, so "Page 3 of 12" and "Page 4 of 12" count as the same line. Then count how many pages each one is on.

import re
import sys
from collections import defaultdict

import pdfplumber  # pip install pdfplumber

def shape(line):
    return re.sub(r"\d+", "#", " ".join(line.lower().split()))

with pdfplumber.open(sys.argv[1]) as pdf:
    pages = [(page.extract_text() or "").splitlines() for page in pdf.pages]

on_pages = defaultdict(set)  # shape -> the pages it is on
first_seen = {}              # shape -> the line as first printed
for number, lines in enumerate(pages, start=1):
    for line in lines:
        if line.strip():
            on_pages[shape(line)].add(number)
            first_seen.setdefault(shape(line), line.strip())

need = max(3, 0.3 * len(pages))  # on 3+ pages, and on 30%+ of them
for s, seen in sorted(on_pages.items(), key=lambda kv: -len(kv[1])):
    if len(seen) >= need:
        print(f"on {len(seen)} of {len(pages)} pages: {first_seen[s]!r}")

Run it as python repeated_lines.py your.pdf. On a four-page sample handbook I made for my guides, it prints:

on 4 of 4 pages: 'Northwind Tools | Staff Handbook | March 2026'
on 4 of 4 pages: 'Confidential - Internal Use Only'
on 4 of 4 pages: 'Page 1 of 4'

That's the easy case. Real documents are where it gets interesting.

Masking the digits is what makes this work. It's also what makes it wrong:

import re

def shape(line):
    return re.sub(r"\d+", "#", " ".join(line.lower().split()))

pairs = [
    ("Page 3 of 12", "Page 4 of 12"),      # furniture: these should match
    ("Table A-1.", "Table A-2."),          # two different table labels
    ("1-800-555-0100", "1-800-555-0199"),  # two different phone numbers
]
for a, b in pairs:
    print(f"{a!r} = {b!r}? {shape(a) == shape(b)}, both {shape(a)!r}")
'Page 3 of 12' = 'Page 4 of 12'? True, both 'page # of #'
'Table A-1.' = 'Table A-2.'? True, both 'table a-#.'
'1-800-555-0100' = '1-800-555-0199'? True, both '#-#-#-#'

(The phone numbers are made-up 555 numbers.)

The study ran into exactly this. We read every removed line, and checked its page when the line alone wasn't clear. 200 of the 2,063 looked like content. There were 36 table cells, 34 column headings repeated on each page, 22 figure and table notes, 21 table labels and 17 phone numbers, plus smaller groups. In the Census income report, "Table A-1." was removed while the table's title on the next line was kept. In NASA's requirements, "3.1.10 042 The project manager shall" was removed at a page edge. The rest of the sentence stayed, so that requirement lost its number.

None of those lines is furniture. They have the same shape as furniture, at the edge of the page, on 3 or more pages.

The safer way to use this rule is to look only at the edges of each page. Agnos looks at the first 2 and last 2 lines. There it removes page markers, page numbers that count up with the pages, and short lines (80 characters or fewer, not sentences) that repeat at the edges of 3 or more pages. That protects the body of the page. But it also means Agnos kept about as much repeated text as it removed.

The biggest case: NASA NPR 7150.2D prints a four-line notice at the foot of all 89 pages. In the order Agnos reads the page, the page number and a header line come after the notice. Those two were removed, so the notice sits just outside the window. All four lines stayed: 26,433 characters, 12.8% of that document's plain text. An IRS proof notice stayed too. At 121 characters, it's over the length limit.

A common rule is "repeated on at least 30% of pages". Medicare & You 2027 changes its running header with each section. No single line is on 30% of its 128 pages, so that rule finds nothing at all. Agnos needs a line on only 3 pages, and it removed a header or page number from 104 of the document's 124 text pages. Neither threshold is right for every file.

One more thing to watch: rotated pages. The Census report has 18 pages of rotated tables. Agnos's reader returned each of those pages as one or two long lines, so no furniture could be found on them. And pdfplumber read the rotated tables backwards: "emocni" for "income".

Agnos does this for whole folders of text-based PDFs, Word, Excel, PowerPoint and web pages, on your own Windows 10 or 11 PC. It lists what it removed with the rule behind each removal. Review shows up to 500 removals per file, and the Everything export has the full log as rejected.jsonl. Its limits: it doesn't do OCR, so scanned pages are skipped and named in its warnings. It runs on Windows only, and the Store version has no command line yet. If you'd rather stay in Python, Docling is free and open source. It labels page headers and footers and leaves them out of its default exports.

The study page has every document, the method, all 200 lines that looked like content, and two scripts to reproduce it. The per-document numbers, 46 columns of them, are in a CSV under CC BY 4.0: https://agnos.cloud/guides/page-furniture-study/ To try this on your own PDFs without code, there's a free tool that runs in your browser and uploads nothing: https://agnos.cloud/tools/pdf-repeated-lines/

How do you check what your cleaning step removed? I'd like to know what other people do.

Drafted with help from an AI assistant (Claude). The numbers were checked against the study page, and both scripts were run on the sample PDF, before publishing.