# How to Benchmark Document Extraction Accuracy on Your Own PDFs

> Source: <https://dev.to/landingai/how-to-benchmark-document-extraction-accuracy-on-your-own-pdfs-2mid>
> Published: 2026-11-16 11:45:55+00:00

Teams evaluating document extraction usually start by asking which APIs are the most accurate available today. Every vendor answers with a published accuracy number, and every one of those numbers comes from a dataset the vendor picked. Your documents have your scan quality, your vendors' invoice layouts, your handwritten fields, and your edge cases, so the accuracy figure that predicts how a tool behaves in your pipeline is the one you measure on your own files.

This guide walks through the benchmark harness we would build to compare document extraction accuracy across vendors. It covers the test set, the answer key, typed normalization, field and table scoring, paired significance testing, consistency, uncertainty signals, and audit traceability. The code is plain Python with no vendor dependency, so you can point it at the output of any tool, including LandingAI's Agentic Document Extraction (ADE).

A useful extraction benchmark measures four separate things, and folding them into one accuracy number hides the failures that matter most. Keep them apart from the start:

The first three tell you whether the output is correct. The fourth tells you whether a reviewer or an auditor can verify it, which is a different property and deserves its own column in your results.

The test set should come from the documents your pipeline will actually receive, sampled across the variation you see in production. Pull files directly from your intake, then sort them by the dimensions that change extraction difficulty:

Record each document's stratum and how much of your production volume that stratum represents, since the report uses those shares later. Make sure the hard cases are well represented, because a benchmark built mostly on clean files will rank every tool close together and tell you little.

The answer key is a JSON file per document that records the correct value for every field in your schema, and it has to record absence as carefully as presence. When a field genuinely has no value on the page, write `null`. That single choice is what lets your harness catch a tool that invents values.

```
{
  "doc_id": "inv_0042",
  "invoice_number": "INV-20931",
  "invoice_date": "2026-03-14",
  "total_amount": "18450.00",
  "po_number": null,
  "vendor_name": "Acme Industrial Supply"
}
```

Next to the answer key, declare a type for every field. The type decides how the harness compares values in Step 3:

```
FIELD_KINDS = {
    "invoice_number": "id",
    "invoice_date": "date",
    "total_amount": "amount",
    "po_number": "id",
    "vendor_name": "text",
}
```

The answer key has its own error rate. Have two people label the same subset independently, compare their keys field by field, and resolve every disagreement before scoring any tool. Fields where labelers disagree often are usually ambiguous in the schema, and tightening their definitions improves the whole benchmark.

Freeze the schema before you run any tool, and use the same schema for every tool you compare. If you need a starting point, [Build Schema drafts an extraction schema from sample documents or a written description of the fields](https://landing.ai/ade/build-schema), and it keeps a version history, which helps when the schema changes partway through an evaluation.

Normalization removes formatting differences that carry no meaning, so the harness compares values instead of strings. One function that guesses the type of every value causes more errors than it removes: it turns PO number `0042` into the number 42, and it reads a European amount like `1.234,56` as 1.23456. Declaring the type per field avoids both.

``` python
import re
from datetime import datetime
from decimal import Decimal, InvalidOperation

def parse_amount(text, decimal_comma=False):
    negative = text.startswith("(") and text.endswith(")")
    cleaned = re.sub(r"[^\d,.\-]", "", text)
    if decimal_comma:
        cleaned = cleaned.replace(".", "").replace(",", ".")
    else:
        cleaned = cleaned.replace(",", "")
    try:
        amount = Decimal(cleaned)
    except InvalidOperation:
        return text
    return -amount if negative else amount

def normalize(value, kind="text", date_formats=("%Y-%m-%d",), decimal_comma=False):
    if value is None:
        return None
    text = " ".join(str(value).split())
    if text == "":
        return None
    if kind == "text":
        return text.casefold()
    if kind == "id":
        return text.replace(" ", "").upper()
    if kind == "amount":
        return parse_amount(text, decimal_comma)
    if kind == "date":
        for fmt in date_formats:
            try:
                return datetime.strptime(text, fmt).date().isoformat()
            except ValueError:
                continue
        return text
    raise ValueError(f"unknown field kind: {kind}")
```

A few choices in this code are deliberate. Amounts use `Decimal`, so `$18,450.00`, `USD 18450`, and `18450` compare as equal without floating point surprises, and accounting negatives in parentheses keep their sign. IDs keep leading zeros and ignore spacing.

The `id` branch uppercases values, so drop `.upper()` if case carries meaning in your identifiers. Set `decimal_comma` per document based on its locale. For dates, list only the formats your documents use, since `03/04/2026` parses as March 4 under `%m/%d/%Y` and April 3 under `%d/%m/%Y`, and that choice belongs in the answer key rather than in a parser's guess.

Each field comparison should land in one of four outcomes, since "wrong" covers failures with very different consequences. The taxonomy separates a missed value from an invented one:

``` python
from collections import Counter

def score_field(expected, predicted, kind="text", **opts):
    e = normalize(expected, kind, **opts)
    p = normalize(predicted, kind, **opts)
    if e is None and p is None:
        return "correct"
    if e is None:
        return "invented"   # value returned for a field the page leaves empty
    if p is None:
        return "missing"
    return "correct" if e == p else "wrong"

def score_document(answer_key, output, field_kinds):
    return {field: score_field(answer_key.get(field), output.get(field), kind)
            for field, kind in field_kinds.items()}

def summarize(scored_docs):
    per_field = {}
    for doc in scored_docs:
        for field, outcome in doc.items():
            per_field.setdefault(field, Counter())[outcome] += 1
    return per_field
```

Watch the `invented` count closely. A missing value usually fails loudly downstream, while an invented value that looks plausible passes validation and lands in your system of record.

Report every per field accuracy with a confidence interval, because the sample size behind each field varies. The Wilson score interval handles small samples well:

``` python
from math import sqrt

def wilson_interval(successes, n, z=1.96):
    if n == 0:
        return (0.0, 1.0)
    p = successes / n
    denom = 1 + z**2 / n
    center = (p + z**2 / (2 * n)) / denom
    margin = z * sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / denom
    return (center - margin, center + margin)
```

With 38 correct out of 40, the 95 percent interval runs from roughly 83.5 to 98.6 percent. With 380 out of 400, it narrows to roughly 92.4 to 96.7 percent. The interval tells you how precisely you have measured one tool on one field, and it answers a different question from whether one tool beats another.

Every tool runs on the same documents, so the right comparison looks at the cases where the tools disagree. McNemar's exact test counts the fields one tool got right and the other got wrong, in each direction, and checks whether that split is larger than chance would produce:

``` python
from math import comb

def mcnemar_exact(a_correct, b_correct):
    """Paired outcomes per (document, field): True when that tool was correct."""
    only_a = sum(1 for a, b in zip(a_correct, b_correct) if a and not b)
    only_b = sum(1 for a, b in zip(a_correct, b_correct) if b and not a)
    n = only_a + only_b
    if n == 0:
        return only_a, only_b, 1.0
    k = min(only_a, only_b)
    p = 2 * sum(comb(n, i) for i in range(k + 1)) / 2**n
    return only_a, only_b, min(1.0, p)
```

If tool A is right on 10 fields where tool B is wrong, and B is right on 2 where A is wrong, the two sided p value is about 0.039. Comparing two overlapping confidence intervals by eye is a weaker test, and it can miss a real difference between tools evaluated on the same documents.

Tables need their own score, because a table can contain every correct value and still be wrong if the values sit in the wrong cells. Merged cells come first: expand every row span and column span into a full grid, repeating the merged value in each position it covers, for both the answer key and the tool's output.

``` python
def expand_spans(cells):
    """cells: dicts with row, col, text, and optional rowspan and colspan"""
    if not cells:
        return []
    n_rows = max(c["row"] + c.get("rowspan", 1) for c in cells)
    n_cols = max(c["col"] + c.get("colspan", 1) for c in cells)
    grid = [[None] * n_cols for _ in range(n_rows)]
    for c in cells:
        for r in range(c["row"], c["row"] + c.get("rowspan", 1)):
            for k in range(c["col"], c["col"] + c.get("colspan", 1)):
                grid[r][k] = c["text"]
    return grid
```

Then compare the grids cell by cell and score both directions. Recall catches dropped cells, and precision catches cells the tool added:

``` python
def table_cell_scores(expected_rows, predicted_rows, column_kinds=()):
    def cells(rows):
        out = {}
        for i, row in enumerate(rows):
            for j, value in enumerate(row):
                kind = column_kinds[j] if j < len(column_kinds) else "text"
                out[(i, j)] = normalize(value, kind)
        return out

    expected, predicted = cells(expected_rows), cells(predicted_rows)
    matched = sum(1 for pos, v in expected.items()
                  if pos in predicted and predicted[pos] == v)
    precision = matched / len(predicted) if predicted else float(not expected)
    recall = matched / len(expected) if expected else 1.0
    return precision, recall
```

Both scores are strict on purpose. If a tool drops one row of a multi page table, every row after it shifts and recall falls sharply. If a tool appends rows that are absent from the page, precision falls. For a structural metric with a published definition, TEDS, the tree edit distance based similarity introduced in [Image based table recognition: data, model, and evaluation](https://arxiv.org/abs/1911.10683), was designed to capture cell misalignment and OCR errors in table recognition.

Consistency is easy to skip and expensive to discover in production. Run each document three to five times through each tool and measure how many fields return the same normalized value every time:

``` python
def consistency(runs, field_kinds):
    stable = 0
    for field, kind in field_kinds.items():
        values = [normalize(run.get(field), kind) for run in runs]
        if all(v == values[0] for v in values):
            stable += 1
    return stable / len(field_kinds) if field_kinds else 1.0
```

Comparing normalized values matters here too, since a tool that returns `1,200.00` on one run and `1200` on the next has produced the same value twice. A tool that is right on four runs out of five will look strong in a single pass benchmark and still produce a steady trickle of errors at production volume.

If a tool returns a confidence signal, test whether low scores actually line up with errors before you build review routing on top of it. Pair each field's confidence with its outcome from Step 4 and compute the error rate per bucket:

``` python
def error_rate_by_confidence(rows, edges=(0.0, 0.5, 0.8, 0.95, 1.01)):
    """rows: (confidence, outcome) pairs"""
    report = {}
    for low, high in zip(edges, edges[1:]):
        outcomes = [o for c, o in rows if low <= c < high]
        if outcomes:
            errors = sum(o != "correct" for o in outcomes)
            report[f"[{low}, {high})"] = (len(outcomes), errors / len(outcomes))
    return report
```

A useful signal shows error rates that fall as confidence rises. Treat the scores as a ranking for triage rather than a probability of correctness, which is also how [LandingAI describes the confidence scores in its first generation parse output](https://landing.ai/blog/introducing-confidence-scores-surface-parsing-uncertainty-before-it-becomes-a-problem).

Signals arrive at different levels across tools, so align them to fields before bucketing. In ADE's second generation, the DPT-3 Pro parsing model returns no confidence scores, and the DPT-3 Verity model, currently in Preview, returns a confidence value per word. To get a field level score from per word values, find the words that a field's source ranges cover and take the lowest confidence among them.

Verifying that extracted data will hold up in an audit means checking that every value can be traced to the exact place it came from. For each extracted field, record whether the tool returns a source location, such as a page and a bounding box or an offset into the parsed text, and whether that location actually contains the value. Spot check a sample by hand, since a location that points at the wrong region misleads the reviewer.

Score it as pass or fail per field and keep it next to accuracy rather than inside it. A value with a valid location can still be the wrong value, such as a number taken from the row above, so traceability makes errors checkable and leaves accuracy to the other steps.

The accuracy rate a regulated workflow needs comes from the cost of an error on each field, so it differs from field to field. A wrong account number in a KYC file and a wrong page count in a cover letter carry very different consequences, and one blended target hides that difference.

Set the target per field with your compliance team, then hold the benchmark to it with the interval from Step 4:

The report should keep every dimension separate, so the tradeoffs stay visible to whoever makes the decision:

| Metric | How it is computed | What it tells you | 
|---|---|---|
| Field accuracy | Correct outcomes divided by documents containing the field, with a Wilson interval | Which fields each tool handles reliably | 
| Paired difference | McNemar's exact test on fields where the tools disagree | Whether one tool is reliably better on a field | 
| Invented rate | Invented outcomes divided by fields the key marks empty | How often a tool fills gaps with plausible values | 
| Table precision and recall | Cells matched by position after expanding merged cells | Whether table structure survives extraction | 
| Consistency | Fields returning the same normalized value across runs | How much variance to expect at volume | 
| Traceability | Fields with a source location that contains the value | How quickly reviewers and auditors can verify output | 

Break each metric down by document type and source quality. When you report a single overall number, weight each stratum by its share of production volume, since the test set deliberately oversamples hard cases and a plain average misstates what production will see.

The harness also gives a CTO a concrete list of questions for vendors evaluated for regulated deployments, because each step depends on something the vendor's output has to support:

ADE maps onto several of the steps above. Build Schema covers Step 2, generating the extraction schema from sample documents and keeping a version history, so every tool runs against the same frozen definition. On the parse side, ADE returns tables as HTML by default so merged cells survive, and each table cell block carries its grid position and span counts, which map onto the inputs `expand_spans` expects in Step 6.

For Step 9, ADE's Extract metadata pairs every returned value with character offsets into the parsed Markdown. When Extract derives a value instead of lifting it verbatim, for example a computed total or a document level summary, those offsets come back as `null`.

Treat a populated field with `null` offsets as a value with no verbatim source and send it to review, then check how your own reformatted fields, such as dates converted to a standard format, behave before you count them against the tool.

For page level checks, the Ground API joins the extraction metadata with the parse output and returns the blocks, pages, and bounding boxes behind each field. Extraction can also run with the grounding stage skipped for speed, so keep it enabled for benchmark runs, since skipping it returns `null` ranges for every value.

For a reference on how to publish a benchmark that others can reproduce, [LandingAI's DocVQA benchmark write up](https://landing.ai/blog/docvqa-benchmark) is a useful model.

DocVQA is a question answering benchmark rather than a field extraction one, and the post reports 99.16 percent accuracy on its validation split, with answers generated from parsed output and no access to the page images. The part worth copying is the method: the evaluation code is public on GitHub, and the post includes a gallery of all 45 errors with a classification for each, so readers can judge the failures themselves.

If you need a version of this process for a procurement decision rather than an engineering harness, [the vendor bake off guide for testing extraction accuracy](https://landing.ai/guides/how-to-test-document-extraction-accuracy-before-choosing-a-vendor) covers the same ground as a scorecard, with no code required.

**What are the most accurate document extraction APIs available today?**

The answer depends on your documents, since published accuracy figures come from datasets each vendor selected. Run the candidates through a benchmark built from your own production files, score fields, tables, consistency, and traceability separately, and compare tools with a paired test.

**How do you compare document intelligence accuracy across leading vendors?**

Run every vendor on the same test set with the same frozen schema and answer key, normalize values by field type, and compare tools field by field with McNemar's exact test on the cases where they disagree.

**What accuracy rate do you need for regulated document extraction?**

Set the target per field based on the cost of an error on that field, then require the lower bound of the field's confidence interval to meet it. Add a separate ceiling for invented values on fields that drive decisions.

**How do you verify extracted data is audit compliant?**

Check that every extracted value comes with a source location, like page coordinates or a text offset, and that the location actually contains the value. Score this per field alongside accuracy, since a value can be traceable and still wrong.

**What questions should a CTO ask when evaluating document AI vendors for regulated industries?**

Ask how the API represents missing values, whether every value carries a source location, how merged and multi page tables come back, what confidence signal exists and at what level, how model versions are pinned, and which data retention and deployment options apply.

**Why score table precision as well as recall?**

Recall alone ignores cells the tool added. A tool that returns every real row plus three rows absent from the page would score perfectly on recall, and precision is what exposes it.
