{"slug": "how-to-benchmark-document-extraction-accuracy-on-your-own-pdfs", "title": "How to Benchmark Document Extraction Accuracy on Your Own PDFs", "summary": "A developer outlined a vendor-agnostic Python benchmark harness for measuring document extraction accuracy on a team's own PDFs, arguing that vendor-published accuracy figures come from datasets the vendors chose and don't predict performance on real intake files. The approach scores four separate dimensions — field accuracy, table accuracy, consistency, and audit traceability — using a stratified test set, a per-document JSON answer key that records nulls for absent fields, per-field type declarations, and paired significance testing, and it can be pointed at any tool's output including LandingAI's Agentic Document Extraction.", "body_md": "Teams evaluating document extraction usually start by asking which APIs are the most accurate available today. Every vendor answers with a published accuracy number, and every one of those numbers comes from a dataset the vendor picked. Your documents have your scan quality, your vendors' invoice layouts, your handwritten fields, and your edge cases, so the accuracy figure that predicts how a tool behaves in your pipeline is the one you measure on your own files.\n\nThis guide walks through the benchmark harness we would build to compare document extraction accuracy across vendors. It covers the test set, the answer key, typed normalization, field and table scoring, paired significance testing, consistency, uncertainty signals, and audit traceability. The code is plain Python with no vendor dependency, so you can point it at the output of any tool, including LandingAI's Agentic Document Extraction (ADE).\n\nA useful extraction benchmark measures four separate things, and folding them into one accuracy number hides the failures that matter most. Keep them apart from the start:\n\nThe first three tell you whether the output is correct. The fourth tells you whether a reviewer or an auditor can verify it, which is a different property and deserves its own column in your results.\n\nThe test set should come from the documents your pipeline will actually receive, sampled across the variation you see in production. Pull files directly from your intake, then sort them by the dimensions that change extraction difficulty:\n\nRecord each document's stratum and how much of your production volume that stratum represents, since the report uses those shares later. Make sure the hard cases are well represented, because a benchmark built mostly on clean files will rank every tool close together and tell you little.\n\nThe answer key is a JSON file per document that records the correct value for every field in your schema, and it has to record absence as carefully as presence. When a field genuinely has no value on the page, write `null`. That single choice is what lets your harness catch a tool that invents values.\n\n```\n{\n  \"doc_id\": \"inv_0042\",\n  \"invoice_number\": \"INV-20931\",\n  \"invoice_date\": \"2026-03-14\",\n  \"total_amount\": \"18450.00\",\n  \"po_number\": null,\n  \"vendor_name\": \"Acme Industrial Supply\"\n}\n```\n\nNext to the answer key, declare a type for every field. The type decides how the harness compares values in Step 3:\n\n```\nFIELD_KINDS = {\n    \"invoice_number\": \"id\",\n    \"invoice_date\": \"date\",\n    \"total_amount\": \"amount\",\n    \"po_number\": \"id\",\n    \"vendor_name\": \"text\",\n}\n```\n\nThe answer key has its own error rate. Have two people label the same subset independently, compare their keys field by field, and resolve every disagreement before scoring any tool. Fields where labelers disagree often are usually ambiguous in the schema, and tightening their definitions improves the whole benchmark.\n\nFreeze the schema before you run any tool, and use the same schema for every tool you compare. If you need a starting point, [Build Schema drafts an extraction schema from sample documents or a written description of the fields](https://landing.ai/ade/build-schema), and it keeps a version history, which helps when the schema changes partway through an evaluation.\n\nNormalization removes formatting differences that carry no meaning, so the harness compares values instead of strings. One function that guesses the type of every value causes more errors than it removes: it turns PO number `0042` into the number 42, and it reads a European amount like `1.234,56` as 1.23456. Declaring the type per field avoids both.\n\n``` python\nimport re\nfrom datetime import datetime\nfrom decimal import Decimal, InvalidOperation\n\ndef parse_amount(text, decimal_comma=False):\n    negative = text.startswith(\"(\") and text.endswith(\")\")\n    cleaned = re.sub(r\"[^\\d,.\\-]\", \"\", text)\n    if decimal_comma:\n        cleaned = cleaned.replace(\".\", \"\").replace(\",\", \".\")\n    else:\n        cleaned = cleaned.replace(\",\", \"\")\n    try:\n        amount = Decimal(cleaned)\n    except InvalidOperation:\n        return text\n    return -amount if negative else amount\n\ndef normalize(value, kind=\"text\", date_formats=(\"%Y-%m-%d\",), decimal_comma=False):\n    if value is None:\n        return None\n    text = \" \".join(str(value).split())\n    if text == \"\":\n        return None\n    if kind == \"text\":\n        return text.casefold()\n    if kind == \"id\":\n        return text.replace(\" \", \"\").upper()\n    if kind == \"amount\":\n        return parse_amount(text, decimal_comma)\n    if kind == \"date\":\n        for fmt in date_formats:\n            try:\n                return datetime.strptime(text, fmt).date().isoformat()\n            except ValueError:\n                continue\n        return text\n    raise ValueError(f\"unknown field kind: {kind}\")\n```\n\nA few choices in this code are deliberate. Amounts use `Decimal`, so `$18,450.00`, `USD 18450`, and `18450` compare as equal without floating point surprises, and accounting negatives in parentheses keep their sign. IDs keep leading zeros and ignore spacing.\n\nThe `id` branch uppercases values, so drop `.upper()` if case carries meaning in your identifiers. Set `decimal_comma` per document based on its locale. For dates, list only the formats your documents use, since `03/04/2026` parses as March 4 under `%m/%d/%Y` and April 3 under `%d/%m/%Y`, and that choice belongs in the answer key rather than in a parser's guess.\n\nEach field comparison should land in one of four outcomes, since \"wrong\" covers failures with very different consequences. The taxonomy separates a missed value from an invented one:\n\n``` python\nfrom collections import Counter\n\ndef score_field(expected, predicted, kind=\"text\", **opts):\n    e = normalize(expected, kind, **opts)\n    p = normalize(predicted, kind, **opts)\n    if e is None and p is None:\n        return \"correct\"\n    if e is None:\n        return \"invented\"   # value returned for a field the page leaves empty\n    if p is None:\n        return \"missing\"\n    return \"correct\" if e == p else \"wrong\"\n\ndef score_document(answer_key, output, field_kinds):\n    return {field: score_field(answer_key.get(field), output.get(field), kind)\n            for field, kind in field_kinds.items()}\n\ndef summarize(scored_docs):\n    per_field = {}\n    for doc in scored_docs:\n        for field, outcome in doc.items():\n            per_field.setdefault(field, Counter())[outcome] += 1\n    return per_field\n```\n\nWatch the `invented` count closely. A missing value usually fails loudly downstream, while an invented value that looks plausible passes validation and lands in your system of record.\n\nReport every per field accuracy with a confidence interval, because the sample size behind each field varies. The Wilson score interval handles small samples well:\n\n``` python\nfrom math import sqrt\n\ndef wilson_interval(successes, n, z=1.96):\n    if n == 0:\n        return (0.0, 1.0)\n    p = successes / n\n    denom = 1 + z**2 / n\n    center = (p + z**2 / (2 * n)) / denom\n    margin = z * sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / denom\n    return (center - margin, center + margin)\n```\n\nWith 38 correct out of 40, the 95 percent interval runs from roughly 83.5 to 98.6 percent. With 380 out of 400, it narrows to roughly 92.4 to 96.7 percent. The interval tells you how precisely you have measured one tool on one field, and it answers a different question from whether one tool beats another.\n\nEvery tool runs on the same documents, so the right comparison looks at the cases where the tools disagree. McNemar's exact test counts the fields one tool got right and the other got wrong, in each direction, and checks whether that split is larger than chance would produce:\n\n``` python\nfrom math import comb\n\ndef mcnemar_exact(a_correct, b_correct):\n    \"\"\"Paired outcomes per (document, field): True when that tool was correct.\"\"\"\n    only_a = sum(1 for a, b in zip(a_correct, b_correct) if a and not b)\n    only_b = sum(1 for a, b in zip(a_correct, b_correct) if b and not a)\n    n = only_a + only_b\n    if n == 0:\n        return only_a, only_b, 1.0\n    k = min(only_a, only_b)\n    p = 2 * sum(comb(n, i) for i in range(k + 1)) / 2**n\n    return only_a, only_b, min(1.0, p)\n```\n\nIf tool A is right on 10 fields where tool B is wrong, and B is right on 2 where A is wrong, the two sided p value is about 0.039. Comparing two overlapping confidence intervals by eye is a weaker test, and it can miss a real difference between tools evaluated on the same documents.\n\nTables need their own score, because a table can contain every correct value and still be wrong if the values sit in the wrong cells. Merged cells come first: expand every row span and column span into a full grid, repeating the merged value in each position it covers, for both the answer key and the tool's output.\n\n``` python\ndef expand_spans(cells):\n    \"\"\"cells: dicts with row, col, text, and optional rowspan and colspan\"\"\"\n    if not cells:\n        return []\n    n_rows = max(c[\"row\"] + c.get(\"rowspan\", 1) for c in cells)\n    n_cols = max(c[\"col\"] + c.get(\"colspan\", 1) for c in cells)\n    grid = [[None] * n_cols for _ in range(n_rows)]\n    for c in cells:\n        for r in range(c[\"row\"], c[\"row\"] + c.get(\"rowspan\", 1)):\n            for k in range(c[\"col\"], c[\"col\"] + c.get(\"colspan\", 1)):\n                grid[r][k] = c[\"text\"]\n    return grid\n```\n\nThen compare the grids cell by cell and score both directions. Recall catches dropped cells, and precision catches cells the tool added:\n\n``` python\ndef table_cell_scores(expected_rows, predicted_rows, column_kinds=()):\n    def cells(rows):\n        out = {}\n        for i, row in enumerate(rows):\n            for j, value in enumerate(row):\n                kind = column_kinds[j] if j < len(column_kinds) else \"text\"\n                out[(i, j)] = normalize(value, kind)\n        return out\n\n    expected, predicted = cells(expected_rows), cells(predicted_rows)\n    matched = sum(1 for pos, v in expected.items()\n                  if pos in predicted and predicted[pos] == v)\n    precision = matched / len(predicted) if predicted else float(not expected)\n    recall = matched / len(expected) if expected else 1.0\n    return precision, recall\n```\n\nBoth scores are strict on purpose. If a tool drops one row of a multi page table, every row after it shifts and recall falls sharply. If a tool appends rows that are absent from the page, precision falls. For a structural metric with a published definition, TEDS, the tree edit distance based similarity introduced in [Image based table recognition: data, model, and evaluation](https://arxiv.org/abs/1911.10683), was designed to capture cell misalignment and OCR errors in table recognition.\n\nConsistency is easy to skip and expensive to discover in production. Run each document three to five times through each tool and measure how many fields return the same normalized value every time:\n\n``` python\ndef consistency(runs, field_kinds):\n    stable = 0\n    for field, kind in field_kinds.items():\n        values = [normalize(run.get(field), kind) for run in runs]\n        if all(v == values[0] for v in values):\n            stable += 1\n    return stable / len(field_kinds) if field_kinds else 1.0\n```\n\nComparing normalized values matters here too, since a tool that returns `1,200.00` on one run and `1200` on the next has produced the same value twice. A tool that is right on four runs out of five will look strong in a single pass benchmark and still produce a steady trickle of errors at production volume.\n\nIf a tool returns a confidence signal, test whether low scores actually line up with errors before you build review routing on top of it. Pair each field's confidence with its outcome from Step 4 and compute the error rate per bucket:\n\n``` python\ndef error_rate_by_confidence(rows, edges=(0.0, 0.5, 0.8, 0.95, 1.01)):\n    \"\"\"rows: (confidence, outcome) pairs\"\"\"\n    report = {}\n    for low, high in zip(edges, edges[1:]):\n        outcomes = [o for c, o in rows if low <= c < high]\n        if outcomes:\n            errors = sum(o != \"correct\" for o in outcomes)\n            report[f\"[{low}, {high})\"] = (len(outcomes), errors / len(outcomes))\n    return report\n```\n\nA useful signal shows error rates that fall as confidence rises. Treat the scores as a ranking for triage rather than a probability of correctness, which is also how [LandingAI describes the confidence scores in its first generation parse output](https://landing.ai/blog/introducing-confidence-scores-surface-parsing-uncertainty-before-it-becomes-a-problem).\n\nSignals arrive at different levels across tools, so align them to fields before bucketing. In ADE's second generation, the DPT-3 Pro parsing model returns no confidence scores, and the DPT-3 Verity model, currently in Preview, returns a confidence value per word. To get a field level score from per word values, find the words that a field's source ranges cover and take the lowest confidence among them.\n\nVerifying that extracted data will hold up in an audit means checking that every value can be traced to the exact place it came from. For each extracted field, record whether the tool returns a source location, such as a page and a bounding box or an offset into the parsed text, and whether that location actually contains the value. Spot check a sample by hand, since a location that points at the wrong region misleads the reviewer.\n\nScore it as pass or fail per field and keep it next to accuracy rather than inside it. A value with a valid location can still be the wrong value, such as a number taken from the row above, so traceability makes errors checkable and leaves accuracy to the other steps.\n\nThe accuracy rate a regulated workflow needs comes from the cost of an error on each field, so it differs from field to field. A wrong account number in a KYC file and a wrong page count in a cover letter carry very different consequences, and one blended target hides that difference.\n\nSet the target per field with your compliance team, then hold the benchmark to it with the interval from Step 4:\n\nThe report should keep every dimension separate, so the tradeoffs stay visible to whoever makes the decision:\n\n| Metric | How it is computed | What it tells you | \n|---|---|---|\n| Field accuracy | Correct outcomes divided by documents containing the field, with a Wilson interval | Which fields each tool handles reliably | \n| Paired difference | McNemar's exact test on fields where the tools disagree | Whether one tool is reliably better on a field | \n| Invented rate | Invented outcomes divided by fields the key marks empty | How often a tool fills gaps with plausible values | \n| Table precision and recall | Cells matched by position after expanding merged cells | Whether table structure survives extraction | \n| Consistency | Fields returning the same normalized value across runs | How much variance to expect at volume | \n| Traceability | Fields with a source location that contains the value | How quickly reviewers and auditors can verify output | \n\nBreak each metric down by document type and source quality. When you report a single overall number, weight each stratum by its share of production volume, since the test set deliberately oversamples hard cases and a plain average misstates what production will see.\n\nThe harness also gives a CTO a concrete list of questions for vendors evaluated for regulated deployments, because each step depends on something the vendor's output has to support:\n\nADE maps onto several of the steps above. Build Schema covers Step 2, generating the extraction schema from sample documents and keeping a version history, so every tool runs against the same frozen definition. On the parse side, ADE returns tables as HTML by default so merged cells survive, and each table cell block carries its grid position and span counts, which map onto the inputs `expand_spans` expects in Step 6.\n\nFor Step 9, ADE's Extract metadata pairs every returned value with character offsets into the parsed Markdown. When Extract derives a value instead of lifting it verbatim, for example a computed total or a document level summary, those offsets come back as `null`.\n\nTreat a populated field with `null` offsets as a value with no verbatim source and send it to review, then check how your own reformatted fields, such as dates converted to a standard format, behave before you count them against the tool.\n\nFor page level checks, the Ground API joins the extraction metadata with the parse output and returns the blocks, pages, and bounding boxes behind each field. Extraction can also run with the grounding stage skipped for speed, so keep it enabled for benchmark runs, since skipping it returns `null` ranges for every value.\n\nFor a reference on how to publish a benchmark that others can reproduce, [LandingAI's DocVQA benchmark write up](https://landing.ai/blog/docvqa-benchmark) is a useful model.\n\nDocVQA is a question answering benchmark rather than a field extraction one, and the post reports 99.16 percent accuracy on its validation split, with answers generated from parsed output and no access to the page images. The part worth copying is the method: the evaluation code is public on GitHub, and the post includes a gallery of all 45 errors with a classification for each, so readers can judge the failures themselves.\n\nIf you need a version of this process for a procurement decision rather than an engineering harness, [the vendor bake off guide for testing extraction accuracy](https://landing.ai/guides/how-to-test-document-extraction-accuracy-before-choosing-a-vendor) covers the same ground as a scorecard, with no code required.\n\n**What are the most accurate document extraction APIs available today?**\n\nThe answer depends on your documents, since published accuracy figures come from datasets each vendor selected. Run the candidates through a benchmark built from your own production files, score fields, tables, consistency, and traceability separately, and compare tools with a paired test.\n\n**How do you compare document intelligence accuracy across leading vendors?**\n\nRun every vendor on the same test set with the same frozen schema and answer key, normalize values by field type, and compare tools field by field with McNemar's exact test on the cases where they disagree.\n\n**What accuracy rate do you need for regulated document extraction?**\n\nSet the target per field based on the cost of an error on that field, then require the lower bound of the field's confidence interval to meet it. Add a separate ceiling for invented values on fields that drive decisions.\n\n**How do you verify extracted data is audit compliant?**\n\nCheck that every extracted value comes with a source location, like page coordinates or a text offset, and that the location actually contains the value. Score this per field alongside accuracy, since a value can be traceable and still wrong.\n\n**What questions should a CTO ask when evaluating document AI vendors for regulated industries?**\n\nAsk how the API represents missing values, whether every value carries a source location, how merged and multi page tables come back, what confidence signal exists and at what level, how model versions are pinned, and which data retention and deployment options apply.\n\n**Why score table precision as well as recall?**\n\nRecall alone ignores cells the tool added. A tool that returns every real row plus three rows absent from the page would score perfectly on recall, and precision is what exposes it.", "url": "https://wpnews.pro/news/how-to-benchmark-document-extraction-accuracy-on-your-own-pdfs", "canonical_source": "https://dev.to/landingai/how-to-benchmark-document-extraction-accuracy-on-your-own-pdfs-2mid", "published_at": "2026-11-16 11:45:55+00:00", "updated_at": "2026-11-16 11:52:29.356337+00:00", "lang": "en", "topics": ["ai-tools", "ai-products", "artificial-intelligence"], "entities": ["LandingAI", "Agentic Document Extraction"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-benchmark-document-extraction-accuracy-on-your-own-pdfs", "markdown": "https://wpnews.pro/news/how-to-benchmark-document-extraction-accuracy-on-your-own-pdfs.md", "text": "https://wpnews.pro/news/how-to-benchmark-document-extraction-accuracy-on-your-own-pdfs.txt", "jsonld": "https://wpnews.pro/news/how-to-benchmark-document-extraction-accuracy-on-your-own-pdfs.jsonld"}}