How to Benchmark Document Extraction Accuracy on Your Own PDFs A developer outlined a vendor-agnostic Python benchmark harness for measuring document extraction accuracy on a team's own PDFs, arguing that vendor-published accuracy figures come from datasets the vendors chose and don't predict performance on real intake files. The approach scores four separate dimensions — field accuracy, table accuracy, consistency, and audit traceability — using a stratified test set, a per-document JSON answer key that records nulls for absent fields, per-field type declarations, and paired significance testing, and it can be pointed at any tool's output including LandingAI's Agentic Document Extraction. Teams evaluating document extraction usually start by asking which APIs are the most accurate available today. Every vendor answers with a published accuracy number, and every one of those numbers comes from a dataset the vendor picked. Your documents have your scan quality, your vendors' invoice layouts, your handwritten fields, and your edge cases, so the accuracy figure that predicts how a tool behaves in your pipeline is the one you measure on your own files. This guide walks through the benchmark harness we would build to compare document extraction accuracy across vendors. It covers the test set, the answer key, typed normalization, field and table scoring, paired significance testing, consistency, uncertainty signals, and audit traceability. The code is plain Python with no vendor dependency, so you can point it at the output of any tool, including LandingAI's Agentic Document Extraction ADE . A useful extraction benchmark measures four separate things, and folding them into one accuracy number hides the failures that matter most. Keep them apart from the start: The first three tell you whether the output is correct. The fourth tells you whether a reviewer or an auditor can verify it, which is a different property and deserves its own column in your results. The test set should come from the documents your pipeline will actually receive, sampled across the variation you see in production. Pull files directly from your intake, then sort them by the dimensions that change extraction difficulty: Record each document's stratum and how much of your production volume that stratum represents, since the report uses those shares later. Make sure the hard cases are well represented, because a benchmark built mostly on clean files will rank every tool close together and tell you little. The answer key is a JSON file per document that records the correct value for every field in your schema, and it has to record absence as carefully as presence. When a field genuinely has no value on the page, write null . That single choice is what lets your harness catch a tool that invents values. { "doc id": "inv 0042", "invoice number": "INV-20931", "invoice date": "2026-03-14", "total amount": "18450.00", "po number": null, "vendor name": "Acme Industrial Supply" } Next to the answer key, declare a type for every field. The type decides how the harness compares values in Step 3: FIELD KINDS = { "invoice number": "id", "invoice date": "date", "total amount": "amount", "po number": "id", "vendor name": "text", } The answer key has its own error rate. Have two people label the same subset independently, compare their keys field by field, and resolve every disagreement before scoring any tool. Fields where labelers disagree often are usually ambiguous in the schema, and tightening their definitions improves the whole benchmark. Freeze the schema before you run any tool, and use the same schema for every tool you compare. If you need a starting point, Build Schema drafts an extraction schema from sample documents or a written description of the fields https://landing.ai/ade/build-schema , and it keeps a version history, which helps when the schema changes partway through an evaluation. Normalization removes formatting differences that carry no meaning, so the harness compares values instead of strings. One function that guesses the type of every value causes more errors than it removes: it turns PO number 0042 into the number 42, and it reads a European amount like 1.234,56 as 1.23456. Declaring the type per field avoids both. python import re from datetime import datetime from decimal import Decimal, InvalidOperation def parse amount text, decimal comma=False : negative = text.startswith " " and text.endswith " " cleaned = re.sub r" ^\d,.\- ", "", text if decimal comma: cleaned = cleaned.replace ".", "" .replace ",", "." else: cleaned = cleaned.replace ",", "" try: amount = Decimal cleaned except InvalidOperation: return text return -amount if negative else amount def normalize value, kind="text", date formats= "%Y-%m-%d", , decimal comma=False : if value is None: return None text = " ".join str value .split if text == "": return None if kind == "text": return text.casefold if kind == "id": return text.replace " ", "" .upper if kind == "amount": return parse amount text, decimal comma if kind == "date": for fmt in date formats: try: return datetime.strptime text, fmt .date .isoformat except ValueError: continue return text raise ValueError f"unknown field kind: {kind}" A few choices in this code are deliberate. Amounts use Decimal , so $18,450.00 , USD 18450 , and 18450 compare as equal without floating point surprises, and accounting negatives in parentheses keep their sign. IDs keep leading zeros and ignore spacing. The id branch uppercases values, so drop .upper if case carries meaning in your identifiers. Set decimal comma per document based on its locale. For dates, list only the formats your documents use, since 03/04/2026 parses as March 4 under %m/%d/%Y and April 3 under %d/%m/%Y , and that choice belongs in the answer key rather than in a parser's guess. Each field comparison should land in one of four outcomes, since "wrong" covers failures with very different consequences. The taxonomy separates a missed value from an invented one: python from collections import Counter def score field expected, predicted, kind="text", opts : e = normalize expected, kind, opts p = normalize predicted, kind, opts if e is None and p is None: return "correct" if e is None: return "invented" value returned for a field the page leaves empty if p is None: return "missing" return "correct" if e == p else "wrong" def score document answer key, output, field kinds : return {field: score field answer key.get field , output.get field , kind for field, kind in field kinds.items } def summarize scored docs : per field = {} for doc in scored docs: for field, outcome in doc.items : per field.setdefault field, Counter outcome += 1 return per field Watch the invented count closely. A missing value usually fails loudly downstream, while an invented value that looks plausible passes validation and lands in your system of record. Report every per field accuracy with a confidence interval, because the sample size behind each field varies. The Wilson score interval handles small samples well: python from math import sqrt def wilson interval successes, n, z=1.96 : if n == 0: return 0.0, 1.0 p = successes / n denom = 1 + z 2 / n center = p + z 2 / 2 n / denom margin = z sqrt p 1 - p / n + z 2 / 4 n 2 / denom return center - margin, center + margin With 38 correct out of 40, the 95 percent interval runs from roughly 83.5 to 98.6 percent. With 380 out of 400, it narrows to roughly 92.4 to 96.7 percent. The interval tells you how precisely you have measured one tool on one field, and it answers a different question from whether one tool beats another. Every tool runs on the same documents, so the right comparison looks at the cases where the tools disagree. McNemar's exact test counts the fields one tool got right and the other got wrong, in each direction, and checks whether that split is larger than chance would produce: python from math import comb def mcnemar exact a correct, b correct : """Paired outcomes per document, field : True when that tool was correct.""" only a = sum 1 for a, b in zip a correct, b correct if a and not b only b = sum 1 for a, b in zip a correct, b correct if b and not a n = only a + only b if n == 0: return only a, only b, 1.0 k = min only a, only b p = 2 sum comb n, i for i in range k + 1 / 2 n return only a, only b, min 1.0, p If tool A is right on 10 fields where tool B is wrong, and B is right on 2 where A is wrong, the two sided p value is about 0.039. Comparing two overlapping confidence intervals by eye is a weaker test, and it can miss a real difference between tools evaluated on the same documents. Tables need their own score, because a table can contain every correct value and still be wrong if the values sit in the wrong cells. Merged cells come first: expand every row span and column span into a full grid, repeating the merged value in each position it covers, for both the answer key and the tool's output. python def expand spans cells : """cells: dicts with row, col, text, and optional rowspan and colspan""" if not cells: return n rows = max c "row" + c.get "rowspan", 1 for c in cells n cols = max c "col" + c.get "colspan", 1 for c in cells grid = None n cols for in range n rows for c in cells: for r in range c "row" , c "row" + c.get "rowspan", 1 : for k in range c "col" , c "col" + c.get "colspan", 1 : grid r k = c "text" return grid Then compare the grids cell by cell and score both directions. Recall catches dropped cells, and precision catches cells the tool added: python def table cell scores expected rows, predicted rows, column kinds= : def cells rows : out = {} for i, row in enumerate rows : for j, value in enumerate row : kind = column kinds j if j < len column kinds else "text" out i, j = normalize value, kind return out expected, predicted = cells expected rows , cells predicted rows matched = sum 1 for pos, v in expected.items if pos in predicted and predicted pos == v precision = matched / len predicted if predicted else float not expected recall = matched / len expected if expected else 1.0 return precision, recall Both scores are strict on purpose. If a tool drops one row of a multi page table, every row after it shifts and recall falls sharply. If a tool appends rows that are absent from the page, precision falls. For a structural metric with a published definition, TEDS, the tree edit distance based similarity introduced in Image based table recognition: data, model, and evaluation https://arxiv.org/abs/1911.10683 , was designed to capture cell misalignment and OCR errors in table recognition. Consistency is easy to skip and expensive to discover in production. Run each document three to five times through each tool and measure how many fields return the same normalized value every time: python def consistency runs, field kinds : stable = 0 for field, kind in field kinds.items : values = normalize run.get field , kind for run in runs if all v == values 0 for v in values : stable += 1 return stable / len field kinds if field kinds else 1.0 Comparing normalized values matters here too, since a tool that returns 1,200.00 on one run and 1200 on the next has produced the same value twice. A tool that is right on four runs out of five will look strong in a single pass benchmark and still produce a steady trickle of errors at production volume. If a tool returns a confidence signal, test whether low scores actually line up with errors before you build review routing on top of it. Pair each field's confidence with its outcome from Step 4 and compute the error rate per bucket: python def error rate by confidence rows, edges= 0.0, 0.5, 0.8, 0.95, 1.01 : """rows: confidence, outcome pairs""" report = {} for low, high in zip edges, edges 1: : outcomes = o for c, o in rows if low <= c < high if outcomes: errors = sum o = "correct" for o in outcomes report f" {low}, {high} " = len outcomes , errors / len outcomes return report A useful signal shows error rates that fall as confidence rises. Treat the scores as a ranking for triage rather than a probability of correctness, which is also how LandingAI describes the confidence scores in its first generation parse output https://landing.ai/blog/introducing-confidence-scores-surface-parsing-uncertainty-before-it-becomes-a-problem . Signals arrive at different levels across tools, so align them to fields before bucketing. In ADE's second generation, the DPT-3 Pro parsing model returns no confidence scores, and the DPT-3 Verity model, currently in Preview, returns a confidence value per word. To get a field level score from per word values, find the words that a field's source ranges cover and take the lowest confidence among them. Verifying that extracted data will hold up in an audit means checking that every value can be traced to the exact place it came from. For each extracted field, record whether the tool returns a source location, such as a page and a bounding box or an offset into the parsed text, and whether that location actually contains the value. Spot check a sample by hand, since a location that points at the wrong region misleads the reviewer. Score it as pass or fail per field and keep it next to accuracy rather than inside it. A value with a valid location can still be the wrong value, such as a number taken from the row above, so traceability makes errors checkable and leaves accuracy to the other steps. The accuracy rate a regulated workflow needs comes from the cost of an error on each field, so it differs from field to field. A wrong account number in a KYC file and a wrong page count in a cover letter carry very different consequences, and one blended target hides that difference. Set the target per field with your compliance team, then hold the benchmark to it with the interval from Step 4: The report should keep every dimension separate, so the tradeoffs stay visible to whoever makes the decision: | Metric | How it is computed | What it tells you | |---|---|---| | Field accuracy | Correct outcomes divided by documents containing the field, with a Wilson interval | Which fields each tool handles reliably | | Paired difference | McNemar's exact test on fields where the tools disagree | Whether one tool is reliably better on a field | | Invented rate | Invented outcomes divided by fields the key marks empty | How often a tool fills gaps with plausible values | | Table precision and recall | Cells matched by position after expanding merged cells | Whether table structure survives extraction | | Consistency | Fields returning the same normalized value across runs | How much variance to expect at volume | | Traceability | Fields with a source location that contains the value | How quickly reviewers and auditors can verify output | Break each metric down by document type and source quality. When you report a single overall number, weight each stratum by its share of production volume, since the test set deliberately oversamples hard cases and a plain average misstates what production will see. The harness also gives a CTO a concrete list of questions for vendors evaluated for regulated deployments, because each step depends on something the vendor's output has to support: ADE maps onto several of the steps above. Build Schema covers Step 2, generating the extraction schema from sample documents and keeping a version history, so every tool runs against the same frozen definition. On the parse side, ADE returns tables as HTML by default so merged cells survive, and each table cell block carries its grid position and span counts, which map onto the inputs expand spans expects in Step 6. For Step 9, ADE's Extract metadata pairs every returned value with character offsets into the parsed Markdown. When Extract derives a value instead of lifting it verbatim, for example a computed total or a document level summary, those offsets come back as null . Treat a populated field with null offsets as a value with no verbatim source and send it to review, then check how your own reformatted fields, such as dates converted to a standard format, behave before you count them against the tool. For page level checks, the Ground API joins the extraction metadata with the parse output and returns the blocks, pages, and bounding boxes behind each field. Extraction can also run with the grounding stage skipped for speed, so keep it enabled for benchmark runs, since skipping it returns null ranges for every value. For a reference on how to publish a benchmark that others can reproduce, LandingAI's DocVQA benchmark write up https://landing.ai/blog/docvqa-benchmark is a useful model. DocVQA is a question answering benchmark rather than a field extraction one, and the post reports 99.16 percent accuracy on its validation split, with answers generated from parsed output and no access to the page images. The part worth copying is the method: the evaluation code is public on GitHub, and the post includes a gallery of all 45 errors with a classification for each, so readers can judge the failures themselves. If you need a version of this process for a procurement decision rather than an engineering harness, the vendor bake off guide for testing extraction accuracy https://landing.ai/guides/how-to-test-document-extraction-accuracy-before-choosing-a-vendor covers the same ground as a scorecard, with no code required. What are the most accurate document extraction APIs available today? The answer depends on your documents, since published accuracy figures come from datasets each vendor selected. Run the candidates through a benchmark built from your own production files, score fields, tables, consistency, and traceability separately, and compare tools with a paired test. How do you compare document intelligence accuracy across leading vendors? Run every vendor on the same test set with the same frozen schema and answer key, normalize values by field type, and compare tools field by field with McNemar's exact test on the cases where they disagree. What accuracy rate do you need for regulated document extraction? Set the target per field based on the cost of an error on that field, then require the lower bound of the field's confidence interval to meet it. Add a separate ceiling for invented values on fields that drive decisions. How do you verify extracted data is audit compliant? Check that every extracted value comes with a source location, like page coordinates or a text offset, and that the location actually contains the value. Score this per field alongside accuracy, since a value can be traceable and still wrong. What questions should a CTO ask when evaluating document AI vendors for regulated industries? Ask how the API represents missing values, whether every value carries a source location, how merged and multi page tables come back, what confidence signal exists and at what level, how model versions are pinned, and which data retention and deployment options apply. Why score table precision as well as recall? Recall alone ignores cells the tool added. A tool that returns every real row plus three rows absent from the page would score perfectly on recall, and precision is what exposes it.