{"slug": "my-ai-said-the-pdf-was-empty-the-pdf-was-not-empty", "title": "My AI said the PDF was empty. The PDF was not empty.", "summary": "A developer using Microsoft's markitdown converter discovered that a PDF containing only raster images produced a zero-byte output with exit code 0, leading an AI model to incorrectly report the document as empty. The developer emphasizes that exit codes indicate process completion, not yield, and that integrations should measure extraction quality rather than rely solely on exit codes.", "body_md": "I asked Claude Code to pull the key dates out of a PDF I had saved from a\n\nwebpage. It came back immediately:\n\nThe document appears to be empty — it contains no extractable text.\n\nWhich was strange, because I had the PDF open on my other monitor and it was\n\nfour and a half megabytes of perfectly legible text.\n\nThe interesting part is not that it was wrong. The interesting part is that\n\nnothing had failed. Every component in that chain did exactly what it was\n\ndesigned to do, reported success, and stacking those successes together\n\nproduced a lie I believed.\n\nI was preprocessing documents with [markitdown](https://github.com/microsoft/markitdown),\n\nMicrosoft's file-to-markdown converter, so I ran it by hand:\n\n``` bash\n$ markitdown screenshot.pdf -o out.md\n$ echo $?\n0\n$ wc -c out.md\n0 out.md\n```\n\nExit code 0. Zero-byte output file. No warning, no stderr, nothing on the\n\nconsole at all.\n\nMy integration did what integrations do: checked the return code, saw success,\n\ncached the result, and handed the model a path to a file with nothing in it. The\n\nmodel read the file, found nothing in it, and told me the document was empty.\n\nFrom its position that was a reasonable conclusion. It had been given an empty\n\nfile and told the conversion worked.\n\nMy first instinct was to file an issue. I am glad I did not, because markitdown\n\nis behaving correctly and I would have been publicly wrong.\n\nThe PDF was a full-page browser screenshot exported to PDF. It contains raster\n\nimages and **no text layer whatsoever**. `pdfminer`\n\nreports 0 characters, and so\n\ndoes PyMuPDF when you ask it. markitdown's PDF backend extracts embedded text\n\nand does not OCR — that is a documented design decision, not an oversight.\n\nSo there was genuinely nothing to find. And *finding nothing is not an error.*\n\nA converter that exited non-zero every time a document happened to be empty\n\nwould be wrong in a much more annoying way.\n\nThe bug is somewhere else entirely, and it is worth naming precisely:\n\nThe bug is in every integration that treats exit code as evidence of yield.\n\nExit code answers \"did the process complete?\" I was reading it as an answer to\n\n\"did we get the text?\" Those are different questions, and for a document\n\nconverter meeting a scanned page they have different answers. Mine was one of\n\nthose integrations. Probably yours is too — `pdftotext`\n\n, `pandoc`\n\nand most\n\nextraction tooling have the same shape, because they should.\n\nThat reframing is the whole story. Everything below is consequences.\n\n**Never measure success by exit code. Measure it by yield.**\n\nSimple to say. The trouble starts immediately, because \"measure the yield\" needs\n\na threshold, and thresholds are where honest engineering goes to become\n\narbitrary. Anyone can write `if len(text) == 0: fail`\n\n. That catches the\n\nscreenshot. It does not catch the case that actually cost me time.\n\nA course completion certificate. One page, a decorative graphic, and a title\n\nline rendered as real text. It converts to this:\n\n```\nCertificate of Completion\n```\n\nThirty-nine characters. Not zero. It sails through an emptiness check, gets\n\ncached as a successful conversion, and the model dutifully reports that your\n\ncertificate says \"Certificate of Completion\" and nothing else — which is, again,\n\ntechnically what it was given.\n\nThat is the shape of the real problem. The fully-empty case is easy and any\n\ncheck catches it. The expensive failures are the near-misses: a certificate,\n\na slide deck exported as page images with a footer on every slide, a contract\n\nscanned at an angle with a header that happened to OCR at some point. They all\n\nreturn *some* characters.\n\nSo the question becomes: how do you tell \"extraction failed\" from \"this document\n\nis legitimately short\"?\n\nThe obvious move is a minimum size — reject anything under, say, 500 bytes. It\n\ndoes not work, and the reason it does not work is worth being precise about:\n\n**raw byte count conflates document length with extraction quality.**\n\n800 bytes is a complete and correct conversion of a one-page memo. 800 bytes\n\nfrom a 200-page report is a catastrophic extraction failure. The same number\n\nmeans opposite things and the measure cannot distinguish them.\n\nWhat you want is **density**, not volume. Characters per page normalises\n\ndocument length away and leaves only the question you actually care about: on\n\neach page, did we recover a page's worth of text?\n\nI ran four real documents — a deliberately mixed set, including the near-miss\n\nthat started this:\n\n| Document | Pages | Chars | Chars/page |\n|---|---|---|---|\n| Webpage screenshot saved as PDF | 1 | 0 | 0 |\n| Course certificate (graphic + title) | 1 | 39 | 39 |\n| Two-page text document | 2 | 1,864 | 932 |\n| Twenty-page slide deck | 20 | 13,289 | 664 |\n\nTwo populations. No overlap. An order of magnitude between them.\n\nThat gap is the finding. It is not a subtle statistical separation requiring a\n\ntuned classifier — a text extractor meeting a page of text produces hundreds of\n\ncharacters per page, and a text extractor meeting a picture produces tens or\n\nzero. There is nothing in between, because there is no such thing as a document\n\nthat is 40% made of text.\n\nA threshold of **100 characters per page** sits in that gap with roughly a 6×\n\nmargin on both sides. The certificate is 2.5× below it; the sparsest real\n\ndocument is 6.6× above it. Small variations in document style — bigger fonts,\n\nmore whitespace, a title slide — cannot cross a gap that wide.\n\nThat margin is the entire justification for the number. I would not defend 100\n\nas optimal. I would defend it as *comfortably inside a gap where nothing lives*,\n\nwhich is a much better property for a threshold than being finely tuned.\n\nEvery threshold gets some cases wrong. What you get to choose is which way.\n\n**A false positive** — a genuinely sparse PDF gets flagged as image-based — means\n\nthe model reads the original document with vision instead. That costs more\n\ntokens. It loses nothing. The user gets a correct answer at a higher price.\n\n**A false negative** — an empty conversion is presented as real — means the model\n\nconfidently reports an empty document. The content is lost entirely and *the\nuser has no signal anything went wrong.* That is the failure I started with.\n\nThose costs are not remotely symmetric, so the threshold is deliberately set to\n\nprefer the first. When it is wrong, it is wrong in the direction that costs\n\nmoney instead of the direction that costs truth.\n\nOne more asymmetry, discovered by getting it wrong: **the density test has to be\nPDF-only.** Applying it to every format looks consistent and is a mistake. A\n\nOnce you start looking for \"success that isn't\", it turns out to be a genre.\n\n**Caching a bad result is worse than producing one.** My first version cached by\n\nmodification time. A zero-byte conversion was therefore served for every future\n\nreference to that document — permanently, with no retry, even after I had fixed\n\nthe underlying cause. A transient failure had been promoted to a permanent one\n\nby the cache. Conversions are now re-graded before reuse, unusable results are\n\ndeleted rather than stored, and zero-byte artifacts from older versions get\n\nswept on the next run so an upgrade heals the cache without anyone intervening.\n\n**Silence is a failure mode.** The hook must never block a prompt, so unexpected\n\nerrors exit 0 quietly. That is correct for almost everything and catastrophic\n\nfor one case: if markitdown is not installed at all, a silent no-op is\n\nindistinguishable from \"this document is empty\" — the exact failure the whole\n\nproject exists to prevent. Missing dependencies are now the one error reported\n\nloudly, with the install command.\n\n**The bug that produced no output at all.** On Windows, PowerShell 5.1 prepends\n\na UTF-8 BOM when piping to a native command. `json.load`\n\nraises on the BOM. That\n\nexception hit the never-block-a-prompt handler and was swallowed, so the hook\n\ndid nothing, silently, on every prompt, on an entire platform. Two invisible\n\nfailure modes composing into a third. The input parsing is BOM-tolerant now, but\n\nthe lesson is the one above: an error handler that guarantees silence will\n\neventually guarantee it for something you needed to hear about.\n\n**A licence is a dependency decision.** PyMuPDF reads some PDFs pdfminer cannot,\n\nso it is used when present — but it is never required. It is dual-licensed\n\nAGPL-3.0/commercial, which does not belong in the dependency set of an MIT\n\nproject. Page counting, which the grading needs, uses pdfminer instead, which\n\nmarkitdown already depends on. The core path adds no dependency and no copyleft.\n\nWhen a conversion recovers real content, the model gets a pointer rather than\n\nthe text:\n\n```\n[markitdown] /path/report.pdf was converted to markdown at\n~/.claude/markitdown-cache/report-a1b2c3d4.md (14973 bytes, 304 lines,\n14472 chars, 20 page(s)). If this document's content is needed, Read or\nGrep the .md file (not the original).\n```\n\nThe pointer matters more than it first looks. Prompts mention documents\n\nspeculatively — \"compare these three reports\" may only genuinely need one of\n\nthem. Loading all three into context costs those tokens on every subsequent turn\n\nof the conversation, used or not. A pointer costs about 400 characters and is\n\npaid once. The model reads or greps the file, with offset and limit for big\n\nones, only if the content turns out to matter.\n\nAnd when extraction recovers nothing, no file is written, nothing is cached, and\n\nthe model is told the truth:\n\n```\n[markitdown] NO USABLE TEXT extracted from /path/screenshot.pdf (no text\nextracted). This is an image-based/scanned PDF -- text extraction cannot\nsee into it and no .md was written. Read the ORIGINAL file natively with\nthe Read tool (use the `pages` parameter for long PDFs); Claude's vision\ncan read it. Do NOT report the document as empty.\n```\n\nThat last line is there because without it the model does exactly that.\n\nNone of this is really about PDFs.\n\nAny pipeline that hands one tool's output to another has this hazard the moment\n\nthe first tool can succeed at doing nothing. Exit codes are a claim about\n\nprocess completion. They were never a claim about yield, and we have all been\n\nreading them as one because for most tools, most of the time, the two happen to\n\ncoincide.\n\nThe habit worth taking from it is small and cheap: **after any extraction step,\nmeasure what came back and decide whether it is plausible for the input.** Not\n\nAnd when it is empty, say so. A downstream model handed an empty file will not\n\nwonder whether something went wrong. It will tell your user their document is\n\nblank, in the same assured voice it uses when it is right.\n\n*The hook I built out of this is MIT and runs on Windows, macOS and Linux:\nclaude-markitdown-hook.\nThe measurements behind the threshold, including the fixtures used to calibrate\nit, are in docs/DESIGN.md.*", "url": "https://wpnews.pro/news/my-ai-said-the-pdf-was-empty-the-pdf-was-not-empty", "canonical_source": "https://dev.to/andrewavery7/my-ai-said-the-pdf-was-empty-the-pdf-was-not-empty-1b1l", "published_at": "2026-08-19 14:51:30+00:00", "updated_at": "2026-08-19 15:13:16.907431+00:00", "lang": "en", "topics": ["developer-tools", "artificial-intelligence"], "entities": ["Microsoft", "markitdown", "Claude Code", "pdfminer", "PyMuPDF", "pdftotext", "pandoc"], "alternates": {"html": "https://wpnews.pro/news/my-ai-said-the-pdf-was-empty-the-pdf-was-not-empty", "markdown": "https://wpnews.pro/news/my-ai-said-the-pdf-was-empty-the-pdf-was-not-empty.md", "text": "https://wpnews.pro/news/my-ai-said-the-pdf-was-empty-the-pdf-was-not-empty.txt", "jsonld": "https://wpnews.pro/news/my-ai-said-the-pdf-was-empty-the-pdf-was-not-empty.jsonld"}}