My AI said the PDF was empty. The PDF was not empty.

A developer using Microsoft's markitdown converter discovered that a PDF containing only raster images produced a zero-byte output with exit code 0, leading an AI model to incorrectly report the document as empty. The developer emphasizes that exit codes indicate process completion, not yield, and that integrations should measure extraction quality rather than rely solely on exit codes.

I asked Claude Code to pull the key dates out of a PDF I had saved from a webpage. It came back immediately: The document appears to be empty — it contains no extractable text. Which was strange, because I had the PDF open on my other monitor and it was four and a half megabytes of perfectly legible text. The interesting part is not that it was wrong. The interesting part is that nothing had failed. Every component in that chain did exactly what it was designed to do, reported success, and stacking those successes together produced a lie I believed. I was preprocessing documents with markitdown https://github.com/microsoft/markitdown , Microsoft's file-to-markdown converter, so I ran it by hand: bash $ markitdown screenshot.pdf -o out.md $ echo $? 0 $ wc -c out.md 0 out.md Exit code 0. Zero-byte output file. No warning, no stderr, nothing on the console at all. My integration did what integrations do: checked the return code, saw success, cached the result, and handed the model a path to a file with nothing in it. The model read the file, found nothing in it, and told me the document was empty. From its position that was a reasonable conclusion. It had been given an empty file and told the conversion worked. My first instinct was to file an issue. I am glad I did not, because markitdown is behaving correctly and I would have been publicly wrong. The PDF was a full-page browser screenshot exported to PDF. It contains raster images and no text layer whatsoever . pdfminer reports 0 characters, and so does PyMuPDF when you ask it. markitdown's PDF backend extracts embedded text and does not OCR — that is a documented design decision, not an oversight. So there was genuinely nothing to find. And finding nothing is not an error. A converter that exited non-zero every time a document happened to be empty would be wrong in a much more annoying way. The bug is somewhere else entirely, and it is worth naming precisely: The bug is in every integration that treats exit code as evidence of yield. Exit code answers "did the process complete?" I was reading it as an answer to "did we get the text?" Those are different questions, and for a document converter meeting a scanned page they have different answers. Mine was one of those integrations. Probably yours is too — pdftotext , pandoc and most extraction tooling have the same shape, because they should. That reframing is the whole story. Everything below is consequences. Never measure success by exit code. Measure it by yield. Simple to say. The trouble starts immediately, because "measure the yield" needs a threshold, and thresholds are where honest engineering goes to become arbitrary. Anyone can write if len text == 0: fail . That catches the screenshot. It does not catch the case that actually cost me time. A course completion certificate. One page, a decorative graphic, and a title line rendered as real text. It converts to this: Certificate of Completion Thirty-nine characters. Not zero. It sails through an emptiness check, gets cached as a successful conversion, and the model dutifully reports that your certificate says "Certificate of Completion" and nothing else — which is, again, technically what it was given. That is the shape of the real problem. The fully-empty case is easy and any check catches it. The expensive failures are the near-misses: a certificate, a slide deck exported as page images with a footer on every slide, a contract scanned at an angle with a header that happened to OCR at some point. They all return some characters. So the question becomes: how do you tell "extraction failed" from "this document is legitimately short"? The obvious move is a minimum size — reject anything under, say, 500 bytes. It does not work, and the reason it does not work is worth being precise about: raw byte count conflates document length with extraction quality. 800 bytes is a complete and correct conversion of a one-page memo. 800 bytes from a 200-page report is a catastrophic extraction failure. The same number means opposite things and the measure cannot distinguish them. What you want is density , not volume. Characters per page normalises document length away and leaves only the question you actually care about: on each page, did we recover a page's worth of text? I ran four real documents — a deliberately mixed set, including the near-miss that started this: | Document | Pages | Chars | Chars/page | |---|---|---|---| | Webpage screenshot saved as PDF | 1 | 0 | 0 | | Course certificate graphic + title | 1 | 39 | 39 | | Two-page text document | 2 | 1,864 | 932 | | Twenty-page slide deck | 20 | 13,289 | 664 | Two populations. No overlap. An order of magnitude between them. That gap is the finding. It is not a subtle statistical separation requiring a tuned classifier — a text extractor meeting a page of text produces hundreds of characters per page, and a text extractor meeting a picture produces tens or zero. There is nothing in between, because there is no such thing as a document that is 40% made of text. A threshold of 100 characters per page sits in that gap with roughly a 6× margin on both sides. The certificate is 2.5× below it; the sparsest real document is 6.6× above it. Small variations in document style — bigger fonts, more whitespace, a title slide — cannot cross a gap that wide. That margin is the entire justification for the number. I would not defend 100 as optimal. I would defend it as comfortably inside a gap where nothing lives , which is a much better property for a threshold than being finely tuned. Every threshold gets some cases wrong. What you get to choose is which way. A false positive — a genuinely sparse PDF gets flagged as image-based — means the model reads the original document with vision instead. That costs more tokens. It loses nothing. The user gets a correct answer at a higher price. A false negative — an empty conversion is presented as real — means the model confidently reports an empty document. The content is lost entirely and the user has no signal anything went wrong. That is the failure I started with. Those costs are not remotely symmetric, so the threshold is deliberately set to prefer the first. When it is wrong, it is wrong in the direction that costs money instead of the direction that costs truth. One more asymmetry, discovered by getting it wrong: the density test has to be PDF-only. Applying it to every format looks consistent and is a mistake. A Once you start looking for "success that isn't", it turns out to be a genre. Caching a bad result is worse than producing one. My first version cached by modification time. A zero-byte conversion was therefore served for every future reference to that document — permanently, with no retry, even after I had fixed the underlying cause. A transient failure had been promoted to a permanent one by the cache. Conversions are now re-graded before reuse, unusable results are deleted rather than stored, and zero-byte artifacts from older versions get swept on the next run so an upgrade heals the cache without anyone intervening. Silence is a failure mode. The hook must never block a prompt, so unexpected errors exit 0 quietly. That is correct for almost everything and catastrophic for one case: if markitdown is not installed at all, a silent no-op is indistinguishable from "this document is empty" — the exact failure the whole project exists to prevent. Missing dependencies are now the one error reported loudly, with the install command. The bug that produced no output at all. On Windows, PowerShell 5.1 prepends a UTF-8 BOM when piping to a native command. json.load raises on the BOM. That exception hit the never-block-a-prompt handler and was swallowed, so the hook did nothing, silently, on every prompt, on an entire platform. Two invisible failure modes composing into a third. The input parsing is BOM-tolerant now, but the lesson is the one above: an error handler that guarantees silence will eventually guarantee it for something you needed to hear about. A licence is a dependency decision. PyMuPDF reads some PDFs pdfminer cannot, so it is used when present — but it is never required. It is dual-licensed AGPL-3.0/commercial, which does not belong in the dependency set of an MIT project. Page counting, which the grading needs, uses pdfminer instead, which markitdown already depends on. The core path adds no dependency and no copyleft. When a conversion recovers real content, the model gets a pointer rather than the text: markitdown /path/report.pdf was converted to markdown at ~/.claude/markitdown-cache/report-a1b2c3d4.md 14973 bytes, 304 lines, 14472 chars, 20 page s . If this document's content is needed, Read or Grep the .md file not the original . The pointer matters more than it first looks. Prompts mention documents speculatively — "compare these three reports" may only genuinely need one of them. Loading all three into context costs those tokens on every subsequent turn of the conversation, used or not. A pointer costs about 400 characters and is paid once. The model reads or greps the file, with offset and limit for big ones, only if the content turns out to matter. And when extraction recovers nothing, no file is written, nothing is cached, and the model is told the truth: markitdown NO USABLE TEXT extracted from /path/screenshot.pdf no text extracted . This is an image-based/scanned PDF -- text extraction cannot see into it and no .md was written. Read the ORIGINAL file natively with the Read tool use the pages parameter for long PDFs ; Claude's vision can read it. Do NOT report the document as empty. That last line is there because without it the model does exactly that. None of this is really about PDFs. Any pipeline that hands one tool's output to another has this hazard the moment the first tool can succeed at doing nothing. Exit codes are a claim about process completion. They were never a claim about yield, and we have all been reading them as one because for most tools, most of the time, the two happen to coincide. The habit worth taking from it is small and cheap: after any extraction step, measure what came back and decide whether it is plausible for the input. Not And when it is empty, say so. A downstream model handed an empty file will not wonder whether something went wrong. It will tell your user their document is blank, in the same assured voice it uses when it is right. The hook I built out of this is MIT and runs on Windows, macOS and Linux: claude-markitdown-hook. The measurements behind the threshold, including the fixtures used to calibrate it, are in docs/DESIGN.md.