Your PDFs Are Eating Your LLM's Tokens for Breakfast

Maneshwar, the developer behind git-lrc, explains how PDFs consume excessive tokens in LLM interactions due to formatting noise. He highlights Microsoft's open-source MarkItDown tool, which converts PDFs and other files to clean Markdown, reducing token usage by up to 70% and improving model comprehension. MarkItDown also offers an MCP server for seamless integration with AI clients.

Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback. Real talk for a second. You drag a chunky PDF into your favorite AI chat, ask one innocent little question, and somewhere in the background your token budget just quietly combusted. I want to walk you through why that happens, and the free Microsoft tool that has been sitting on GitHub the whole time waiting for you to notice it. Grab a coffee. This one is going to save you money. Here is the thing about PDFs. They are Pretty Darn Fat. When you upload one to a model, it does not just read the words like you and I do. It has to wade through the fonts, the layout coordinates, the half-broken tables, the embedded images, and every other bit of packaging that PDFs love to hoard. All of that formatting noise gets turned into tokens, and tokens are the currency you are paying with. Rough numbers: a single PDF page can run somewhere between 1,500 and 3,000 tokens depending on how messy it is. Do the math on a 20-page document and you can be looking at up to 70,000 tokens poof That is before your first question. You paid the cover charge and you have not even ordered a drink yet. Here is that same idea as a picture, because I like pictures. The left path is where most of us live. The right path is where I want to take you. The fix is a free, open-source tool from Microsoft called MarkItDown . It has blown past 110,000 stars on GitHub sitting around 119k as I write this , so this is not some abandoned weekend hack. It is a "boring tool that just works," which in developer language is the highest possible compliment. What it does is beautifully simple. You give it a file, and it hands you back clean Markdown. That is it. That is the whole trick. And by "a file" I mean almost anything you can throw at it: PDFs, Word docs, Excel sheets, PowerPoint decks, images, audio, HTML, and yes, even YouTube links, which it happily turns into a transcript. It is a bit of a "convert anything to Markdown" Swiss Army knife. Why does Markdown help so much? Two reasons, and both matter. First, Markdown is featherweight. It keeps the structure that carries meaning headings, lists, tables, links and throws out the formatting fluff that does not. Less noise, fewer tokens, up to around 70 percent lighter in the good cases. Your mileage will vary, but the direction is always down. Second, and this is the part people sleep on, models actually read Markdown better. Mainstream LLMs were trained on mountains of Markdown and speak it natively. They will even reach for it in their own answers when you did not ask them to. So you are not just spending fewer tokens, you are handing the model its favorite format on a silver platter. It is a genuine handshake moment. Mark my words, once you see the format click, you will not want to go back. Here is where it stops being a manual chore. MarkItDown ships an MCP server the little markitdown-mcp package . Wire it into any MCP-capable client desktop assistants, editors like Cursor and VS Code, and friends , and instead of you converting files by hand, you just point the model at a file path or a URL and it reaches for the converter itself. Under the hood it exposes a single tool, convert to markdown uri , and the model knows when to call it. So your flow goes from "export, convert, re-upload, sigh" to "hey, read this report and summarize it." The conversion happens in the middle, quietly, without you babysitting it. Setup is about a two-minute job. You install the package with pip, drop a small block into your client's MCP config, and then fully quit and reopen the app. That last step trips people up constantly, so let me say it loudly: the MCP server only loads on a clean launch. Closing the window is not quitting. Cmd or Ctrl plus Q, then reopen. Otherwise you will sit there wondering why nothing happened. One honest note. This auto-magic only kicks in on clients that support MCP. If you are in a plain browser chat, you can still get all the savings, you just run the file through MarkItDown yourself first and paste the Markdown in. Slightly more manual, same payoff. You have been paying a formatting tax on every file you feed your model, and Microsoft handed out the coupon months ago. Convert to Markdown first, spend a fraction of the tokens, and get answers in the format the model likes best. Set up the MCP server once and it mostly disappears into the background. Two minutes of setup for up to 70 percent fewer tokens is the kind of trade I take every single time. If you want to poke at it yourself, here is where to start: Go convert something. Your token budget has suffered enough. If you tried this and it saved you a pile of tokens tell me in the comments. I read them. AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs — without telling you. You often find out in production. git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free. Any feedback or contributors are welcome It's online, source-available, and ready for anyone to use. ⭐ Star it on GitHub: | 🇩🇰 Dansk https://github.com/HexmosTech/git-lrc/readme/README.da.md | 🇪🇸 Español https://github.com/HexmosTech/git-lrc/readme/README.es.md | 🇮🇷 Farsi https://github.com/HexmosTech/git-lrc/readme/README.fa.md | 🇫🇮 Suomi https://github.com/HexmosTech/git-lrc/readme/README.fi.md | 🇯🇵 日本語 https://github.com/HexmosTech/git-lrc/readme/README.ja.md | 🇳🇴 Norsk https://github.com/HexmosTech/git-lrc/readme/README.nn.md | 🇵🇹 Português https://github.com/HexmosTech/git-lrc/readme/README.pt.md | 🇷🇺 Русский https://github.com/HexmosTech/git-lrc/readme/README.ru.md | 🇦🇱 Shqip https://github.com/HexmosTech/git-lrc/readme/README.sq.md | 🇨🇳 中文 https://github.com/HexmosTech/git-lrc/readme/README.zh.md | 🇮🇳 हिन्दी https://github.com/HexmosTech/git-lrc/readme/README.hi.md | GenAI today is a race car without brakes . It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things : they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production. git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen At a glance: 10 risk categories https://github.com/HexmosTech/git-lrc what-git-lrc-checks-for · 100+ failure patterns tracked https://github.com/HexmosTech/git-lrc what-git-lrc-checks-for · every commit…