{"slug": "your-pdfs-are-eating-your-llm-s-tokens-for-breakfast", "title": "Your PDFs Are Eating Your LLM's Tokens for Breakfast", "summary": "Maneshwar, the developer behind git-lrc, explains how PDFs consume excessive tokens in LLM interactions due to formatting noise. He highlights Microsoft's open-source MarkItDown tool, which converts PDFs and other files to clean Markdown, reducing token usage by up to 70% and improving model comprehension. MarkItDown also offers an MCP server for seamless integration with AI clients.", "body_md": "*Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback.*\n\nReal talk for a second.\n\nYou drag a chunky PDF into your favorite AI chat, ask one innocent little question, and somewhere in the background your token budget just quietly combusted.\n\nI want to walk you through why that happens, and the free Microsoft tool that has been sitting on GitHub the whole time waiting for you to notice it.\n\nGrab a coffee. This one is going to save you money.\n\nHere is the thing about PDFs. They are Pretty Darn Fat.\n\nWhen you upload one to a model, it does not just read the words like you and I do.\n\nIt has to wade through the fonts, the layout coordinates, the half-broken tables, the embedded images, and every other bit of packaging that PDFs love to hoard.\n\nAll of that formatting noise gets turned into tokens, and tokens are the currency you are paying with.\n\nRough numbers: a single PDF page can run somewhere between 1,500 and 3,000 tokens depending on how messy it is. Do the math on a 20-page document and you can be looking at up to 70,000 tokens poof! That is before your first question. You paid the cover charge and you have not even ordered a drink yet.\n\nHere is that same idea as a picture, because I like pictures.\n\nThe left path is where most of us live.\n\nThe right path is where I want to take you.\n\nThe fix is a free, open-source tool from Microsoft called **MarkItDown**.\n\nIt has blown past 110,000 stars on GitHub (sitting around 119k as I write this), so this is not some abandoned weekend hack.\n\nIt is a \"boring tool that just works,\" which in developer language is the highest possible compliment.\n\nWhat it does is beautifully simple. You give it a file, and it hands you back clean Markdown. That is it. That is the whole trick.\n\nAnd by \"a file\" I mean almost anything you can throw at it: PDFs, Word docs, Excel sheets, PowerPoint decks, images, audio, HTML, and yes, even YouTube links, which it happily turns into a transcript.\n\nIt is a bit of a \"convert anything to Markdown\" Swiss Army knife.\n\nWhy does Markdown help so much? Two reasons, and both matter.\n\nFirst, Markdown is featherweight. It keeps the structure that carries meaning (headings, lists, tables, links) and throws out the formatting fluff that does not. Less noise, fewer tokens, up to around 70 percent lighter in the good cases. Your mileage will vary, but the direction is always down.\n\nSecond, and this is the part people sleep on, models actually read Markdown better.\n\nMainstream LLMs were trained on mountains of Markdown and speak it natively.\n\nThey will even reach for it in their own answers when you did not ask them to.\n\nSo you are not just spending fewer tokens, you are handing the model its favorite format on a silver platter.\n\nIt is a genuine handshake moment.\n\nMark my words, once you see the format click, you will not want to go back.\n\nHere is where it stops being a manual chore.\n\nMarkItDown ships an MCP server (the little `markitdown-mcp`\n\npackage).\n\nWire it into any MCP-capable client (desktop assistants, editors like Cursor and VS Code, and friends), and instead of you converting files by hand, you just point the model at a file path or a URL and it reaches for the converter itself.\n\nUnder the hood it exposes a single tool, `convert_to_markdown(uri)`\n\n, and the model knows when to call it.\n\nSo your flow goes from \"export, convert, re-upload, sigh\" to \"hey, read this report and summarize it.\"\n\nThe conversion happens in the middle, quietly, without you babysitting it.\n\nSetup is about a two-minute job. You install the package with pip, drop a small block into your client's MCP config, and then fully quit and reopen the app.\n\nThat last step trips people up constantly, so let me say it loudly: the MCP server only loads on a clean launch.\n\nClosing the window is not quitting. Cmd or Ctrl plus Q, then reopen. Otherwise you will sit there wondering why nothing happened.\n\nOne honest note. This auto-magic only kicks in on clients that support MCP.\n\nIf you are in a plain browser chat, you can still get all the savings, you just run the file through MarkItDown yourself first and paste the Markdown in. Slightly more manual, same payoff.\n\nYou have been paying a formatting tax on every file you feed your model, and Microsoft handed out the coupon months ago.\n\nConvert to Markdown first, spend a fraction of the tokens, and get answers in the format the model likes best.\n\nSet up the MCP server once and it mostly disappears into the background.\n\nTwo minutes of setup for up to 70 percent fewer tokens is the kind of trade I take every single time.\n\nIf you want to poke at it yourself, here is where to start:\n\nGo convert something. Your token budget has suffered enough.\n\n*If you tried this and it saved you a pile of tokens tell me in the comments. I read them.*\n\nAI agents write code fast. They also silently remove logic, change behavior, and introduce bugs — without telling you. You often find out in production.\n\ngit-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.\n\nAny feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.\n\n⭐ Star it on GitHub:\n\n| [🇩🇰 Dansk](https://github.com/HexmosTech/git-lrc/readme/README.da.md) | [🇪🇸 Español](https://github.com/HexmosTech/git-lrc/readme/README.es.md) | [🇮🇷 Farsi](https://github.com/HexmosTech/git-lrc/readme/README.fa.md) | [🇫🇮 Suomi](https://github.com/HexmosTech/git-lrc/readme/README.fi.md) | [🇯🇵 日本語](https://github.com/HexmosTech/git-lrc/readme/README.ja.md) | [🇳🇴 Norsk](https://github.com/HexmosTech/git-lrc/readme/README.nn.md) | [🇵🇹 Português](https://github.com/HexmosTech/git-lrc/readme/README.pt.md) | [🇷🇺 Русский](https://github.com/HexmosTech/git-lrc/readme/README.ru.md) | [🇦🇱 Shqip](https://github.com/HexmosTech/git-lrc/readme/README.sq.md) | [🇨🇳 中文](https://github.com/HexmosTech/git-lrc/readme/README.zh.md) | [🇮🇳 हिन्दी](https://github.com/HexmosTech/git-lrc/readme/README.hi.md) |\n\nGenAI today is a **race car without brakes**. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents *silently break things*: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.\n\n** git-lrc is your braking system.** It hooks into\n\n`git commit`\n\nand runs an AI review on every diff In short, git-lrc helps **Prevent Outages, Breaches, and Technical Debt Before They Happen**\n\n**At a glance:** [10 risk categories](https://github.com/HexmosTech/git-lrc#what-git-lrc-checks-for) · [100+ failure patterns tracked](https://github.com/HexmosTech/git-lrc#what-git-lrc-checks-for) · every commit…", "url": "https://wpnews.pro/news/your-pdfs-are-eating-your-llm-s-tokens-for-breakfast", "canonical_source": "https://dev.to/lovestaco/your-pdfs-are-eating-your-llms-tokens-for-breakfast-1k96", "published_at": "2026-07-18 17:48:36+00:00", "updated_at": "2026-07-18 17:58:00.359496+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["Maneshwar", "git-lrc", "Microsoft", "MarkItDown", "GitHub", "Cursor", "VS Code"], "alternates": {"html": "https://wpnews.pro/news/your-pdfs-are-eating-your-llm-s-tokens-for-breakfast", "markdown": "https://wpnews.pro/news/your-pdfs-are-eating-your-llm-s-tokens-for-breakfast.md", "text": "https://wpnews.pro/news/your-pdfs-are-eating-your-llm-s-tokens-for-breakfast.txt", "jsonld": "https://wpnews.pro/news/your-pdfs-are-eating-your-llm-s-tokens-for-breakfast.jsonld"}}