leantext: up to 99% fewer tokens when your coding agent reads the web

A developer released leantext, an open-source tool that strips navigation, footers, scripts and hidden markup from web pages, PDFs and Office files before a coding agent reads them, cutting token counts by up to 99% on pages such as PyPI's NumPy listing (92,348 tokens to 755) while preserving all code blocks and tables. The tool verifies each conversion against the source's structure and falls back to the original page when extraction looks suspicious or savings fall under 15%, and a Claude Code hook routes Read and curl calls through cached Markdown conversions. In end-to-end runs inside Claude Code, the agent answered 21 of 21 tasks correctly with leantext enabled.

Your coding agent opens a docs page to answer one question. It gets the navigation bar, the sidebar, the footer, hidden menus, scripts and markup. Somewhere in there is the answer, and you pay for all of it, in tokens, in context window and in latency. On PyPI's NumPy page, the project description is about 2,000 tokens. The whole page, converted to Markdown the usual way, is 92,348. Most of it is a hidden list of every file in every release, with hashes, that you never see in a browser. leantext removes that noise before it reaches your agent, checks that nothing important went with it, and tells you what it saved. https://github.com/shouvik12/leantext https://github.com/shouvik12/leantext Point it at a URL or a file. Markdown comes out on stdout. leantext https://docs.stripe.com/api/customers/create --report Create a customer Request curl curl https://api.stripe.com/v1/customers \ -u "<<YOUR SECRET KEY " \ -d "name=Jenny Rosen" \ --data-urlencode "email=jennyrosen@example.com" Parameters - email string, optional Customer's email address. ... ... leantext report markdown served by the site raw HTML 521,378 - 1,194 100% saved naive Markdown whole page 3,958 - 1,194 70% saved The agent gets the request, the response, every parameter with its type, and the return value. Nothing else. It handles web pages, PDFs text and scanned , screenshots OCR first, a vision model only as a fallback and Office files. Savings are measured against the page converted to Markdown the usual way, not against raw HTML, because that's what an agent would otherwise get. Token counts use tiktoken. | Input | Before | With leantext | Saved | |---|---|---|---| | PyPI: NumPy | 92,348 | 755 | 99% | | PyPI: uv | 43,177 | 2,501 | 94% | | Next.js README | 7,394 | 452 | 94% | | Screenshot: rich table | 1,999 | 127 | 94% | | Stripe API docs | 3,958 | 1,194 | 70% | | MDN: Promise | 11,901 | 7,503 | 37% | | Python tutorial | 11,950 | 10,746 | 10% | | Attention Is All You Need PDF | 10,168 | 9,838 | 3% | I also ran it against reference pages developers open daily: | Page | Before | With leantext | Saved | |---|---|---|---| | MDN: Array.prototype.map | 8,265 | 2,754 | 67% | | FastAPI: first steps | 7,671 | 4,104 | 46% | | git: git rebase | 24,115 | 17,225 | 29% | | Node.js: fs | 129,576 | 102,898 | 21% | | Python: pathlib | 23,339 | 22,863 | 2% | Every row is lossless: all code blocks and tables from the source are in the output. Pages that are almost pure content, like Python's pathlib reference or a dense research paper, have little to cut, and leantext correctly leaves them alone more on that below . In end-to-end runs inside Claude Code, the agent answered 21 of 21 tasks correctly with leantext switched on. Say yes to the hook during install, restart Claude Code, and keep working. When the agent reads a web page, a scanned PDF or an Office file, leantext converts it first: | When the agent does | The hook does | |---|---| | Read page.html , Read scan.pdf , Read spec.docx | points Read at a cached Markdown conversion | | curl -s URL , curl -s URL \ | grep … , wget -qO- URL | | anything else source code, curl -o , curl \ | jq , POSTs | Every decision is logged with a reason: tail -3 ~/.cache/leantext/hook.jsonl Turn it off any time with LEANTEXT HOOK=off . This is the part I care about most. Cutting tokens is easy if you don't mind cutting the answer too. After every conversion, leantext compares the output's structure with the source: code blocks, tables, headings, how much text survived, whether the page title is there. It also recognises bot walls and pages that only render with JavaScript. If anything looks off, or the saving is under 15%, the hook steps aside and the agent reads the original. leantext: extraction suspicious: code blocks: 83 in source, 14 in output So the worst case is the same as not having leantext installed. That check paid for itself before launch. When I first ran the table above, Python's pathlib page kept only 14 of its 83 code examples intact. Sphinx puts examples inside <dl <dd API blocks, and the extractor flattened them into the surrounding prose. Node's docs lost 116 of 123 version-history tables, which sit inside collapsed <details boxes. Both were flagged, so no agent would have read the broken version. The fix: before extraction, every <pre and data table is swapped for a placeholder. The extractor still decides what's main content so sidebar code is still dropped , and afterwards the blocks go back in verbatim, with their language tags. Python's page is now there with all 83 examples, as a regression test. About 20 seconds on a normal connection, 39 MB on a Mac, no system packages needed: git clone https://github.com/shouvik12/leantext cd leantext && ./install.sh The installer asks whether to add the Claude Code hook. The best test is the docs you read most. Run it with --report , and if something comes out wrong or gets flagged, open an issue with the URL. The flag tells you exactly what went missing, which makes it quick to fix. leantext is Apache-2.0: github.com/shouvik12/leantext https://github.com/shouvik12/leantext