Your coding agent opens a docs page to answer one question. It gets the navigation bar, the sidebar, the footer, hidden menus, scripts and markup. Somewhere in there is the answer, and you pay for all of it, in tokens, in context window and in latency.
On PyPI's NumPy page, the project description is about 2,000 tokens. The whole page, converted to Markdown the usual way, is 92,348. Most of it is a hidden list of every file in every release, with hashes, that you never see in a browser.
leantext removes that noise before it reaches your agent, checks that nothing important went with it, and tells you what it saved.
https://github.com/shouvik12/leantext
Point it at a URL or a file. Markdown comes out on stdout.
leantext https://docs.stripe.com/api/customers/create --report
## Request
``` curl
curl https://api.stripe.com/v1/customers \
-u "<<YOUR_SECRET_KEY>>" \
-d "name=Jenny Rosen" \
--data-urlencode "email=jennyrosen@example.com"
Parameters #
email(string, optional) Customer's email address. ... ...
leantext report (markdown served by the site) raw HTML 521,378 -> 1,194 100% saved naive Markdown (whole page) 3,958 -> 1,194 70% saved
The agent gets the request, the response, every parameter with its type, and the return value. Nothing else.
It handles web pages, PDFs (text and scanned), screenshots (OCR first, a vision model only as a fallback) and Office files.
Savings are measured against the page converted to Markdown the usual way, not against raw HTML, because that's what an agent would otherwise get. Token counts use tiktoken.
| Input | Before | With leantext | Saved |
|---|---|---|---|
| PyPI: NumPy | 92,348 | 755 | **99%** |
| PyPI: uv | 43,177 | 2,501 | **94%** |
| Next.js README | 7,394 | 452 | **94%** |
| Screenshot: rich table | 1,999 | 127 | **94%** |
| Stripe API docs | 3,958 | 1,194 | **70%** |
| MDN: Promise | 11,901 | 7,503 | **37%** |
| Python tutorial | 11,950 | 10,746 | 10% |
| Attention Is All You Need (PDF) | 10,168 | 9,838 | 3% |
I also ran it against reference pages developers open daily:
| Page | Before | With leantext | Saved |
|---|---|---|---|
| MDN: `Array.prototype.map()` | 8,265 | 2,754 | **67%** |
| FastAPI: first steps | 7,671 | 4,104 | **46%** |
| git: `git rebase` | 24,115 | 17,225 | **29%** |
| Node.js: `fs` | 129,576 | 102,898 | **21%** |
| Python: `pathlib` | 23,339 | 22,863 | 2% |
Every row is lossless: all code blocks and tables from the source are in the output. Pages that are almost pure content, like Python's pathlib reference or a dense research paper, have little to cut, and leantext correctly leaves them alone (more on that below).
In end-to-end runs inside Claude Code, the agent answered **21 of 21 tasks correctly** with leantext switched on.
Say yes to the hook during install, restart Claude Code, and keep working. When the agent reads a web page, a scanned PDF or an Office file, leantext converts it first:
| When the agent does | The hook does |
|---|---|
| `Read page.html` ,`Read scan.pdf` ,`Read spec.docx` | points `Read` at a cached Markdown conversion |
| `curl -s URL` , `curl -s URL \ | grep … `,` wget -qO- URL` |
| anything else (source code, `curl -o` , `curl \ | jq`, POSTs) |
Every decision is logged with a reason:
tail -3 ~/.cache/leantext/hook.jsonl
Turn it off any time with `LEANTEXT_HOOK=off`.
This is the part I care about most. Cutting tokens is easy if you don't mind cutting the answer too.
After every conversion, leantext compares the output's structure with the source: code blocks, tables, headings, how much text survived, whether the page title is there. It also recognises bot walls and pages that only render with JavaScript. If anything looks off, or the saving is under 15%, the hook steps aside and the agent reads the original.
leantext: extraction suspicious: code blocks: 83 in source, 14 in output
So the worst case is the same as not having leantext installed.
That check paid for itself before launch. When I first ran the table above, Python's pathlib page kept only 14 of its 83 code examples intact. Sphinx puts examples inside `<dl><dd>` API blocks, and the extractor flattened them into the surrounding prose. Node's docs lost 116 of 123 version-history tables, which sit inside collapsed `<details>` boxes. Both were flagged, so no agent would have read the broken version.
The fix: before extraction, every `<pre>` and data table is swapped for a placeholder. The extractor still decides what's main content (so sidebar code is still dropped), and afterwards the blocks go back in verbatim, with their language tags. Python's page is now there with all 83 examples, as a regression test.
About 20 seconds on a normal connection, 39 MB on a Mac, no system packages needed:
git clone https://github.com/shouvik12/leantext cd leantext && ./install.sh
The installer asks whether to add the Claude Code hook.
The best test is the docs you read most. Run it with `--report`, and if something comes out wrong or gets flagged, open an issue with the URL. The flag tells you exactly what went missing, which makes it quick to fix.
leantext is Apache-2.0: [github.com/shouvik12/leantext](https://github.com/shouvik12/leantext)source & further reading
dev.to — original article
PauseDock: let a local model keep your place while you rest
Notice Sloth: an open-weight model reads my university notices so I don't have to
7 ways I cut my own LLM agent bill (with real numbers from an agent paying for itself)