{"slug": "leantext-up-to-99-fewer-tokens-when-your-coding-agent-reads-the-web", "title": "leantext: up to 99% fewer tokens when your coding agent reads the web", "summary": "A developer released leantext, an open-source tool that strips navigation, footers, scripts and hidden markup from web pages, PDFs and Office files before a coding agent reads them, cutting token counts by up to 99% on pages such as PyPI's NumPy listing (92,348 tokens to 755) while preserving all code blocks and tables. The tool verifies each conversion against the source's structure and falls back to the original page when extraction looks suspicious or savings fall under 15%, and a Claude Code hook routes Read and curl calls through cached Markdown conversions. In end-to-end runs inside Claude Code, the agent answered 21 of 21 tasks correctly with leantext enabled.", "body_md": "Your coding agent opens a docs page to answer one question. It gets the navigation bar, the sidebar, the footer, hidden menus, scripts and markup. Somewhere in there is the answer, and you pay for all of it, in tokens, in context window and in latency.\n\nOn PyPI's NumPy page, the project description is about 2,000 tokens. The whole page, converted to Markdown the usual way, is 92,348. Most of it is a hidden list of every file in every release, with hashes, that you never see in a browser.\n\n**leantext** removes that noise before it reaches your agent, checks that nothing important went with it, and tells you what it saved.\n\n[https://github.com/shouvik12/leantext](https://github.com/shouvik12/leantext)\n\nPoint it at a URL or a file. Markdown comes out on stdout.\n\n```\nleantext https://docs.stripe.com/api/customers/create --report\n# Create a customer\n\n## Request\n\n``` curl\ncurl https://api.stripe.com/v1/customers \\\n  -u \"<<YOUR_SECRET_KEY>>\" \\\n  -d \"name=Jenny Rosen\" \\\n  --data-urlencode \"email=jennyrosen@example.com\"\n```\n\n## Parameters\n\n- `email` (string, optional)\n  Customer's email address. ...\n...\n\nleantext report  (markdown served by the site)\n  raw HTML                        521,378  ->    1,194   100% saved\n  naive Markdown (whole page)       3,958  ->    1,194    70% saved\n```\n\nThe agent gets the request, the response, every parameter with its type, and the return value. Nothing else.\n\nIt handles web pages, PDFs (text and scanned), screenshots (OCR first, a vision model only as a fallback) and Office files.\n\nSavings are measured against the page converted to Markdown the usual way, not against raw HTML, because that's what an agent would otherwise get. Token counts use tiktoken.\n\n| Input | Before | With leantext | Saved | \n|---|---|---|---|\n| PyPI: NumPy | 92,348 | 755 | **99%** | \n| PyPI: uv | 43,177 | 2,501 | **94%** | \n| Next.js README | 7,394 | 452 | **94%** | \n| Screenshot: rich table | 1,999 | 127 | **94%** | \n| Stripe API docs | 3,958 | 1,194 | **70%** | \n| MDN: Promise | 11,901 | 7,503 | **37%** | \n| Python tutorial | 11,950 | 10,746 | 10% | \n| Attention Is All You Need (PDF) | 10,168 | 9,838 | 3% | \n\nI also ran it against reference pages developers open daily:\n\n| Page | Before | With leantext | Saved | \n|---|---|---|---|\n| MDN: `Array.prototype.map()` | 8,265 | 2,754 | **67%** | \n| FastAPI: first steps | 7,671 | 4,104 | **46%** | \n| git: `git rebase` | 24,115 | 17,225 | **29%** | \n| Node.js: `fs` | 129,576 | 102,898 | **21%** | \n| Python: `pathlib` | 23,339 | 22,863 | 2% | \n\nEvery row is lossless: all code blocks and tables from the source are in the output. Pages that are almost pure content, like Python's pathlib reference or a dense research paper, have little to cut, and leantext correctly leaves them alone (more on that below).\n\nIn end-to-end runs inside Claude Code, the agent answered **21 of 21 tasks correctly** with leantext switched on.\n\nSay yes to the hook during install, restart Claude Code, and keep working. When the agent reads a web page, a scanned PDF or an Office file, leantext converts it first:\n\n| When the agent does | The hook does | \n|---|---|\n| `Read page.html` ,`Read scan.pdf` ,`Read spec.docx` | points `Read` at a cached Markdown conversion | \n| `curl -s URL` , `curl -s URL \\ | grep … `,` wget -qO- URL` | \n| anything else (source code, `curl -o` , `curl \\ | jq`, POSTs) | \n\nEvery decision is logged with a reason:\n\n```\ntail -3 ~/.cache/leantext/hook.jsonl\n```\n\nTurn it off any time with `LEANTEXT_HOOK=off`.\n\nThis is the part I care about most. Cutting tokens is easy if you don't mind cutting the answer too.\n\nAfter every conversion, leantext compares the output's structure with the source: code blocks, tables, headings, how much text survived, whether the page title is there. It also recognises bot walls and pages that only render with JavaScript. If anything looks off, or the saving is under 15%, the hook steps aside and the agent reads the original.\n\n```\nleantext: extraction suspicious: code blocks: 83 in source, 14 in output\n```\n\nSo the worst case is the same as not having leantext installed.\n\nThat check paid for itself before launch. When I first ran the table above, Python's pathlib page kept only 14 of its 83 code examples intact. Sphinx puts examples inside `<dl><dd>` API blocks, and the extractor flattened them into the surrounding prose. Node's docs lost 116 of 123 version-history tables, which sit inside collapsed `<details>` boxes. Both were flagged, so no agent would have read the broken version.\n\nThe fix: before extraction, every `<pre>` and data table is swapped for a placeholder. The extractor still decides what's main content (so sidebar code is still dropped), and afterwards the blocks go back in verbatim, with their language tags. Python's page is now there with all 83 examples, as a regression test.\n\nAbout 20 seconds on a normal connection, 39 MB on a Mac, no system packages needed:\n\n```\ngit clone https://github.com/shouvik12/leantext\ncd leantext && ./install.sh\n```\n\nThe installer asks whether to add the Claude Code hook.\n\nThe best test is the docs you read most. Run it with `--report`, and if something comes out wrong or gets flagged, open an issue with the URL. The flag tells you exactly what went missing, which makes it quick to fix.\n\nleantext is Apache-2.0: [github.com/shouvik12/leantext](https://github.com/shouvik12/leantext)", "url": "https://wpnews.pro/news/leantext-up-to-99-fewer-tokens-when-your-coding-agent-reads-the-web", "canonical_source": "https://dev.to/shouvik12/leantext-up-to-99-fewer-tokens-when-your-coding-agent-reads-the-web-4gk1", "published_at": "2026-10-24 12:08:40+00:00", "updated_at": "2026-10-24 12:12:40.252455+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-crawlers"], "entities": ["leantext", "Claude Code", "PyPI", "NumPy", "Stripe", "MDN", "FastAPI", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/leantext-up-to-99-fewer-tokens-when-your-coding-agent-reads-the-web", "markdown": "https://wpnews.pro/news/leantext-up-to-99-fewer-tokens-when-your-coding-agent-reads-the-web.md", "text": "https://wpnews.pro/news/leantext-up-to-99-fewer-tokens-when-your-coding-agent-reads-the-web.txt", "jsonld": "https://wpnews.pro/news/leantext-up-to-99-fewer-tokens-when-your-coding-agent-reads-the-web.jsonld"}}