Show HN: Shadow Web – Cut 64–97% of web page tokens for LLM agents

Shadow Web, an open-source Python SDK, cuts web page tokens by 64–97% for LLM agents by flattening Shadow DOM, building typed Action Maps, and converting HTML tables, forms, and lists into clean JSON. The tool runs between the browser and the LLM, reducing token costs from 99K to 16K on a Wikipedia page, and works offline without cloud dependencies.

Cut 64–97% of tokens from web pages before your LLM sees them — then extract structured data. Open-source Python SDK that flattens Shadow DOM, builds a typed Action Map with semantic groups, heals broken selectors, and turns HTML tables/forms/lists into clean JSON — no cloud required. python from shadow web.compressor import process html clean html, actions, groups = process html raw html ✅ actions = {"id":"1","type":"button","label":"Buy Now","group":"Checkout"}, ... ✅ 164 → 46 tokens on a typical page from shadow web.schema snap import parse page data = parse page clean html ✅ tables: {columns, types, rows, total rows} ✅ forms: {action, method, fields: {name, type, required, label} } ✅ lists: {type, items, total} AI agents need to see web pages. But raw HTML is full of <script , <style , inline CSS, and interactive elements buried in Shadow DOM trees that Playwright can't reach. A typical Wikipedia page costs 99K tokens raw. Your LLM bill doesn't need that. Shadow Web is what runs between the browser and the LLM: a compression layer that keeps only what matters — interactive elements, their labels, and a clean DOM skeleton. SchemaSnap then takes that clean HTML and turns it into structured data agents can actually use: table rows, form fields with validation, list items. | Feature | Raw HTML | Playwright locators | Shadow Web | |---|---|---|---| | Token cost Wikipedia | 99,343 | — | 16,462 −83% | | Token cost GitHub Trending | 167,875 | — | 37,833 −77% | | Shadow DOM readable | ❌ | ❌ partial | ✅ flattened | | Semantic groups | ❌ | ❌ | ✅ Login / Cart / Nav | | Self-healing selectors | ❌ | ❌ | ✅ local + LLM fallback | Tables → JSON columns+rows | ❌ | ❌ | ✅ SchemaSnap | Forms → fields with validation | ❌ | ❌ | ✅ SchemaSnap | Lists → typed items | ❌ | ❌ | ✅ SchemaSnap | | Works offline | ✅ | ✅ | ✅ | | PyPI package | — | playwright | shadow-web | | You're building … | Why Shadow Web | |---|---| | A browser-based AI agent | Action Map + self-healing + SchemaSnap = fewer failures, structured data | | An MCP tool for Cursor/Claude | Built-in MCP server with 22 tools, one-command setup | | A Playwright scraper that breaks on every deploy | heal local.py catches DOM drift without LLM cost | | A Shadow DOM-heavy app Web components, Lit, Angular | Read-only flatten — no React/Vue breakage | An agent that needs data from web pages | SchemaSnap parses tables, forms, and lists into clean JSON | pip install shadow-web playwright install chromium Extras: pip install "shadow-web mcp " Cursor/Claude MCP server pip install "shadow-web server " FastAPI heal API pip install "shadow-web all " everything Full agent loop with token counts at each step: pip install -e ". mcp " playwright install chromium python examples/golden path/demo.py Output: raw HTML vs navigate minimal + schema session json + shadow query — side-by-side token table. Playbook: examples/golden path/CASE.md /ulinycoin/shadow-web/blob/main/examples/golden path/CASE.md Smoke test install + unit tests + one live site : bash scripts/smoke install.sh python from shadow web.compressor import process html, generate grouped xml map clean html, actions, groups = process html open "page.html" .read xml map = generate grouped xml map "https://example.com", "Example", groups print xml map python from playwright.sync api import sync playwright from shadow web.wrapper import ShadowPage with sync playwright as p: page = p.chromium.launch headless=True .new page page.goto "https://example.com" shadow = ShadowPage page , xml map = shadow.refresh print shadow.capture stats shadow hosts, iframes, a11y supplement result = shadow.query "intent:login", fmt="terse" @1 button Sign in Login Form @2 input email Email Login Form shadow.refresh full baseline shadow.click "3" navigate , delta xml = shadow.refresh diff=True only what changed python from shadow web.schema snap import parse page, parse tables Parse everything from a page data = parse page raw html { "tables": { "columns": "Product", "Price", "Stock" , "types": "string", "currency", "integer" , "rows": "Widget A", "$19.99", "150" , ... , "total rows": 12, "column count": 3 } , "forms": { "action": "/checkout", "method": "POST", "fields": {"tag": "input", "type": "email", "name": "email", "required": true, "label": "Email Address"}, {"tag": "select", "name": "country", "options": {"value": "US", "label": "United States"}, ... } } , "lists": {"type": "unordered", "items": "Apples", "Bananas" , "total": 2} } Or export table rows directly from shadow web.schema snap import export table json, export table csv records = export table json clean html, max rows=50 {"Name": "Alice", "Age": 30}, ... csv text = export table csv clean html "Name,Age\nAlice,30\n..." From HTML string no browser : | Tool | Output | |---|---| schema table html | columns + types + rows | schema form html | fields + validation | schema list html | ul/ol/standalone select | schema page html | all of the above | schema json html | {column: value}, ... | schema csv html | "col1,col2\n..." | From browser session after navigate / snapshot : | Tool | Output | |---|---| schema session | tables + forms + lists | schema session json | JSON records | schema session csv | CSV string | get page html max chars=50000 | clean HTML truncated by default | All table tools accept max rows=50 default . Set max rows=0 for full data. Browser live DOM │ ├─ default DOM capture — flatten Shadow DOM + same-origin iframes read-only │ ↓ ├─ optional a11y CDP supplement — catch closed Shadow DOM elements │ ↓ ├─ Chrome 145+ WebMCP bridge — page exposes document.modelContext.getTools │ └─→ compressor.py → Action Map data-sid, type, label, group ↓ shadow grep.py → filter before LLM ↓ schema snap.py → structured data tables, forms, lists ↓ heal local.py → fuzzy selector recovery no LLM ↓ FastAPI /v1/heal → LLM fallback + verification No live DOM mutation. Shadow Web reads your page; it never writes back. React/Vue/Svelte listeners stay intact. shadow web/ ├── compressor.py DOM strip + Action Map + semantic groups ├── dom capture.py Shadow DOM / iframe flatten in-browser, read-only ├── grouping.py Semantic groups forms, nav, modals ├── schema snap.py Tables, forms, lists → JSON/CSV export ├── heal local.py Local selector heal + ~/.shadow-web/heal cache.json ├── query.py shadow grep type:, intent:, label~, AND ├── webmcp.py WebMCP bridge Chrome 145+ ├── diff.py Page diff skeleton + delta XML ├── a11y capture.py CDP Accessibility dual capture ├── verified heal.py Playwright selector verification ├── wrapper.py ShadowPage Playwright ├── mcp/server.py Cursor / Claude MCP tools └── server/main.py FastAPI /v1/compress, /v1/heal | Page | Raw HTML tokens | Grouped XML tokens | Actions | Reduction | |---|---|---|---|---| | Hacker News | 8,637 | 6,704 | 227 | −22% 1.3× | | Wikipedia Web Scraping | 99,343 | 16,462 | 501 | −83% 6.0× | | GitHub Trending | 167,875 | 37,833 | 1,290 | −77% 4.4× | Run locally: pip install tiktoken && python benchmarks/run.py One-command setup: bash scripts/cursor-setup.sh Or manually: { "mcpServers": { "shadow-web": { "command": "shadow-web-mcp" } } } | Category | Tool | What it does | |---|---|---| Browse | navigate | Open URL → snapshot detail : minimal / terse / xml / full | snapshot | Refresh page; diff=true for delta only | | click , fill | Interact by data-sid | | Filter control plane | shadow query | grep-style filter on live session | query page | Alias for shadow query json output | | shadow grep html | Filter raw/clean HTML without browser | | Compress offline | compress html | Strip + Action Map + groups | compress html to xml | Grouped XML from HTML | | Data SchemaSnap | schema table | Table columns + types + rows | schema form | Form fields + validation | | schema list | Lists + standalone selects | | schema page | All structured data at once | | schema json | Table → JSON records | | schema csv | Table → CSV string | | schema session | Structured data from browser session | | schema session json | JSON records from session | | schema session csv | CSV from session | | get page html | Clean HTML max chars default 50000 | | Search | web search | Brave Search no API keys | WebMCP | webmcp list tools | Chrome 145+ page tools | webmcp execute tool | Execute WebMCP tool by name | navigate url, detail="minimal" ~200 tokens — action count, page class schema session json max rows=50 data plane — table records shadow query "intent:login" control plane — what to click click sid → snapshot diff=true delta only after action See examples/golden path/CASE.md /ulinycoin/shadow-web/blob/main/examples/golden path/CASE.md for the full playbook. Shadow Web provides out-of-the-box integration with browser-use the popular agentic framework . It drops token usage by up to 90% and allows the agent to interact with elements inside Shadow DOM and iframes using a single line setup. pip install "shadow-web browser-use " python from browser use import Agent from shadow web import ShadowTools Default format is "terse" compact . Use format="xml" in get xml action map when needed. tools = ShadowTools heal api url="http://localhost:8000/v1/heal", Optional: LLM fallback self-healing API agent = Agent task="...", llm=llm, tools=tools get xml action map also accepts query e.g. intent:login and format terse | xml per call. See examples/browser use/ /ulinycoin/shadow-web/blob/main/examples/browser use for a complete working implementation. click "3" → binding path → element not found? ↓ local heal fuzzy label + stable attr match, 85% threshold → no LLM, no cost ↓ LLM heal DeepSeek / OpenAI via /v1/heal → generates candidate selector ↓ selector verified in headless Chromium → cached to ~/.shadow-web/heal cache.json SchemaSnap is the data plane complement to the Action Map control plane : | Layer | Question | Tools | |---|---|---| | Control | What can I click? | navigate , shadow query , click , fill | | Data | What data is on the page? | schema session json , schema session , schema csv | Default max rows=50 per table. Set max rows=0 for full export when needed. Type inference: string , integer , number , currency , percentage , date , email , url . | Limitation | Workaround | |---|---| Anti-bot / Cloudflare headless | page class: Anti-bot — stop, don't retry; use headed browser or manual step | colspan / rowspan tables | Column alignment may drift; verify row shape | JS-rendered grids AG Grid, React Table | May not use <table — use shadow query + Action Map instead | Closed Shadow DOM | navigate ..., capture mode="dual" or "a11y" | Cross-origin iframes | Not accessible — page class: Iframe-heavy | Token bombs | Never default to detail="full" , get page html max chars=0 , or max rows=0 unless debugging | - You need one document.querySelector — use Playwright directly. - You're building a static site scraper with no interaction. - The page is plain HTML with no Shadow DOM — overhead isn't worth it. MIT. Free for anything. Stars are the oxygen of open-source. If Shadow Web saved you tokens or debugging time, ★ the repo.