{"slug": "show-hn-shadow-web-cut-64-97-of-web-page-tokens-for-llm-agents", "title": "Show HN: Shadow Web – Cut 64–97% of web page tokens for LLM agents", "summary": "Shadow Web, an open-source Python SDK, cuts web page tokens by 64–97% for LLM agents by flattening Shadow DOM, building typed Action Maps, and converting HTML tables, forms, and lists into clean JSON. The tool runs between the browser and the LLM, reducing token costs from 99K to 16K on a Wikipedia page, and works offline without cloud dependencies.", "body_md": "**Cut 64–97% of tokens from web pages before your LLM sees them — then extract structured data.**\n\nOpen-source Python SDK that flattens Shadow DOM, builds a typed Action Map with semantic groups, heals broken selectors, and turns HTML tables/forms/lists into clean JSON — no cloud required.\n\n``` python\nfrom shadow_web.compressor import process_html\n\nclean_html, actions, groups = process_html(raw_html)\n# ✅ actions = [{\"id\":\"1\",\"type\":\"button\",\"label\":\"Buy Now\",\"group\":\"Checkout\"}, ...]\n# ✅ 164 → 46 tokens on a typical page\n\nfrom shadow_web.schema_snap import parse_page\n\ndata = parse_page(clean_html)\n# ✅ tables: [{columns, types, rows, total_rows}]\n# ✅ forms:  [{action, method, fields: [{name, type, required, label}]}]\n# ✅ lists:  [{type, items, total}]\n```\n\nAI agents need to see web pages. But raw HTML is full of `<script>`\n\n, `<style>`\n\n, inline CSS, and interactive elements buried in Shadow DOM trees that Playwright can't reach. A typical Wikipedia page costs **99K tokens** raw. Your LLM bill doesn't need that.\n\nShadow Web is what runs **between** the browser and the LLM: a compression layer that keeps only what matters — interactive elements, their labels, and a clean DOM skeleton. **SchemaSnap** then takes that clean HTML and turns it into structured data agents can actually use: table rows, form fields with validation, list items.\n\n| Feature | Raw HTML | Playwright locators | Shadow Web |\n|---|---|---|---|\n| Token cost (Wikipedia) | 99,343 | — | 16,462 (−83%) |\n| Token cost (GitHub Trending) | 167,875 | — | 37,833 (−77%) |\n| Shadow DOM readable | ❌ | ❌ partial | ✅ flattened |\n| Semantic groups | ❌ | ❌ | ✅ Login / Cart / Nav |\n| Self-healing selectors | ❌ | ❌ | ✅ local + LLM fallback |\nTables → JSON columns+rows |\n❌ | ❌ | ✅ SchemaSnap |\nForms → fields with validation |\n❌ | ❌ | ✅ SchemaSnap |\nLists → typed items |\n❌ | ❌ | ✅ SchemaSnap |\n| Works offline | ✅ | ✅ | ✅ |\n| PyPI package | — | `playwright` |\n`shadow-web` |\n\n| You're building … | Why Shadow Web |\n|---|---|\n| A browser-based AI agent | Action Map + self-healing + SchemaSnap = fewer failures, structured data |\n| An MCP tool for Cursor/Claude | Built-in MCP server with 22 tools, one-command setup |\n| A Playwright scraper that breaks on every deploy | `heal_local.py` catches DOM drift without LLM cost |\n| A Shadow DOM-heavy app (Web components, Lit, Angular) | Read-only flatten — no React/Vue breakage |\nAn agent that needs data from web pages |\nSchemaSnap parses tables, forms, and lists into clean JSON |\n\n```\npip install shadow-web\nplaywright install chromium\n```\n\n**Extras:**\n\n```\npip install \"shadow-web[mcp]\"          # Cursor/Claude MCP server\npip install \"shadow-web[server]\"        # FastAPI heal API\npip install \"shadow-web[all]\"           # everything\n```\n\nFull agent loop with token counts at each step:\n\n```\npip install -e \".[mcp]\"\nplaywright install chromium\npython examples/golden_path/demo.py\n```\n\nOutput: raw HTML vs `navigate(minimal)`\n\n+ `schema_session_json`\n\n+ `shadow_query`\n\n— side-by-side token table.\n\nPlaybook: [examples/golden_path/CASE.md](/ulinycoin/shadow-web/blob/main/examples/golden_path/CASE.md)\n\nSmoke test (install + unit tests + one live site):\n\n```\nbash scripts/smoke_install.sh\npython\nfrom shadow_web.compressor import process_html, generate_grouped_xml_map\n\nclean_html, actions, groups = process_html(open(\"page.html\").read())\nxml_map = generate_grouped_xml_map(\"https://example.com\", \"Example\", groups)\nprint(xml_map)\npython\nfrom playwright.sync_api import sync_playwright\nfrom shadow_web.wrapper import ShadowPage\n\nwith sync_playwright() as p:\n    page = p.chromium.launch(headless=True).new_page()\n    page.goto(\"https://example.com\")\n    shadow = ShadowPage(page)\n    _, xml_map = shadow.refresh()\n    print(shadow.capture_stats)  # shadow_hosts, iframes, a11y supplement\nresult = shadow.query(\"intent:login\", fmt=\"terse\")\n# @1 button Sign in [Login Form]\n# @2 input[email] Email [Login Form]\nshadow.refresh()               # full baseline\nshadow.click(\"3\")              # navigate\n_, delta_xml = shadow.refresh(diff=True)  # only what changed\npython\nfrom shadow_web.schema_snap import parse_page, parse_tables\n\n# Parse everything from a page\ndata = parse_page(raw_html)\n# {\n#   \"tables\": [\n#     {\n#       \"columns\": [\"Product\", \"Price\", \"Stock\"],\n#       \"types\": [\"string\", \"currency\", \"integer\"],\n#       \"rows\": [[\"Widget A\", \"$19.99\", \"150\"], ...],\n#       \"total_rows\": 12,\n#       \"column_count\": 3\n#     }\n#   ],\n#   \"forms\": [\n#     {\n#       \"action\": \"/checkout\",\n#       \"method\": \"POST\",\n#       \"fields\": [\n#         {\"tag\": \"input\", \"type\": \"email\", \"name\": \"email\",\n#          \"required\": true, \"label\": \"Email Address\"},\n#         {\"tag\": \"select\", \"name\": \"country\", \"options\": [\n#           {\"value\": \"US\", \"label\": \"United States\"}, ...]}\n#       ]\n#     }\n#   ],\n#   \"lists\": [\n#     {\"type\": \"unordered\", \"items\": [\"Apples\", \"Bananas\"], \"total\": 2}\n#   ]\n# }\n\n# Or export table rows directly\nfrom shadow_web.schema_snap import export_table_json, export_table_csv\n\nrecords = export_table_json(clean_html, max_rows=50)\n# [{\"Name\": \"Alice\", \"Age\": 30}, ...]\n\ncsv_text = export_table_csv(clean_html)\n# \"Name,Age\\nAlice,30\\n...\"\n```\n\n**From HTML string (no browser):**\n\n| Tool | Output |\n|---|---|\n`schema_table(html)` |\ncolumns + types + rows |\n`schema_form(html)` |\nfields + validation |\n`schema_list(html)` |\nul/ol/standalone select |\n`schema_page(html)` |\nall of the above |\n`schema_json(html)` |\n`[{column: value}, ...]` |\n`schema_csv(html)` |\n`\"col1,col2\\n...\"` |\n\n**From browser session** (after `navigate`\n\n/ `snapshot`\n\n):\n\n| Tool | Output |\n|---|---|\n`schema_session()` |\ntables + forms + lists |\n`schema_session_json()` |\nJSON records |\n`schema_session_csv()` |\nCSV string |\n`get_page_html(max_chars=50000)` |\nclean HTML (truncated by default) |\n\nAll table tools accept `max_rows=50`\n\n(default). Set `max_rows=0`\n\nfor full data.\n\n```\nBrowser (live DOM)\n    │\n    ├─ [default] DOM capture — flatten Shadow DOM + same-origin iframes (read-only)\n    │              ↓\n    ├─ [optional] a11y CDP supplement — catch closed Shadow DOM elements\n    │              ↓\n    ├─ [Chrome 145+] WebMCP bridge — page exposes document.modelContext.getTools()\n    │\n    └─→ compressor.py → Action Map (data-sid, type, label, group)\n                          ↓\n                    shadow_grep.py → filter before LLM\n                          ↓\n                    schema_snap.py → structured data (tables, forms, lists)\n                          ↓\n                    heal_local.py → fuzzy selector recovery (no LLM)\n                          ↓\n                    FastAPI /v1/heal → LLM fallback + verification\n```\n\n**No live DOM mutation.** Shadow Web reads your page; it never writes back. React/Vue/Svelte listeners stay intact.\n\n```\nshadow_web/\n├── compressor.py      # DOM strip + Action Map + semantic groups\n├── dom_capture.py     # Shadow DOM / iframe flatten (in-browser, read-only)\n├── grouping.py        # Semantic groups (forms, nav, modals)\n├── schema_snap.py     # Tables, forms, lists → JSON/CSV export\n├── heal_local.py      # Local selector heal + ~/.shadow-web/heal_cache.json\n├── query.py           # shadow_grep (type:, intent:, label~, AND)\n├── webmcp.py          # WebMCP bridge (Chrome 145+)\n├── diff.py            # Page diff (skeleton + delta XML)\n├── a11y_capture.py    # CDP Accessibility dual capture\n├── verified_heal.py   # Playwright selector verification\n├── wrapper.py         # ShadowPage (Playwright)\n├── mcp/server.py      # Cursor / Claude MCP tools\n└── server/main.py     # FastAPI (/v1/compress, /v1/heal)\n```\n\n| Page | Raw HTML (tokens) | Grouped XML (tokens) | Actions | Reduction |\n|---|---|---|---|---|\n| Hacker News | 8,637 | 6,704 | 227 | −22% (1.3×) |\n| Wikipedia (Web Scraping) | 99,343 | 16,462 | 501 | −83% (6.0×) |\n| GitHub Trending | 167,875 | 37,833 | 1,290 | −77% (4.4×) |\n\nRun locally: `pip install tiktoken && python benchmarks/run.py`\n\nOne-command setup:\n\n```\nbash scripts/cursor-setup.sh\n```\n\nOr manually:\n\n```\n{\n  \"mcpServers\": {\n    \"shadow-web\": {\n      \"command\": \"shadow-web-mcp\"\n    }\n  }\n}\n```\n\n| Category | Tool | What it does |\n|---|---|---|\nBrowse |\n`navigate` |\nOpen URL → snapshot (`detail` : minimal / terse / xml / full) |\n`snapshot` |\nRefresh page; `diff=true` for delta only |\n|\n`click` , `fill` |\nInteract by `data-sid` |\n|\nFilter (control plane) |\n`shadow_query` |\ngrep-style filter on live session |\n`query_page` |\nAlias for shadow_query (json output) | |\n`shadow_grep_html` |\nFilter raw/clean HTML without browser | |\nCompress (offline) |\n`compress_html` |\nStrip + Action Map + groups |\n`compress_html_to_xml` |\nGrouped XML from HTML | |\nData (SchemaSnap) |\n`schema_table` |\nTable columns + types + rows |\n`schema_form` |\nForm fields + validation | |\n`schema_list` |\nLists + standalone selects | |\n`schema_page` |\nAll structured data at once | |\n`schema_json` |\nTable → JSON records | |\n`schema_csv` |\nTable → CSV string | |\n`schema_session` |\nStructured data from browser session | |\n`schema_session_json` |\nJSON records from session | |\n`schema_session_csv` |\nCSV from session | |\n`get_page_html` |\nClean HTML (`max_chars` default 50000) |\n|\nSearch |\n`web_search` |\nBrave Search (no API keys) |\nWebMCP |\n`webmcp_list_tools` |\nChrome 145+ page tools |\n`webmcp_execute_tool` |\nExecute WebMCP tool by name |\n\n```\nnavigate(url, detail=\"minimal\")     # ~200 tokens — action_count, page_class\nschema_session_json(max_rows=50)    # data plane — table records\nshadow_query(\"intent:login\")        # control plane — what to click\nclick(sid) → snapshot(diff=true)    # delta only after action\n```\n\nSee [examples/golden_path/CASE.md](/ulinycoin/shadow-web/blob/main/examples/golden_path/CASE.md) for the full playbook.\n\nShadow Web provides out-of-the-box integration with **browser-use** (the popular agentic framework). It drops token usage by up to 90% and allows the agent to interact with elements inside **Shadow DOM** and iframes using a single line setup.\n\n```\npip install \"shadow-web[browser-use]\"\npython\nfrom browser_use import Agent\nfrom shadow_web import ShadowTools\n\n# Default format is \"terse\" (compact). Use format=\"xml\" in get_xml_action_map when needed.\ntools = ShadowTools(\n    heal_api_url=\"http://localhost:8000/v1/heal\",  # Optional: LLM fallback self-healing API\n)\n\nagent = Agent(task=\"...\", llm=llm, tools=tools)\n```\n\n`get_xml_action_map`\n\nalso accepts `query`\n\n(e.g. `intent:login`\n\n) and `format`\n\n(`terse`\n\n| `xml`\n\n) per call.\n\nSee [examples/browser_use/](/ulinycoin/shadow-web/blob/main/examples/browser_use) for a complete working implementation.\n\n```\nclick(\"3\") → binding path → element not found?\n    ↓\nlocal heal (fuzzy label + stable attr match, 85% threshold) → no LLM, no cost\n    ↓\nLLM heal (DeepSeek / OpenAI via /v1/heal) → generates candidate selector\n    ↓\nselector verified in headless Chromium → cached to ~/.shadow-web/heal_cache.json\n```\n\nSchemaSnap is the **data plane** complement to the Action Map **control plane**:\n\n| Layer | Question | Tools |\n|---|---|---|\n| Control | What can I click? | `navigate` , `shadow_query` , `click` , `fill` |\n| Data | What data is on the page? | `schema_session_json` , `schema_session` , `schema_csv` |\n\nDefault **max_rows=50** per table. Set `max_rows=0`\n\nfor full export when needed.\n\n**Type inference:** `string`\n\n, `integer`\n\n, `number`\n\n, `currency`\n\n, `percentage`\n\n, `date`\n\n, `email`\n\n, `url`\n\n.\n\n| Limitation | Workaround |\n|---|---|\nAnti-bot / Cloudflare headless |\n`page_class: Anti-bot` — stop, don't retry; use headed browser or manual step |\n`colspan` / `rowspan` tables |\nColumn alignment may drift; verify row shape |\nJS-rendered grids (AG Grid, React Table) |\nMay not use `<table>` — use `shadow_query` + Action Map instead |\nClosed Shadow DOM |\n`navigate(..., capture_mode=\"dual\")` or `\"a11y\"` |\nCross-origin iframes |\nNot accessible — `page_class: Iframe-heavy` |\nToken bombs |\nNever default to `detail=\"full\"` , `get_page_html(max_chars=0)` , or `max_rows=0` unless debugging |\n\n- You need\n**one**`document.querySelector`\n\n— use Playwright directly. - You're building a static site scraper with no interaction.\n- The page is plain HTML with no Shadow DOM — overhead isn't worth it.\n\nMIT. Free for anything.\n\n*Stars are the oxygen of open-source. If Shadow Web saved you tokens or debugging time, ★ the repo.*", "url": "https://wpnews.pro/news/show-hn-shadow-web-cut-64-97-of-web-page-tokens-for-llm-agents", "canonical_source": "https://github.com/ulinycoin/shadow-web", "published_at": "2026-07-07 08:38:33+00:00", "updated_at": "2026-07-07 09:00:21.191637+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "large-language-models", "natural-language-processing"], "entities": ["Shadow Web", "Playwright", "SchemaSnap", "PyPI", "FastAPI", "Wikipedia", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/show-hn-shadow-web-cut-64-97-of-web-page-tokens-for-llm-agents", "markdown": "https://wpnews.pro/news/show-hn-shadow-web-cut-64-97-of-web-page-tokens-for-llm-agents.md", "text": "https://wpnews.pro/news/show-hn-shadow-web-cut-64-97-of-web-page-tokens-for-llm-agents.txt", "jsonld": "https://wpnews.pro/news/show-hn-shadow-web-cut-64-97-of-web-page-tokens-for-llm-agents.jsonld"}}