# Show HN: Shadow Web – Cut 64–97% of web page tokens for LLM agents

> Source: <https://github.com/ulinycoin/shadow-web>
> Published: 2026-07-07 08:38:33+00:00

**Cut 64–97% of tokens from web pages before your LLM sees them — then extract structured data.**

Open-source Python SDK that flattens Shadow DOM, builds a typed Action Map with semantic groups, heals broken selectors, and turns HTML tables/forms/lists into clean JSON — no cloud required.

``` python
from shadow_web.compressor import process_html

clean_html, actions, groups = process_html(raw_html)
# ✅ actions = [{"id":"1","type":"button","label":"Buy Now","group":"Checkout"}, ...]
# ✅ 164 → 46 tokens on a typical page

from shadow_web.schema_snap import parse_page

data = parse_page(clean_html)
# ✅ tables: [{columns, types, rows, total_rows}]
# ✅ forms:  [{action, method, fields: [{name, type, required, label}]}]
# ✅ lists:  [{type, items, total}]
```

AI agents need to see web pages. But raw HTML is full of `<script>`

, `<style>`

, inline CSS, and interactive elements buried in Shadow DOM trees that Playwright can't reach. A typical Wikipedia page costs **99K tokens** raw. Your LLM bill doesn't need that.

Shadow Web is what runs **between** the browser and the LLM: a compression layer that keeps only what matters — interactive elements, their labels, and a clean DOM skeleton. **SchemaSnap** then takes that clean HTML and turns it into structured data agents can actually use: table rows, form fields with validation, list items.

| Feature | Raw HTML | Playwright locators | Shadow Web |
|---|---|---|---|
| Token cost (Wikipedia) | 99,343 | — | 16,462 (−83%) |
| Token cost (GitHub Trending) | 167,875 | — | 37,833 (−77%) |
| Shadow DOM readable | ❌ | ❌ partial | ✅ flattened |
| Semantic groups | ❌ | ❌ | ✅ Login / Cart / Nav |
| Self-healing selectors | ❌ | ❌ | ✅ local + LLM fallback |
Tables → JSON columns+rows |
❌ | ❌ | ✅ SchemaSnap |
Forms → fields with validation |
❌ | ❌ | ✅ SchemaSnap |
Lists → typed items |
❌ | ❌ | ✅ SchemaSnap |
| Works offline | ✅ | ✅ | ✅ |
| PyPI package | — | `playwright` |
`shadow-web` |

| You're building … | Why Shadow Web |
|---|---|
| A browser-based AI agent | Action Map + self-healing + SchemaSnap = fewer failures, structured data |
| An MCP tool for Cursor/Claude | Built-in MCP server with 22 tools, one-command setup |
| A Playwright scraper that breaks on every deploy | `heal_local.py` catches DOM drift without LLM cost |
| A Shadow DOM-heavy app (Web components, Lit, Angular) | Read-only flatten — no React/Vue breakage |
An agent that needs data from web pages |
SchemaSnap parses tables, forms, and lists into clean JSON |

```
pip install shadow-web
playwright install chromium
```

**Extras:**

```
pip install "shadow-web[mcp]"          # Cursor/Claude MCP server
pip install "shadow-web[server]"        # FastAPI heal API
pip install "shadow-web[all]"           # everything
```

Full agent loop with token counts at each step:

```
pip install -e ".[mcp]"
playwright install chromium
python examples/golden_path/demo.py
```

Output: raw HTML vs `navigate(minimal)`

+ `schema_session_json`

+ `shadow_query`

— side-by-side token table.

Playbook: [examples/golden_path/CASE.md](/ulinycoin/shadow-web/blob/main/examples/golden_path/CASE.md)

Smoke test (install + unit tests + one live site):

```
bash scripts/smoke_install.sh
python
from shadow_web.compressor import process_html, generate_grouped_xml_map

clean_html, actions, groups = process_html(open("page.html").read())
xml_map = generate_grouped_xml_map("https://example.com", "Example", groups)
print(xml_map)
python
from playwright.sync_api import sync_playwright
from shadow_web.wrapper import ShadowPage

with sync_playwright() as p:
    page = p.chromium.launch(headless=True).new_page()
    page.goto("https://example.com")
    shadow = ShadowPage(page)
    _, xml_map = shadow.refresh()
    print(shadow.capture_stats)  # shadow_hosts, iframes, a11y supplement
result = shadow.query("intent:login", fmt="terse")
# @1 button Sign in [Login Form]
# @2 input[email] Email [Login Form]
shadow.refresh()               # full baseline
shadow.click("3")              # navigate
_, delta_xml = shadow.refresh(diff=True)  # only what changed
python
from shadow_web.schema_snap import parse_page, parse_tables

# Parse everything from a page
data = parse_page(raw_html)
# {
#   "tables": [
#     {
#       "columns": ["Product", "Price", "Stock"],
#       "types": ["string", "currency", "integer"],
#       "rows": [["Widget A", "$19.99", "150"], ...],
#       "total_rows": 12,
#       "column_count": 3
#     }
#   ],
#   "forms": [
#     {
#       "action": "/checkout",
#       "method": "POST",
#       "fields": [
#         {"tag": "input", "type": "email", "name": "email",
#          "required": true, "label": "Email Address"},
#         {"tag": "select", "name": "country", "options": [
#           {"value": "US", "label": "United States"}, ...]}
#       ]
#     }
#   ],
#   "lists": [
#     {"type": "unordered", "items": ["Apples", "Bananas"], "total": 2}
#   ]
# }

# Or export table rows directly
from shadow_web.schema_snap import export_table_json, export_table_csv

records = export_table_json(clean_html, max_rows=50)
# [{"Name": "Alice", "Age": 30}, ...]

csv_text = export_table_csv(clean_html)
# "Name,Age\nAlice,30\n..."
```

**From HTML string (no browser):**

| Tool | Output |
|---|---|
`schema_table(html)` |
columns + types + rows |
`schema_form(html)` |
fields + validation |
`schema_list(html)` |
ul/ol/standalone select |
`schema_page(html)` |
all of the above |
`schema_json(html)` |
`[{column: value}, ...]` |
`schema_csv(html)` |
`"col1,col2\n..."` |

**From browser session** (after `navigate`

/ `snapshot`

):

| Tool | Output |
|---|---|
`schema_session()` |
tables + forms + lists |
`schema_session_json()` |
JSON records |
`schema_session_csv()` |
CSV string |
`get_page_html(max_chars=50000)` |
clean HTML (truncated by default) |

All table tools accept `max_rows=50`

(default). Set `max_rows=0`

for full data.

```
Browser (live DOM)
    │
    ├─ [default] DOM capture — flatten Shadow DOM + same-origin iframes (read-only)
    │              ↓
    ├─ [optional] a11y CDP supplement — catch closed Shadow DOM elements
    │              ↓
    ├─ [Chrome 145+] WebMCP bridge — page exposes document.modelContext.getTools()
    │
    └─→ compressor.py → Action Map (data-sid, type, label, group)
                          ↓
                    shadow_grep.py → filter before LLM
                          ↓
                    schema_snap.py → structured data (tables, forms, lists)
                          ↓
                    heal_local.py → fuzzy selector recovery (no LLM)
                          ↓
                    FastAPI /v1/heal → LLM fallback + verification
```

**No live DOM mutation.** Shadow Web reads your page; it never writes back. React/Vue/Svelte listeners stay intact.

```
shadow_web/
├── compressor.py      # DOM strip + Action Map + semantic groups
├── dom_capture.py     # Shadow DOM / iframe flatten (in-browser, read-only)
├── grouping.py        # Semantic groups (forms, nav, modals)
├── schema_snap.py     # Tables, forms, lists → JSON/CSV export
├── heal_local.py      # Local selector heal + ~/.shadow-web/heal_cache.json
├── query.py           # shadow_grep (type:, intent:, label~, AND)
├── webmcp.py          # WebMCP bridge (Chrome 145+)
├── diff.py            # Page diff (skeleton + delta XML)
├── a11y_capture.py    # CDP Accessibility dual capture
├── verified_heal.py   # Playwright selector verification
├── wrapper.py         # ShadowPage (Playwright)
├── mcp/server.py      # Cursor / Claude MCP tools
└── server/main.py     # FastAPI (/v1/compress, /v1/heal)
```

| Page | Raw HTML (tokens) | Grouped XML (tokens) | Actions | Reduction |
|---|---|---|---|---|
| Hacker News | 8,637 | 6,704 | 227 | −22% (1.3×) |
| Wikipedia (Web Scraping) | 99,343 | 16,462 | 501 | −83% (6.0×) |
| GitHub Trending | 167,875 | 37,833 | 1,290 | −77% (4.4×) |

Run locally: `pip install tiktoken && python benchmarks/run.py`

One-command setup:

```
bash scripts/cursor-setup.sh
```

Or manually:

```
{
  "mcpServers": {
    "shadow-web": {
      "command": "shadow-web-mcp"
    }
  }
}
```

| Category | Tool | What it does |
|---|---|---|
Browse |
`navigate` |
Open URL → snapshot (`detail` : minimal / terse / xml / full) |
`snapshot` |
Refresh page; `diff=true` for delta only |
|
`click` , `fill` |
Interact by `data-sid` |
|
Filter (control plane) |
`shadow_query` |
grep-style filter on live session |
`query_page` |
Alias for shadow_query (json output) | |
`shadow_grep_html` |
Filter raw/clean HTML without browser | |
Compress (offline) |
`compress_html` |
Strip + Action Map + groups |
`compress_html_to_xml` |
Grouped XML from HTML | |
Data (SchemaSnap) |
`schema_table` |
Table columns + types + rows |
`schema_form` |
Form fields + validation | |
`schema_list` |
Lists + standalone selects | |
`schema_page` |
All structured data at once | |
`schema_json` |
Table → JSON records | |
`schema_csv` |
Table → CSV string | |
`schema_session` |
Structured data from browser session | |
`schema_session_json` |
JSON records from session | |
`schema_session_csv` |
CSV from session | |
`get_page_html` |
Clean HTML (`max_chars` default 50000) |
|
Search |
`web_search` |
Brave Search (no API keys) |
WebMCP |
`webmcp_list_tools` |
Chrome 145+ page tools |
`webmcp_execute_tool` |
Execute WebMCP tool by name |

```
navigate(url, detail="minimal")     # ~200 tokens — action_count, page_class
schema_session_json(max_rows=50)    # data plane — table records
shadow_query("intent:login")        # control plane — what to click
click(sid) → snapshot(diff=true)    # delta only after action
```

See [examples/golden_path/CASE.md](/ulinycoin/shadow-web/blob/main/examples/golden_path/CASE.md) for the full playbook.

Shadow Web provides out-of-the-box integration with **browser-use** (the popular agentic framework). It drops token usage by up to 90% and allows the agent to interact with elements inside **Shadow DOM** and iframes using a single line setup.

```
pip install "shadow-web[browser-use]"
python
from browser_use import Agent
from shadow_web import ShadowTools

# Default format is "terse" (compact). Use format="xml" in get_xml_action_map when needed.
tools = ShadowTools(
    heal_api_url="http://localhost:8000/v1/heal",  # Optional: LLM fallback self-healing API
)

agent = Agent(task="...", llm=llm, tools=tools)
```

`get_xml_action_map`

also accepts `query`

(e.g. `intent:login`

) and `format`

(`terse`

| `xml`

) per call.

See [examples/browser_use/](/ulinycoin/shadow-web/blob/main/examples/browser_use) for a complete working implementation.

```
click("3") → binding path → element not found?
    ↓
local heal (fuzzy label + stable attr match, 85% threshold) → no LLM, no cost
    ↓
LLM heal (DeepSeek / OpenAI via /v1/heal) → generates candidate selector
    ↓
selector verified in headless Chromium → cached to ~/.shadow-web/heal_cache.json
```

SchemaSnap is the **data plane** complement to the Action Map **control plane**:

| Layer | Question | Tools |
|---|---|---|
| Control | What can I click? | `navigate` , `shadow_query` , `click` , `fill` |
| Data | What data is on the page? | `schema_session_json` , `schema_session` , `schema_csv` |

Default **max_rows=50** per table. Set `max_rows=0`

for full export when needed.

**Type inference:** `string`

, `integer`

, `number`

, `currency`

, `percentage`

, `date`

, `email`

, `url`

.

| Limitation | Workaround |
|---|---|
Anti-bot / Cloudflare headless |
`page_class: Anti-bot` — stop, don't retry; use headed browser or manual step |
`colspan` / `rowspan` tables |
Column alignment may drift; verify row shape |
JS-rendered grids (AG Grid, React Table) |
May not use `<table>` — use `shadow_query` + Action Map instead |
Closed Shadow DOM |
`navigate(..., capture_mode="dual")` or `"a11y"` |
Cross-origin iframes |
Not accessible — `page_class: Iframe-heavy` |
Token bombs |
Never default to `detail="full"` , `get_page_html(max_chars=0)` , or `max_rows=0` unless debugging |

- You need
**one**`document.querySelector`

— use Playwright directly. - You're building a static site scraper with no interaction.
- The page is plain HTML with no Shadow DOM — overhead isn't worth it.

MIT. Free for anything.

*Stars are the oxygen of open-source. If Shadow Web saved you tokens or debugging time, ★ the repo.*