shadow-web MCP server
Commit Oct 24, 2026Compresses web pages and extracts structured data to reduce LLM token costs while enabling AI agents to interact with Shadow DOM and complex forms.
Also known as ulinycoin/shadow-web
- Stars
- 10
- Tools
- —
- Downloads, 30 days
- —
- Latest version
- —
- Last commit
- Oct 24, 2026
- License
- MIT
What is shadow-web MCP?
shadow-web compresses web pages and extracts structured data to reduce LLM token costs while enabling AI agents to interact with Shadow DOM and complex forms.
Summary generated from the project's README
About shadow-web MCP server
Shadow Web acts as a compression layer between web browsers and large language models to reduce token consumption. It processes raw HTML to strip away unnecessary elements like scripts and styles, focusing instead on interactive elements, labels, and a clean DOM skeleton. This allows developers to feed significantly smaller, more relevant data to AI agents, often reducing token costs by 77% to 83% compared to raw HTML.
The server provides a suite of 27 tools for use in environments like Cursor or Claude via the Model Context Protocol. It includes specialized capabilities for flattening Shadow DOM trees, building typed Action Maps, and using the SchemaSnap package to convert tables, forms, and lists into structured JSON. Users can run it as a local MCP server, a standalone Python SDK, or via the shadow-web exec command for bounded browser tasks.
Setup requires installing the shadow-web package and Playwright. For remote or production use, users must configure SHADOW_WEB_API_KEYS. The system supports various models through OpenAI-compatible APIs, including DeepSeek, OpenAI, or OpenRouter, and defaults to the Camoufox browser for enhanced fingerprinting.
What you can do with it
- 1Reduce LLM token costsA developer can compress a Wikipedia page to significantly lower the number of tokens sent to an AI model.
- 2Extract structured web dataAn AI agent can transform complex web tables and forms into clean, usable JSON objects.
- 3Automate browser-based tasksA user can execute a single command to have an agent navigate a website and answer a specific question.
Written from the project's README. Every use case is checked against a quote from it.
Configuration
DEEPSEEK_API_KEYSHADOW_WEB_API_KEYS
| Name | Description | |
|---|---|---|
| DEEPSEEK_API_KEY | — | |
| SHADOW_WEB_API_KEYS | — |
Automated checks
Ran Nov 14, 2026Full report →Not verified yet. A server needs a score of 80, no serious finding, and enough data for us to check. How checks work · What to fix for more points
- Source repository linked and reachableulinycoin/shadow-web is public and reachablegithub.com/ulinycoin/shadow-web ↗Pass
- Package names this repository as its sourceNot checked
- Build provenance (Sigstore attestation)Not checked
- Open source license (OSI)MITspdx.org/licenses/MIT.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-10-24github.com/ulinycoin/shadow-web ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNot checked
- No critical advisory in the package or its dependenciesNot checked
- No hidden instructions to the model0 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNot checked
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubShadow Web
Compress web pages 4–6× before your LLM sees them — then extract structured data.
Open-source Python SDK + MCP + standalone shadow-web exec agent: flattens Shadow DOM, builds a typed Action Map, indexes rendered text on demand, heals broken selectors, and turns tables/forms/lists into clean JSON — Camoufox by default, no cloud required.
v0.4.4 — the extraction core moved out of the tree: tables/forms/lists are parsed by the standalone
schemasnappackage (installed automatically bypip install shadow-web), andshadow_web.schema_snapis a 64-line facade over it — same imports, one implementation. Running this suite against the shared core surfaced four correctness bugs in it (phantom empty tables, a swallowed ARIA header row, dropped single-column grids, zero-padded ids rewritten asinteger) — all fixed, all in 0.1.9. 0.4.3 was the verified-review pass — the scanner finally reachesscript/link/integrity(raw_html), field classification no longer routes profile values into the wrong field, MCP sessions cannot recycle keys, and anagent_runtimeout actually stops the work.
from shadow_web.compressor import process_html
clean_html, actions, groups = process_html(raw_html)
# ✅ actions = [{"id":"1","type":"button","label":"Buy Now","group":"Checkout"}, ...]
# ✅ 164 → 46 tokens on a typical page
from shadow_web.schema_snap import parse_page
data = parse_page(clean_html)
# ✅ tables: [{columns, types, rows, total_rows}]
# ✅ forms: [{action, method, fields: [{name, type, required, label}]}]
# ✅ lists: [{type, items, total}]
Pain point
AI agents need to see web pages. But raw HTML is full of <script>, <style>, inline CSS, and interactive elements buried in Shadow DOM trees that Playwright can't reach. A typical Wikipedia page costs 99K tokens raw. Your LLM bill doesn't need that.
Shadow Web is what runs between the browser and the LLM: a compression layer that keeps only what matters — interactive elements, their labels, and a clean DOM skeleton. SchemaSnap then takes that clean HTML and turns it into structured data agents can actually use: table rows, form fields with validation, list items.
What you get
| Feature | Raw HTML | Playwright locators | Shadow Web |
|---|---|---|---|
| Token cost (Wikipedia) | 99,343 | — | 16,462 (−83%) |
| Token cost (GitHub Trending) | 167,875 | — | 37,833 (−77%) |
| Long-form via Content Index | full page | — | ~600t outline → fetch blocks |
| Shadow DOM readable | ❌ | ❌ partial | ✅ flattened |
| Semantic groups | ❌ | ❌ | ✅ Login / Cart / Nav |
| Self-healing selectors | ❌ | ❌ | ✅ local + LLM fallback |
| Consent + lazy hydration | ❌ | manual | ✅ readiness + scroll-until-content |
| Catalog / feed outline ranking | ❌ | ❌ | ✅ cards / feeds before chrome |
| Tables → JSON columns+rows | ❌ | ❌ | ✅ SchemaSnap |
| Forms → fields with validation | ❌ | ❌ | ✅ SchemaSnap |
| Lists → typed items | ❌ | ❌ | ✅ SchemaSnap |
| Works offline | ✅ | ✅ | ✅ |
| PyPI package | — | playwright |
shadow-web |
Who this is for
| You're building … | Why Shadow Web |
|---|---|
| A browser-based AI agent | Action Map + self-healing + SchemaSnap = fewer failures, structured data |
| A coding agent that needs one-shot browsing | shadow-web exec — one shell call, one JSON answer (transcript stays inside) |
| An MCP tool for Cursor/Claude | Built-in MCP server with 27 tools, one-command setup |
| A Playwright scraper that breaks on every deploy | heal_local.py catches DOM drift without LLM cost |
| A Shadow DOM-heavy app (Web components, Lit, Angular) | Read-only flatten — no React/Vue breakage |
| An agent that needs data from web pages | SchemaSnap parses tables, forms, and lists into clean JSON |
| Long articles, catalogs, social feeds | Rendered Text Index + ranked content_outline (cards / feeds) |
| Attack surface / security recon | security_scan.py + CLI — forms, links, headers, cookies, page_class (not pentest) |
| Competitor monitoring & SEO content | Shallow multi-site scans with token-bounded Action Map |
| SaaS onboarding automation | AgentOps form fill — schema knows fields, LLM only picks values |
Quick install
pip install shadow-web
playwright install chromium
Extras:
pip install "shadow-web[mcp]" # Cursor/Claude MCP server
pip install "shadow-web[server]" # FastAPI heal API
pip install "shadow-web[agent]" # standalone exec agent loop
pip install "shadow-web[all]" # everything
Run the optional API locally:
export DEEPSEEK_API_KEY="..."
export SHADOW_WEB_API_KEYS="local-secret"
shadow-web-server
Remote and production requests are rejected unless SHADOW_WEB_API_KEYS is configured.
Standalone agent (shadow-web exec)
Hand a bounded browser task to the built-in observe→act→verify loop. One command in, one JSON object out — browsing transcript stays inside the sub-agent. Same thick entrypoint over MCP: agent_run(goal).
This is for answering a question / short flow — not a full-site scrape. Large volumes stay on thin tools + pagination (schema_session, content_outline(offset=…)). The envelope always reports honesty fields: complete, truncated, stop_reason, optional next_hint.
Unlike a11y-tree browser wrappers, observations are HTML Action Maps + SchemaSnap (works on any HTML, not only Playwright CDP). Default browser is Camoufox (Firefox fingerprint); Chromium is opt-in.
# 30-second demo (Chromium path — no camoufox fetch)
pip install "shadow-web[agent]" && playwright install chromium
export DEEPSEEK_API_KEY="..." # or OPENAI_API_KEY / OPENROUTER_API_KEY
shadow-web exec "What is the heading on https://example.com?" \
--browser chromium --model deepseek-chat
# one-liner from a clone:
# bash scripts/demo_30s.sh
# Default production path (Camoufox fingerprint):
# camoufox fetch && shadow-web exec "…" --model deepseek-chat
An explicit --browser chromium needs no extra flag: the Camoufox→Chromium
fallback switch (SHADOW_WEB_ALLOW_CHROMIUM_FALLBACK=1) only governs what
happens when Camoufox is requested but unavailable, and by default that is a
hard failure rather than a silent fallback.
{
"ok": true,
"answer": "The page title is \"Example Domain\".",
"steps": 2,
"usage": {"prompt_tokens": 1821, "completion_tokens": 144, "total_tokens": 1965},
"url": "https://example.com/",
"browser": "chromium",
"stop_reason": "done",
"complete": true,
"truncated": false
}
| Flag | Default | Notes |
|---|---|---|
--model |
deepseek-v4-flash |
Any OpenAI-compatible model id (or SHADOW_WEB_LLM_MODEL) |
--base-url |
auto from env | DeepSeek / OpenRouter inferred from API key |
--api-key |
env | Else DEEPSEEK_API_KEY / OPENAI_API_KEY / OPENROUTER_API_KEY |
--browser |
camoufox |
chromium for CDP/a11y supplement path |
--max-steps |
20 |
Model turns budget |
--max-tokens |
15000 |
Cumulative LLM token budget, 0 = unlimited (or SHADOW_WEB_MAX_TOKENS) |
--session |
— | Persistent profile ~/.shadow-web/sessions/<name> |
--headed |
off | Show the browser window |
--heal-url |
env | Optional self-heal API (SHADOW_WEB_HEAL_URL) |
--transcript |
off | Include step list in JSON |
--jev |
off | TypeSafe Jev pre-selects cheap steps (needs TYPESAFE_API_KEY); low confidence still goes to the model |
--jev-threshold |
0.6 |
Confidence gate for Jev steps (SHADOW_WEB_JEV_THRESHOLD) |
Dense pages (action_count > 40) auto-return minimal observations + a hint to use shadow_query / schema_page / content_outline then done — same terse pipeline as MCP, not site-specific. SPA catalogs with empty SchemaSnap should use Rendered Text Index (content_outline → content_blocks).
Programmatic: from shadow_web import run_agent_task.
Model env help: shadow-web models.
Demo
Golden path (recommended — run locally)
Full agent loop with token counts at each step:
pip install -e ".[mcp]"
playwright install chromium
python examples/golden_path/demo.py
Output: raw HTML vs navigate(minimal) + schema_session_json + shadow_query — side-by-side token table.
Playbook: examples/golden_path/CASE.md
Attack surface security scan
Automated surface mapping (not penetration testing): forms, links, page_class, HTTP security headers (HSTS, CSP, XFO, nosniff, CORS), cookie flags (Secure, HttpOnly, SameSite), Markdown/JSON reports.
pip install -e ".[mcp]"
playwright install chromium
# Single URL
python scripts/security_surface_scan.py https://example.com
# Shallow same-domain crawl + reports (--no-headers / --no-cookies to skip checks)
python scripts/security_surface_scan.py https://yoursite.com \
--crawl-depth 1 --max-pages 20 \
--json report.json --markdown report.md
Rule engine (importable without browser):
from shadow_web.security_scan import analyze_surface, render_markdown_report
result = analyze_surface(
"https://app.example.com/login",
raw_html=await page.content(), # SRI / mixed content / form fields need the served DOM
clean_html=clean_html, # compressor output: labels, action map, tokens
action_map=actions,
page_class="Static",
)
# findings: FORM_PASSWORD_GET, HEADER_MISSING_CSP, COOKIE_MISSING_HTTPONLY, ...
# Passing only clean_html still works, but adds SCAN_ON_SANITIZED_HTML: the
# compressor strips <script>/<link>/<meta> and integrity/rel/id/required, so
# SRI_MISSING / MIXED_* / form-field findings would be false negatives.
Example output: examples/security_scan/localpdf-full-report.md (20 pages, 0 critical/high on public marketing layer). Header-only sample: localpdf-headers-only.json.
Does not test: XSS, SQLi, auth bypass, or deep TLS/cipher analysis. Use only on authorized targets.
AgentOps form fill
See the full Form Fill section below. Quick start:
python scripts/form_fill_demo.py https://httpbin.org/forms/post \
--profile examples/form_fill/profile.json --json plan.json
Competitor intelligence scan
Token-bounded weekly audit for programmatic SEO / compare pages:
python scripts/localpdf_competitor_scan.py --json reports/scan.json
Playbook: examples/localpdf/CASE.md
Smoke test (install + unit tests + one live site):
bash scripts/smoke_install.sh
Compress a page (3 lines)
from shadow_web.compressor import process_html, generate_grouped_xml_map
clean_html, actions, groups = process_html(open("page.html").read())
xml_map = generate_grouped_xml_map("https://example.com", "Example", groups)
print(xml_map)
Playwright + Shadow DOM flatten
from playwright.sync_api import sync_playwright
from shadow_web.wrapper import ShadowPage
with sync_playwright() as p:
page = p.chromium.launch(headless=True).new_page()
page.goto("https://example.com")
shadow = ShadowPage(page)
_, xml_map = shadow.refresh()
print(shadow.capture_stats) # shadow_hosts, iframes, a11y supplement
shadow_grep — send only what the LLM needs
result = shadow.query("intent:login", fmt="terse")
# @1 button Sign in [Login Form]
# @2 input[email] Email [Login Form]
Stream a delta after clicking
shadow.refresh() # full baseline
shadow.click("3") # navigate
_, delta_xml = shadow.refresh(diff=True) # only what changed
SchemaSnap — extract tables, forms, and lists
from shadow_web.schema_snap import parse_page, parse_tables
# Parse everything from a page
data = parse_page(raw_html)
# {
# "tables": [
# {
# "columns": ["Product", "Price", "Stock"],
# "types": ["string", "currency", "integer"],
# "rows": [["Widget A", "$19.99", "150"], ...],
# "total_rows": 12,
# "column_count": 3
# }
# ],
# "forms": [
# {
# "action": "/checkout",
# "method": "POST",
# "fields": [
# {"tag": "input", "type": "email", "name": "email",
# "required": true, "label": "Email Address"},
# {"tag": "select", "name": "country", "options": [
# {"value": "US", "label": "United States"}, ...]}
# ]
# }
# ],
# "lists": [
# {"type": "unordered", "items": ["Apples", "Bananas"], "total": 2}
# ]
# }
# Or export table rows directly
from shadow_web.schema_snap import export_table_json, export_table_csv
records = export_table_json(clean_html, max_rows=50)
# [{"Name": "Alice", "Age": 30}, ...]
csv_text = export_table_csv(clean_html)
# "Name,Age\nAlice,30\n..."
SchemaSnap in MCP
From HTML string (no browser):
| Tool | Output |
|---|---|
schema_table(html) |
columns + types + rows |
schema_form(html) |
fields + validation |
schema_list(html) |
ul/ol/standalone select |
schema_page(html) |
all of the above |
schema_json(html) |
[{column: value}, ...] |
schema_csv(html) |
"col1,col2\n..." |
From browser session (after navigate / snapshot):
| Tool | Output |
|---|---|
schema_session() |
tables + forms + lists |
schema_session_json() |
JSON records |
schema_session_csv() |
CSV string |
get_page_html(max_chars=50000) |
clean HTML (truncated by default) |
All table tools accept max_rows=50 (default). Set max_rows=0 for full data.
How it works
Browser (live DOM)
│
├─ capture_ready — cookie consent dismiss + wait + scroll-until-content
│ ↓
├─ [default] DOM capture — flatten Shadow DOM + same-origin iframes (read-only)
│ ↓
├─ [optional] a11y CDP supplement — catch closed Shadow DOM elements
│ ↓
├─ [Chrome 145+] WebMCP bridge — page exposes document.modelContext.getTools()
│
└─→ compressor.py → Action Map (data-sid, type, label, group)
↓
content_index.py → ranked outline (p0, p1…; cards/feeds) + fetch
↓
query.py (shadow_grep) → filter Action Map before LLM
↓
schema_snap.py → structured data (tables, forms, lists)
↓
security_scan.py → attack surface rules (forms, links, headers, cookies)
↓
heal_local.py → fuzzy selector recovery (no LLM)
↓
FastAPI /v1/heal → LLM fallback + verification
No live DOM mutation. Shadow Web reads your page; it never writes back. React/Vue/Svelte listeners stay intact.
Architecture
shadow_web/
├── compressor.py # DOM strip + Action Map + semantic groups
├── content_index.py # Rendered Text Index — outline + on-demand fetch
├── capture_ready.py # Consent dismiss + readiness wait + scroll-until-content
├── dom_capture.py # Shadow DOM / iframe flatten (in-browser, read-only)
├── grouping.py # Semantic groups (forms, nav, modals)
├── schema_snap.py # Tables, forms, lists → JSON/CSV (facade over the `schemasnap` package)
├── form_fill.py # AgentOps form fill (auto_fill / ask / handoff)
├── security_scan.py # Attack surface rules (forms, links, headers, cookies)
├── heal_local.py # Local selector heal + ~/.shadow-web/heal_cache.json
├── query.py # shadow_grep (type:, intent:, label~, AND)
├── webmcp.py # WebMCP bridge (Chrome 145+)
├── diff.py # Page diff (skeleton + delta XML)
├── a11y_capture.py # CDP Accessibility dual capture + page_class
├── verified_heal.py # Playwright selector verification
├── wrapper.py # ShadowPage (sync Playwright)
├── browser_use.py # AsyncShadowPage (browser-use / async Playwright)
├── mcp/server.py # Cursor / Claude MCP tools (27)
└── server.py # FastAPI (/health, /v1/compress, /v1/heal)
scripts/
├── security_surface_scan.py # CLI: crawl + JSON/Markdown security reports
├── form_fill_demo.py # AgentOps form fill demo (plan + execute + wizard)
├── localpdf_competitor_scan.py # Multi-site competitor snapshot
├── smoke_install.sh # Install + pytest + one live navigate
└── cursor-setup.sh # MCP one-command setup
deploy/
└── oci.sh # Oracle Cloud deployment script
Examples
| Path | What it demonstrates |
|---|---|
| examples/golden_path/ | Full agent loop + token budget |
| examples/security_scan/ | Attack surface scan reports (JSON + Markdown) |
| examples/localpdf/ | Competitor intel playbook + sample scan |
| examples/form_fill/ | AgentOps form fill playbook + profile sample |
| examples/browser_use/ | browser-use integration |
Benchmarks
| Page | Raw HTML (tokens) | Grouped XML (tokens) | Actions | Reduction |
|---|---|---|---|---|
| Hacker News | 8,637 | 6,704 | 227 | −22% (1.3×) |
| Wikipedia (Web Scraping) | 99,343 | 16,462 | 501 | −83% (6.0×) |
| GitHub Trending | 167,875 | 37,833 | 1,290 | −77% (4.4×) |
Full compression benchmark: python benchmarks/run.py
Read this before quoting the numbers. The table measures one thing: raw page
HTML → grouped XML Action Map, on three pages, with tiktoken. It is not the
token cost of an agent run. The project's own end-to-end agent benchmark
(below) measures 1.6× total across 10 tasks, with overhead on 4 of 10
small pages, because the agent's system prompt and tool loop cost more than
dumping a small page. The win is real on pages where naive HTML is ≳5–10k
tokens, and that is the honest claim.
The Rendered Text Index (content_outline(max_tokens=…)) is a retrieval
budget, not a compression ratio: it returns at most the requested number of
tokens, so a "99% smaller" framing of it would be true of any budget and
therefore says nothing. Use it when you cannot afford the whole page, and read
coverage/mode/signals in its response to see what was left out.
Agent Task Benchmark (end-to-end, v0.4.1)
Measured 2026-08-01 with deepseek-chat on 10 bounded public-page tasks.
Fair baseline: Playwright Chromium HTML → one LLM call (same model/goal, HTML capped at 24k tokens).
Shadow Web: shadow-web exec agent loop.
Success uses shared answer checks (expected_contains / expected_any / regex) — not “any non-empty string”.
DEEPSEEK_API_KEY=sk-... python benchmarks/agent_tasks.py
# python benchmarks/agent_tasks.py --only mdn-404 --limit 3
| Task | Page type | Baseline LLM tok | SW tok | vs baseline | Steps | Success (BL/SW) |
|---|---|---|---|---|---|---|
| Page heading | static | 225 | 5,652 | overhead² | 3 | ✓ / ✓ |
| Wikipedia fact | text-heavy | 26,466 | 12,456 | 2.1× | 5 | ✓ / ✓ |
| HN top story | table | 12,000 | 9,671 | 1.2× | 4 | ✓ / ✓ |
| GitHub trending | SPA | 25,622 | 7,176 | 3.6× | 3 | ✓ / ✓ |
| Form fields | form | 24,848 | 6,756 | 3.7× | 3 | ✓ / ✓ |
| Books first price | catalog | 10,340 | 6,273 | 1.6× | 3 | ✓ / ✓ |
| Quotes count | list | 2,965 | 9,042 | overhead² | 4 | ✓ / ✓ |
| Login fields | form | 750 | 3,678 | overhead² | 2 | ✓ / ✓ |
| MDN HTTP 404 | text-heavy | 25,350 | 13,644 | 1.9× | 5 | ✓ / ✓ |
| Table Due (Smith) | table | 1,899 | 6,233 | overhead² | 3 | ✓ / ✓ |
| Total | — | 130,465 | 80,581 | 1.6× (6/10 wins) | — | 10/10 |
² Tiny / already-small pages: agent system prompt + tool loop costs more than dumping HTML.
Advantage shows on pages where naive HTML ≥ ~5–10k tokens.
Raw JSON: benchmarks/results/agent_tasks_live_20260801_094423_corrected.json.
MCP for Cursor / Claude
One-command setup:
bash scripts/cursor-setup.sh
Or manually:
{
"mcpServers": {
"shadow-web": {
"command": "shadow-web-mcp"
}
}
}
All 27 tools
| Category | Tool | What it does |
|---|---|---|
| Browse | navigate |
Open URL → snapshot (detail: minimal / terse / xml / full). http/https on the public web only |
snapshot |
Refresh page; diff=true for delta only |
|
click, fill |
Interact by data-sid |
|
| Agent | agent_run |
Thick observe→act→verify loop for one bounded task; always returns an envelope (stop_reason, complete, truncated, next_hint) |
| AgentOps | form_fill_plan |
Profile JSON → fill plan (auto_fill / ask / handoff) |
form_fill_execute |
Run plan (+ answers); validation_error feedback | |
| Filter (control plane) | shadow_query |
grep-style filter on live session |
query_page |
Alias for shadow_query (json output) | |
shadow_grep_html |
Filter raw/clean HTML without browser | |
| Compress (offline) | compress_html |
Strip + Action Map + groups |
compress_html_to_xml |
Grouped XML from HTML | |
| Data (SchemaSnap) | schema_table |
Table columns + types + rows |
schema_form |
Form fields + validation | |
schema_list |
Lists + standalone selects | |
schema_page |
All structured data at once | |
schema_json |
Table → JSON records | |
schema_csv |
Table → CSV string | |
schema_session |
Structured data from browser session | |
schema_session_json |
JSON records from session | |
schema_session_csv |
CSV from session | |
get_page_html |
Clean HTML (max_chars default 50000) |
|
| Content | content_outline |
Ranked rendered-text index (cards / feeds + coverage) |
content_blocks |
Fetch selected text blocks by compact IDs | |
| Search | web_search |
Brave Search (no API keys) |
| WebMCP | webmcp_list_tools |
Chrome 145+ page tools |
webmcp_execute_tool |
Execute WebMCP tool by name |
shadow-web MCP questions
Does it support Shadow DOM?
Yes, it flattens the Shadow DOM so that interactive elements are readable by AI agents.
Can it extract structured data from tables and forms?
Yes, using the SchemaSnap component, it turns tables, forms, and lists into clean JSON.