Lets an AI client fetch, crawl and search web pages locally, converting them to Markdown and reporting anti-bot walls.
Also known as ilien-dev/svipall
- Stars
- 21
- Tools
- 21
- Downloads, 30 days
- —
- Latest version
- 2.1.0
- Last commit
- Nov 13, 2026
- License
- AGPL-3.0
What is Svipall MCP?
Svipall lets an AI client fetch, crawl and search web pages locally, converting them to Markdown and reporting anti-bot walls. It exposes 21 tools, such as web_fetch, web_fetch_many, web_search.
Summary generated from the project's README
About Svipall MCP server
Extracting clean web content for AI agents is what Svipall enables by running entirely on your local machine. It allows developers to fetch, crawl, and search web pages without needing third-party API keys, paid captcha solvers, or external accounts. The server converts HTML into clean Markdown, helping to prevent agents from wasting context on raw markup or boilerplate text.
Users can interact with the server through an MCP client or a command-line interface. It features specialized tools for different tasks, such as using a schema auto-detection mode to turn web listings into typed rows or using the web_video tool to extract captions and chapters. Because it runs locally, all processing and storage stay on your hardware, and the system provides explicit reports when it encounters anti-bot walls or different vendor challenges.
When a site presents a challenge that requires human intervention, the server provides a local dashboard to resolve it. This ensures that even when automated attempts fail, the agent can still access the necessary data through a managed fallback.
What you can do with it
- 1Convert web listings to structured dataTransform a webpage containing multiple similar items into a structured format with named columns and typed rows.
- 2Extract video informationRetrieve captions, chapters, and visual contact sheets from video pages.
- 3Crawl websites for large datasetsNavigate through multiple pages of a site and save the results directly to a CSV file to avoid context limits.
- 4Identify website access blocksDetermine exactly why a web request failed by identifying the specific type of anti-bot wall encountered.
Written from the project's README. Every use case is checked against a quote from it.
How to install Svipall MCP
Formats verified 10 Nov 2026Add to CursorAdd to GooseInstall Svipall MCP in Claude Code
$ claude mcp add svipall -- npx -y svipallInstall Svipall MCP in Cursor
~/.cursor/mcp.json
{
"mcpServers": {
"svipall": {
"command": "npx",
"args": [
"-y",
"svipall"
]
}
}
}Install Svipall MCP in VS Code
.vscode/mcp.json
{
"servers": {
"svipall": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"svipall"
]
}
}
}Install Svipall MCP in VS Code CLI
$ code --add-mcp '{"name":"svipall","type":"stdio","command":"npx","args":["-y","svipall"]}'Install Svipall MCP in Claude Desktop
claude_desktop_config.json
{
"mcpServers": {
"svipall": {
"command": "npx",
"args": [
"-y",
"svipall"
]
}
}
}Install Svipall MCP in Codex
$ codex mcp add svipall -- npx -y svipallInstall Svipall MCP in Codex config.toml
~/.codex/config.toml
[mcp_servers.svipall] command = "npx" args = ["-y", "svipall"]
Install Svipall MCP in Gemini CLI
$ gemini mcp add -s user svipall npx -- -y svipallInstall Svipall MCP in Gemini settings.json
~/.gemini/settings.json
{
"mcpServers": {
"svipall": {
"command": "npx",
"args": [
"-y",
"svipall"
]
}
}
}Svipall MCP tools 21
Detected from the READMEweb_fetchweb_fetch_manyweb_searchweb_site_searchweb_crawlweb_mapweb_snapshotweb_actweb_captureweb_videoweb_screenshotweb_diff
Show all 21 toolsShow fewer tools
web_watchweb_notesweb_logweb_loginweb_routeweb_profileweb_statusbrowser_setupsolve_and_continue
Automated checks
Ran Nov 13, 2026Full report →Passed our automated checks. That is not a security audit: read the code and give the server the narrowest access it needs. How checks work
- Source repository linked and reachablePass
- Open source license (OSI)Pass
- Commit or release in the last yearPass
- Package name not close to a popular packagePass
- Remote endpoint uses HTTPSNot checked
- Destructive tools are labelledPass
- No hidden instructions to the modelPass
- No critical advisory in the package or its dependenciesPass
README
View on GitHubEverything runs locally. There are no third-party API keys, no paid captcha services and no telemetry. Web requests reach the sites you visit, and results reach the agent or client you connect.
Svipall fetches and renders pages locally, extracts their content, and reports detected challenges and extraction-quality labels. Some sites still block it, and an apparently successful response can contain incomplete records or a page shell. The published comparisons therefore audit useful content separately from HTTP status and the tool's own verdict.
Why Svipall
| What goes wrong | Svipall |
|---|---|
Your agent reads a "checking your browser" screen and summarises it as the article. It was a 200, so nothing flagged it |
Twelve wall kinds, each naming the move it implies. Detected blocks carry an explicit verdict; classification is heuristic details |
| You crawl 5,000 pages and can't tell which are worth keeping | Assessed pages carry quality and duplicate observations, and quality labels never discard a page details |
| One page = 300,000 tokens of raw HTML. The fixes are four manual jobs you now own | Clean Markdown by default; opt into tables as rows, out_file to disk, or capture of the site's own JSON API details |
| You want to attempt a supported captcha without a paid solver | Fifteen widget families and eleven answer modalities, all local, optional vision models depending on the build, and a human dashboard for unresolved challenges details |
It records successful visits, failures, incomplete extraction and rejected changes. Historical benchmark logs and the current comparison use different scoring rules and configurations; the results section distinguishes them.
Rust · MCP + CLI + REST · no Node, no Python, no API key · local storage and processing
The comparison table describes other projects' documented scope.
Install
Every channel below installs the latest release,
built from the tree this README describes. svipall doctor reports what an installed build can do
on the machine it is on.
The README factual audit records the source checks, documentation corrections and limits of this review.
Choose agent-assisted setup, the Claude Code plugin, or a manual installation.
Ask the agent you already have
Paste this into Claude Code, Cursor, Codex, opencode, or anything else that can run a command:
Install and configure Svipall by following the instructions here:
https://raw.githubusercontent.com/ilien-dev/svipall/main/docs/install.md
That page guides an agent through platform detection and verification, then asks which integration you want: CLI + Skill (recommended, lower context use) or MCP + Skill (the full MCP tool surface). It also asks whether to configure every project or only the current one, and verifies the chosen result instead of treating a skill as MCP registration. When Svipall already exists, it shows the installed and latest versions and asks whether to update the shared user installation or keep the current version before it writes the integration. Completion depends on the agent, client configuration and available permissions.
Claude Code: install the plugin
/plugin marketplace add ilien-dev/svipall
/plugin install svipall@svipall
/svipall:setup
/svipall:setup installs the binary if it is missing, checks the server answers, and offers to make
Svipall the way Claude reaches the web in every project. It asks before each of those.
/svipall:doctor reports the installation's capabilities. /svipall:update shows the installed
and latest releases, then updates the binaries only if you choose to. /svipall:uninstall offers
removal of setup's registration, memory and strict-mode changes. Removing the binary and the data
are separate choices.
Install it yourself
One line, with no toolchain and nothing to compile:
curl -fsSL https://raw.githubusercontent.com/ilien-dev/svipall/main/install.sh | sh # macOS, Linux
irm https://raw.githubusercontent.com/ilien-dev/svipall/main/install.ps1 | iex # Windows
Or a package manager, or the container image:
brew install ilien-dev/svipall/svipall # macOS, Linux
scoop bucket add svipall https://github.com/ilien-dev/scoop-svipall && scoop install svipall
docker pull ghcr.io/ilien-dev/svipall:latest # or :slim, the http tier only; see container notes
npx --yes svipall doctor # if node is already there
cargo install svipall # from source; add `svipall models install` for the models
The installers verify checksums where they can, and a mismatch stops the install. A missing checksum file, entry or hashing utility only warns, so a successful exit is not by itself proof the archive was verified: read the output. If the destination already contains another version, the installer states that the binaries are shared by every harness and asks before replacing them.
When a newer release exists, Svipall says so once per session. One command updates both binaries with the channel that installed them and refreshes the skills you copied by hand:
svipall update # --check only compares versions
Then close every session that uses Svipall and open it again. An agent can do the same from docs/update.md.
Platform builds, what ships where, building from source and wiring it into any MCP client are all in docs/install.md. If you have never installed anything from a terminal, GET-STARTED.md is this section with nothing assumed.
Then ask for something
There is no key to paste and no account to create.
"Read this page and summarise the pricing." "Crawl these docs and write me an
llms.txt." "Watch this listing and tell me when the price moves." "Get me every row of that table as CSV."
The assistant can choose among the exposed tools. Supported challenges that need a pair of eyes
go to a human dashboard at http://localhost:8787/human.
Or drive it from a shell
svipall fetch https://example.com/article
svipall fetch https://shop.example/item --query "shipping costs"
svipall fetch https://docs.example/api --schema auto # rows from a listing you've never seen
svipall crawl https://docs.example/ --pages 50 --out pages.csv
svipall search "rust async runtime" --engine all
svipall snapshot https://news.ycombinator.com # the page as roles and refs, not markup
svipall serve --port 8788 # the same server as a local REST API
Completed data commands print one JSON object to stdout and diagnostics to stderr, so the
output can be piped to jq. serve is a long-running server, and help is written to stderr.
What comes back
A historical run of svipall fetch https://example.com, with the content string cut short.
Current automatic fetches also report identity_used, and the fallback fields described below when
they have something to report:
{
"attempts": ["http: 200 (170ms) OK"],
"chars": 167,
"content": "# Example Domain\n\nThis domain is for use in documentation examples…",
"final_url": "https://example.com/",
"optimization": "ordinary",
"quality": "thin",
"quality_reasons": ["thin_text"],
"status": 200,
"tier_used": "http",
"title": "Example Domain",
"tokens_estimated": 42,
"url": "https://example.com"
}
tier_used says how hard it had to try and quality says what actually arrived. When a page does
not arrive, the same object carries blocked_reason, wall_kind, wall_vendor, wall_evidence
and a note telling your agent what to do next. Straight from a committed benchmark record:
{ "wall_kind": "vendor", "wall_vendor": "kpsdk.io", "wall_evidence": "header x-kpsdk-ct" }
A detected block carries a verdict alongside the returned content. A clear verdict still needs a content check, because the classifier cannot show that the requested records arrived intact.
What you can actually do with it
| You want to… | It looks like this |
|---|---|
| Read one page cleanly | web_fetch → Markdown with heuristic boilerplate removal and sanitization; query= ranks text by lexical relevance |
| Turn a listing into rows | schema: "auto" reads the page's own repeated structure, names the columns and hands back typed rows, with no model and no API call, in one parse |
| Pull a data table | tables=true → typed rows; out_file: rows.csv writes them to disk so thousands of rows never touch your context |
| Read what a video says | web_video → captions and chapters as one [m:ss] timeline, from a watch page, a page that embeds a player, <video> tracks or the stream manifest |
| Skip the scraping entirely | web_capture returns the JSON the page fetched while loading, usually the site's real API, with ?page=2 waiting for you |
| Turn a docs site into a corpus | web_crawl with llms.txt output, near-duplicate labels, resumable frontier and lexical saturation stopping, subject to page/token/traffic limits |
| Search without a key | web_search scrapes DuckDuckGo, Bing and Brave; engine="all" merges them by agreement |
| Let the agent click things | web_snapshot (roles + refs, a fraction of the tokens) then web_act to click, type, scroll and wait, all through human-like input |
| Attempt a browser challenge | Automatic routing can escalate to a patient browser tier; unresolved challenges and detected blocks are reported, but the remote cause is not always identifiable |
| Attempt a captcha locally | Fifteen widget families and eleven answer modalities, model support where available and a phone-friendly human dashboard. No paid solver; local budgets and remote restrictions still apply |
| Log in once and stay in | web_login opens a real window; you sign in; the cookies are kept in a profile you can export |
| Watch a page | web_watch checks the whole page or one CSS region while the server runs; saved selector fingerprints can help recover some redesigns |
| Read PDFs and Office files | docx, xlsx, pptx, odt, epub, rtf, csv and pdf come back as Markdown, from the web or from file:// |
| Drive it from any language | svipall serve → 20 local REST routes, one per tool, behind a bearer key it generates for you |
Who it is for
| You are… | Svipall gives you… |
|---|---|
| A Claude Code, Claude Desktop or Cursor user | One line of setup and tools your assistant picks by itself. Research, documentation, price comparison, monitoring |
| A developer building AI agents | A local web layer with structured output, token budgets, file export and resumable crawls; live web outcomes remain variable |
| A RAG or dataset builder | Bounded site crawls to Markdown, near-duplicate labels, llms.txt, and quality observations for assessed pages |
| A data or research person | Pages that sit behind "checking your browser" walls, and an honest answer when your address cannot open one |
| A privacy-conscious operator | No scraping API, no captcha farm, no geolocation lookup, no update check, no telemetry. Additional downloads are the managed browser when needed (automatic provisioning can be disabled) and the blocklists you enabled |
| A security or QA engineer testing your own site | A reproducible benchmark whose raw run logs are committed in this repository, and a request log that names which tier answered and which wall appeared |
Svipall runs on your own hardware. If you want a URL you can curl from a serverless function,
use a hosted scraping service instead. If you want the web inside your own agent, Svipall does that
processing locally; browser traffic and optional downloads are described under
Privacy and safety.
How it works, in plain words
The steps below describe the emulated tiers. The current automatic policy can promote a supported emulated route and append one eligible native fallback. Wall verdicts can end the attempt. Content-quality labels alone do not trigger escalation.
- Ask for the page through HTTP first on a new route. The default engine emulates selected
Chrome network characteristics. In the historical
public31runs above, 59 of 93 cells stopped at this tier, and 44 of those scoredokunder that benchmark's rule. A route that stops here has still not necessarily delivered the page. - If the page needs JavaScript, open a browser. Headless Chromium runs the scripts and hands back the rendered document.
- If the site checks for robots, wear a disguise. The stealth tier patches known browser surfaces to match the emulated identity. The offline probes check those surfaces; they cannot prove that an arbitrary detector will accept them.
- If the site wants a real person, act like one. The
realtier is a visible-but-offscreen browser with a persistent profile. It moves the pointer along curves and scrolls with a wheel. - If there is a challenge, answer it or wait it out. The
warmtier runs the captcha strategy loop during its bounded wait and avoids pointer activity on recognized self-verifying interstitials. This does not guarantee clearance. - Use native only as a last resort. When eligible and within the remaining budgets, try one native browser attempt. It exposes real device characteristics and reports a privacy notice.
- Remember what worked. Two supporting observations can promote a useful emulated route for later visits in the same context. Native stays last even when it succeeds.
- Report the observed failure. Where available, a blocked result includes
blocked_reason, the classified wall, recognized vendor/evidence and a suggested next step. Classification is heuristic. Transport errors and local budget deferrals may have less page evidence.
MCP tools
Twenty-nine tools, all local.
| Tool | What it does |
|---|---|
web_fetch |
Fetch a page as Markdown or structured JSON. mode=auto climbs the ladder. schema (self-healing), tables, scroll, query, max_tokens/cursor, cache, include_metadata, include_links, include_quality, use_site_template, robots, out_file, mobile, text_only, isolated, css_selector, profile, proxy, method/body/headers. URLs may be raw:<html> or file:// under local_roots |
web_fetch_many |
Bounded-parallel fetch of many URLs, with schema and tables as on web_fetch. Reports corroboration (how many distinct documents the set actually is), marks each duplicate with same_text_as, and moves the different ones up. It says reordered_for_diversity when it did, because a set that comes back in a different order without saying so will mislead the caller |
web_search |
DuckDuckGo / Bing / Brave without an API key; engine="all" merges by agreement |
web_site_search |
Discover a site's search form and learn its query-URL pattern when possible; later fetches still follow normal routing and policy |
web_crawl |
Same-domain crawl with robots.txt, dedup, boilerplate removal, strategy=dfs, scroll, schema/tables for rows, llms.txt, file export, a saturation stop, and a crawl_id to resume |
web_map |
A site's URLs without crawling it: robots.txt, sitemaps (nested indexes and .gz included), RSS/Atom feeds and homepage links, a few hundred tokens of structure instead of the thousands a crawl costs |
web_snapshot |
The page as roles, accessible names and short refs that web_act accepts. Deterministic, no vision model |
web_act |
click, type, fill, press, hover, select, scroll, wait, eval, goto, screenshot, hold, verify, console; supported pointer/keyboard/wheel actions use the behavior layer, while eval runs caller-supplied JavaScript |
web_capture |
Observe matching JSON/network responses during a bounded browser visit; API usability and completeness are not guaranteed |
web_video |
A video's captions (a person's first, else the platform's automatic ones, in lang when asked) and chapters as one timeline. Reads a known player's boot JSON, JSON-LD VideoObject, <track>, and HLS/DASH caption renditions; a player whose caption addresses only answer its own requests is read through a borrowed request in a live page. frames adds keyframes where the picture changes, found in the player's storyboard or by comparing sampled frames, and returned as PNG paths inside the timeline. Each one is read from the page's own <video> element, so the player's controls and captions are not in it; a file from another origin that forbids this gets a screenshot instead, and the notes say so. An advert that plays first is run through, muted, before any frame is taken. A video with no captions and a media file behind it is transcribed on this machine when the speech model is installed (svipall models install). Encrypted streams are reported as drm and never read, so they get no frames |
browser_open / browser_do / browser_close |
Persistent session with cookies and page state across calls |
web_screenshot |
PNG of the rendered page, full_page or mobile |
web_diff |
What changed on a page since Svipall last saw it |
web_watch |
Persist a watch and check it while the server runs; list or check to retrieve changes. Region recovery after a redesign is heuristic |
web_notes |
Key-value memory that outlives the session |
web_log |
Which tier answered, which wall appeared, how long it took, per domain |
web_login |
Visible window for a manual login or challenge; cookies saved to a profile |
web_route |
Per-domain proxy, or a pool of proxies with countries; subdomains inherit; exit_strategy sticky or round-robin; check=true tests the exits (liveness, latency, DNS leak) with no third-party service |
web_profile |
Export/import an encrypted browser profile between machines |
web_status |
Learned tiers, cooldowns, routes, per-exit health and latency, profiles, open browsers, solver stats, which models answer and from where, whether the host has a real GPU, h3_offered_by |
browser_setup |
Download or manage Chrome for Testing |
solve_and_continue |
Attempt a captcha on the blocked page and return the resulting content or unresolved state |
solve_image_captcha / solve_recaptcha_v2 / solve_turnstile / solve_hcaptcha / captcha_status / report_captcha |
Local captcha attempts; the dashboard also exposes in.php / res.php / createTask / getTaskResult compatibility endpoints for supported tasks. This is not full compatibility with every solver-client option |
The CLI
svipall fetch | crawl | snapshot | capture | search | map | log | notes | watch
profile | browser | route | status | serve | doctor | update | hook
config show | set | preset
solver export-corpus
quality ask | export-training | train
A test asserts the usage text names every command the binary answers to, and a second test keeps
skill/SKILL.md in step with both.
Svipall MCP questions
How do I add Svipall MCP to Claude Code?
Run claude mcp add svipall -- npx -y svipall in your terminal. Add --scope user to make it available in every project.
How do I add Svipall MCP to Cursor?
Open ~/.cursor/mcp.json and add the server under mcpServers, using the Cursor config shown in the install section above.
What tools does Svipall MCP provide?
It exposes 21 tools, including web_fetch, web_fetch_many, web_search, web_site_search, web_crawl.
Does Svipall need an API key or account?
No. It runs locally with no third-party API keys, paid captcha services or telemetry, and there is no account to create.
Can Svipall crawl a docs site and output llms.txt?
Yes, web_crawl supports llms.txt output along with near-duplicate labels, a resumable frontier and saturation stopping.
How does Svipall handle captchas?
It attempts supported challenges locally across fifteen widget families and eleven answer modalities, with a human dashboard for unresolved ones.
Can it search the web without an API key?
Yes, it searches without an API key and can query all engines at once from the CLI.
Does Svipall tell me when a page was blocked by an anti-bot wall?
Yes, detected blocks return fields like wall_kind, wall_vendor and wall_evidence with a note telling the agent what to do next.