Lets an AI client fetch web pages with stealth HTTP or real browsers, bypass Cloudflare, and extract data using CSS selectors.
Also known as D4Vinci/Scrapling, scrapling
- Stars
- 103,113
- Tools
- —
- Downloads, 30 days
- 1,414,194
- Latest version
- 0.5.1
- Last commit
- Nov 14, 2026
- License
- BSD-3-Clause
What is Scrapling MCP?
Scrapling lets an AI client fetch web pages with stealth HTTP or real browsers, bypass Cloudflare, and extract data using CSS selectors.
Summary generated from the project's README
About Scrapling MCP server
Extracting data from websites through an AI agent is possible with Scrapling. This MCP server allows AI clients to browse the web, handle JavaScript rendering, and bypass protections like Cloudflare Turnstile or Interstitials. It provides a way for developers to turn web content into clean Markdown or specific data structures directly from their chat interface.
Users can choose between fast HTTP requests for static pages or stealth browsers for more complex interactions. The server supports various selection methods including CSS selectors, XPath, and regex search to pinpoint specific information. It is designed for those who need to collect data from websites that use heavy bot protection or require real browser behavior to load content.
Integration is handled through the Model Context Protocol, enabling an AI agent to act as a web crawler. The toolset includes features for persistent sessions, background API capture, and adaptive element tracking to ensure data is retrieved accurately even from dynamic sites.
What you can do with it
- 1Extracting structured website dataAn AI client can use CSS selectors to pull specific fields from a webpage.
- 2Bypassing advanced bot protectionsThe server can navigate through sites protected by Cloudflare.
- 3Converting web pages to MarkdownDevelopers can instruct an AI to transform an entire website into Markdown format.
- 4Automating complex web crawlingYou can task an AI with running concurrent spiders to collect large datasets.
Written from the project's README. Every use case is checked against a quote from it.
How to install Scrapling MCP
Formats verified 10 Nov 2026Add to CursorAdd to GooseInstall Scrapling MCP in Claude Code
$ claude mcp add scrapling -- uvx scraplingInstall Scrapling MCP in Cursor
~/.cursor/mcp.json
{
"mcpServers": {
"scrapling": {
"command": "uvx",
"args": [
"scrapling"
]
}
}
}Install Scrapling MCP in VS Code
.vscode/mcp.json
{
"servers": {
"scrapling": {
"type": "stdio",
"command": "uvx",
"args": [
"scrapling"
]
}
}
}Install Scrapling MCP in VS Code CLI
$ code --add-mcp '{"name":"scrapling","type":"stdio","command":"uvx","args":["scrapling"]}'Install Scrapling MCP in Claude Desktop
claude_desktop_config.json
{
"mcpServers": {
"scrapling": {
"command": "uvx",
"args": [
"scrapling"
]
}
}
}Install Scrapling MCP in Codex
$ codex mcp add scrapling -- uvx scraplingInstall Scrapling MCP in Codex config.toml
~/.codex/config.toml
[mcp_servers.scrapling] command = "uvx" args = ["scrapling"]
Install Scrapling MCP in Gemini CLI
$ gemini mcp add -s user scrapling uvx -- scraplingInstall Scrapling MCP in Gemini settings.json
~/.gemini/settings.json
{
"mcpServers": {
"scrapling": {
"command": "uvx",
"args": [
"scrapling"
]
}
}
}Automated checks
Ran Nov 11, 2026Full report →Passed our automated checks. That is not a security audit: read the code and give the server the narrowest access it needs. How checks work
- Source repository linked and reachablePass
- Open source license (OSI)Pass
- Commit or release in the last yearPass
- Package name not close to a popular packagePass
- Remote endpoint uses HTTPSNot checked
- Destructive tools are labelledNot checked
- No hidden instructions to the modelPass
- No critical advisory in the package or its dependenciesPass
README
View on GitHubScrapling is a Python Web Scraping framework that brings fast HTTP requests, stealth browsers, powerful parsing, and concurrent crawling into one library.
Scrape static pages, render JavaScript, interact with websites, and extract data from the API calls behind them. Build crawlers with reusable sessions, proxy rotation, automatic retries, pause/resume, and built-in data exports.
Use it from Python, the terminal, or your AI agent through its MCP server and Agent Skill. From extracting a few fields to collecting a whole website as clean Markdown, Scrapling covers the full scraping workflow.
What you can do with Scrapling
- Fetch & interact: Fast HTTP requests, stealth browsers, JavaScript rendering, Cloudflare Turnstile/Interstitial handling, persistent sessions, and background API capture.
- Parse & extract: CSS, XPath, text and regex search, adaptive element tracking, similar-element matching, and clean Markdown output.
- Crawl & export: Concurrent spiders, proxy rotation, retries, AutoThrottle, pause/resume, streaming, ready-made templates, and JSON/JSONL/CSV/XML exports.
- Scrape with AI or from your terminal: An MCP server for AI agents to browse and extract data, an Agent Skill, CLI commands, and an interactive shell.
Start with a single page:
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()
print(quotes)
Or scale up with a spider:
from scrapling.spiders import Spider, Response
class QuotesSpider(Spider):
name = "quotes"
start_urls = ["https://quotes.toscrape.com/"]
async def parse(self, response: Response):
for quote in response.css('.quote'):
yield {
"text": quote.css('.text::text').get(),
"author": quote.css('.author::text').get(),
}
result = QuotesSpider().start()
print(len(result.items))
result.items.to_json("quotes.json")
This prints 10 and saves the quotes and authors to quotes.json.
Platinum Sponsors
Do you want to show your ad here? Click here
Sponsors
Do you want to show your ad here? Click here and choose the tier that suits you!
Key Features
Spiders - A Full Crawling Framework
- 🕷️ Scrapy-like Spider API: Define spiders with
start_urls, asyncparsecallbacks, andRequest/Responseobjects. - ⚡ Concurrent Crawling: Configurable concurrency limits, per-domain throttling, and download delays.
- 🔄 Multi-Session Support: Unified interface for HTTP requests, and stealthy headless browsers in a single spider - route requests to different sessions by ID.
- 💾 Pause & Resume: Save pending requests and seen URLs with
crawldir. Press Ctrl+C to stop gracefully and restart with the same directory. - 📡 Streaming Mode: Stream scraped items as they arrive via
async for item in spider.stream()with real-time stats - ideal for UI, pipelines, and long-running crawls. - 🛡️ Blocked Request Detection: Automatic detection and retry of blocked requests with customizable logic.
- 🚦 AutoThrottle: Stop guessing delays. The spider tunes the delay of each domain on its own from how fast the website responds, then doubles it (or waits what
Retry-Afterasks) whenever the website starts blocking or rate-limiting you, and speeds back up once it stops. - 🤖 Robots.txt Compliance: Optional
robots_txt_obeyflag that respectsDisallow,Crawl-delay, andRequest-ratedirectives with per-domain caching. - 🧪 Development Mode: Cache responses to disk on the first run and replay them on subsequent runs - iterate on your
parse()logic without re-hitting the target servers. - 🧩 Ready-made Spider Templates: Skip the boilerplate with
CrawlSpiderfor rule-based link following,SitemapSpiderfor sitemap/robots.txt-driven crawls,XMLFeedSpider/CSVFeedSpiderfor iterating XML/RSS and CSV feeds, andShopifySpiderto collect products from public Shopify stores, one item per variant. - 🔗 Link Extraction: A standalone
LinkExtractorprimitive with allow/deny patterns, domain filters, CSS/XPath scoping, extension filtering, and canonicalization - use it inside the templates or on its own. - 📦 Built-in Export: Export results through hooks and your own pipeline or the built-in JSON/JSONL/CSV/XML exporters with
result.items.to_json(),to_jsonl(),to_csv(), andto_xml().
Advanced Websites Fetching with Session Support
- HTTP Requests: Fast and stealthy HTTP requests with the
Fetcherclass. Can impersonate browsers' TLS fingerprint, headers, and use HTTP/3. - Dynamic Loading: Fetch dynamic websites with full browser automation through the
StealthyFetcherclass supporting Playwright's Chromium and Google's Chrome. - Anti-bot Bypass: Advanced stealth capabilities with
StealthyFetcherand fingerprint spoofing. Can easily bypass all types of Cloudflare's Turnstile/Interstitial with automation. - Session Management: Persistent session support with
FetcherSessionandStealthySessionclasses for cookie and state management across requests. - Proxy Rotation: Built-in
ProxyRotatorwith cyclic or custom rotation strategies across all session types, plus per-request proxy overrides. - Domain & Ad Blocking: Block requests to specific domains (and their subdomains) or enable built-in ad blocking (~3,500 known ad/tracker domains) in browser-based fetchers.
- DNS Leak Prevention: Optional DNS-over-HTTPS support to route DNS queries through Cloudflare's DoH, preventing DNS leaks when using proxies.
- Remote Browsers: Instead of launching a browser locally, connect to one that's already running through CDP with
cdp_url, whether it's on the same machine, another host, or a managed browser provider. You can also point any browser fetcher at your own Chromium build withexecutable_path. - Background API Capture: Enable
record_requests=Trueon a browser session, then search its saved responses withsession.network.search(limit=None). Read API data and HTTP errors across page loads without sending requests again. History is bounded and saves text response bodies. - Async Support: Complete async support across all fetchers and dedicated async session classes.
Parsing and Data Extraction
- 🔄 Smart Element Tracking: Relocate elements after website changes using intelligent similarity algorithms.
- 🎯 Smart Flexible Selection: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more.
- 🔍 Find Similar Elements: Automatically locate elements similar to found elements.
Scrapling MCP questions
How do I add Scrapling MCP to Claude Code?
Run claude mcp add scrapling -- uvx scrapling in your terminal. Add --scope user to make it available in every project.
How do I add Scrapling MCP to Cursor?
Open ~/.cursor/mcp.json and add the server under mcpServers, using the Cursor config shown in the install section above.
Does Scrapling support Cloudflare bypass?
Yes, Scrapling handles Cloudflare Turnstile and Interstitial challenges through its stealth browsers.
What selectors can I use to extract data?
You can use CSS and XPath selectors, plus text and regex search, and get clean Markdown output.
Can an AI agent use Scrapling through MCP?
Yes, Scrapling ships an MCP server that lets AI agents browse and extract web data.
What export formats does Scrapling support?
Crawled results can be exported to JSON, JSONL, CSV, or XML.