Deepcrawl MCP server
Commit Nov 13, 2026Clone websites, scrape pages to markdown, and crawl entire sites for content analysis or RAG pipelines.
Also known as Soflutionltd/deepcrawl
- Stars
- 0
- Tools
- 2
- Downloads, 30 days
- —
- Latest version
- 1.1.1
- Last commit
- Nov 13, 2026
- License
- MIT
What is Deepcrawl MCP?
Deepcrawl is an MCP server. Clone websites, scrape pages to markdown, and crawl entire sites for content analysis or RAG pipelines. It exposes 2 tools, such as url, depth.
Summary generated from the project's README
About Deepcrawl MCP server
Deepcrawl provides a way to extract web content by cloning websites, scraping individual pages into markdown, or crawling entire domains. It is designed for developers building RAG pipelines, performing SEO audits, or conducting content analysis. The server runs locally via npx and can be integrated into clients like Cursor or Claude. It does not require any API keys or paid subscriptions to function.
The toolkit includes four specific commands: deepcrawl_scrape for extracting clean markdown from a single URL, deepcrawl_clone for downloading a full page with all its assets like CSS and images, deepcrawl_crawl for traversing a site up to 100 pages, and deepcrawl_map for discovering URLs through sitemaps. For websites built with heavy JavaScript frameworks like React or Next.js, the server can utilize Playwright for full rendering if installed on the local machine.
This tool distinguishes itself from alternatives by offering full site cloning with relative path rewriting and by operating without rate limits. It includes built-in features like user agent rotation and random delays to help manage basic bot detection during web scraping tasks.
What you can do with it
- 1Convert web pages to markdownExtract the main text content from a specific URL while stripping away navigation and footers.
- 2Download complete website assetsSave a webpage locally including all its CSS, JavaScript, and images for offline viewing.
- 3Prepare data for RAG pipelinesTraverse an entire documentation site to gather all pages as markdown files for an AI knowledge base.
- 4Map site architectureIdentify all available URLs on a domain by parsing the sitemap or crawling the homepage.
Written from the project's README. Every use case is checked against a quote from it.
Deepcrawl MCP tools 2
Detected from the READMEurldepth
Automated checks
Ran Nov 14, 2026Full report →Not verified yet. A server needs a score of 80, no serious finding, and enough data for us to check. How checks work · What to fix for more points
- Source repository linked and reachablesoflutionltd/deepcrawl is public and reachablegithub.com/soflutionltd/deepcrawl ↗Pass
- Package names this repository as its sourceNot checked
- Build provenance (Sigstore attestation)Not checked
- Open source license (OSI)MITspdx.org/licenses/MIT.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-11-13github.com/soflutionltd/deepcrawl ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNot checked
- No critical advisory in the package or its dependenciesNot checked
- No hidden instructions to the model2 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNo tool named like delete, drop or execPass
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubClone, scrape and crawl any website. Free Firecrawl alternative. No API keys, no rate limits, no subscription.
Quick Start
Cursor (one-click): Click the "Install in Cursor" button above.
Manual MCP config:
{
"mcpServers": {
"deepcrawl-mcp": {
"command": "npx",
"args": ["-y", "deepcrawl-mcp@latest"]
}
}
}
Automatic updates
Every install below keeps itself up to date, without restarting Claude or Cursor. The MCP client talks to a small supervisor (bin/supervise.cjs) that stays connected while the server runs behind it. At start-up and every 6 hours it asks npm for the latest deepcrawl-mcp, installs it in the background in ~/.soflution/mcp/deepcrawl-mcp/, and switches over as soon as no request is in flight (the client is told the tool list may have changed). A version that fails to start is rolled back at once and never picked again on that machine. npm checks the integrity of every download, and releases are published from GitHub Actions with provenance.
The only network call is that version check to registry.npmjs.org; nothing about your machine or your use is sent. Turn it off with NO_UPDATE_NOTIFIER=1 (or SOFLUTION_AUTO_UPDATE=0). Started directly with node dist/..., deepcrawl-mcp does not update itself and only mentions a newer version in its first tool result.
CLI:
npx deepcrawl-mcp@latest
Tools
deepcrawl_scrape
Scrape a single page and return clean markdown. Extracts title, description, links, images, and metadata. Strips navigation, footer, ads, and tracking.
"Scrape https://example.com and give me the main content"
| Parameter | Default | Description |
|---|---|---|
url |
required | Page URL to scrape |
mainContentOnly |
true |
Extract only main content (skip nav/footer) |
includeLinks |
true |
Include discovered links |
includeImages |
true |
Include image URLs |
deepcrawl_clone
Clone a full page with all assets: HTML, CSS, JS, images, fonts, favicons. Downloads everything into a local folder, rewrites URLs to relative paths. Open index.html in a browser and it works.
"Clone https://competitor.com into a local folder"
| Parameter | Default | Description |
|---|---|---|
url |
required | Page URL to clone |
outputDir |
~/deepcrawl-clones/<domain> |
Output folder |
depth |
0 |
Link depth: 0 = single page, 1+ = follow links |
deepcrawl_crawl
Crawl an entire site following internal links. Returns every page as clean markdown. Great for content analysis, SEO audit, or feeding a RAG pipeline.
"Crawl https://docs.example.com and return all pages as markdown"
| Parameter | Default | Description |
|---|---|---|
url |
required | Starting URL |
maxPages |
20 |
Max pages to crawl (max: 100) |
includeImages |
false |
Include image URLs per page |
deepcrawl_map
Discover all URLs from a site via sitemap.xml parsing and homepage link crawling. Run this before a crawl to see the site's scope.
"Map all pages on https://example.com"
| Parameter | Default | Description |
|---|---|---|
url |
required | Site URL |
maxUrls |
200 |
Max URLs to discover |
vs Firecrawl
| deepcrawl | Firecrawl | |
|---|---|---|
| Price | Free | $19+/mo |
| API key | None | Required |
| Rate limits | None | Yes |
| Scrape to markdown | Yes | Yes |
| Full site crawl | Yes | Yes |
| Site map | Yes | Yes |
| Clone with assets | Yes | No |
| JS rendering | Yes (via Playwright) | Yes |
| Anti-bot bypass | Partial (UA rotation, headers, delays) | Yes |
deepcrawl handles static sites out of the box. For JS-heavy SPAs (React, Next.js, SvelteKit), install Playwright for full rendering:
npm install -g playwright
npx playwright install chromium
Once installed, deepcrawl auto-detects Playwright and enables JS rendering. Use jsRender: true on any tool to activate it. All tools also include UA rotation, realistic browser headers, and random delays to avoid basic bot detection.
Also by Soflution
- brandcheck - Check brand name availability across 27 platforms
- depsonar - Dependency audit, security scan, license check
License
MIT
Deepcrawl MCP questions
What tools does Deepcrawl MCP provide?
It exposes 2 tools, including url, depth.
Does Deepcrawl require an API key?
No, Deepcrawl does not require any API keys to function.
Can Deepcrawl handle JavaScript-heavy websites?
Yes, it supports JS rendering via Playwright for SPAs like React or Next.js.
How does Deepcrawl compare to Firecrawl?
Deepcrawl is free, requires no API key, has no rate limits, and can clone websites with all assets including CSS and JS.
Can I clone a website with its images and styles?
Yes, the clone tool downloads HTML, CSS, JS, images, and fonts, rewriting URLs to relative paths so the site works locally.