Enables AI agents to scrape, crawl, and monitor websites with built-in anti-bot bypass and structured data extraction.
Also known as SanggonBoy/PyreCrawl
- Stars
- 25
- Tools
- 13
- Downloads, 30 days
- 334
- Latest version
- 0.8.1
- Last commit
- Sep 21, 2026
- License
- MIT
What is PyreCrawl MCP?
PyreCrawl enables AI agents to scrape, crawl, and monitor websites with built-in anti-bot bypass and structured data extraction. It exposes 13 tools, such as scrape, extract, map_site.
Summary generated from the project's README
About PyreCrawl MCP server
PyreCrawl provides web browsing capabilities for AI agents through the Model Context Protocol. It allows developers to integrate scraping, crawling, and monitoring directly into compatible clients like Claude Desktop, Cursor, and VS Code. The server is self-hosted and operates without requiring API keys or subscriptions, making it a private alternative to hosted services.
It uses an automated fallback system to handle different web environments. The process starts with fast HTTP requests and escalates to a stealth browser using Chromium to bypass Cloudflare challenges or render JavaScript when necessary. For complex tasks, it offers deep processing to produce LLM-ready markdown and structured data. Users can manage their data through built-in caching and persistent browser sessions to navigate login walls or multi-step workflows.
What you can do with it
- 1Perform deep academic researchAn AI agent can search for scholarly papers and compile an evidence pack with citations.
- 2Extract structured website dataYou can instruct an agent to pull specific product details from a page into a JSON format using a CSS schema.
- 3Monitor websites for changesA developer can set up a task to track a specific URL and receive notifications when content changes.
- 4Automate multi-page site crawlingAn agent can map out all internal links of a website and crawl through them to gather information.
Written from the project's README. Every use case is checked against a quote from it.
How to install PyreCrawl MCP
Formats verified 10 Nov 2026Add to CursorAdd to GooseInstall PyreCrawl MCP in Claude Code
$ claude mcp add pyrecrawl -- uvx pyrecrawl serveInstall PyreCrawl MCP in Cursor
~/.cursor/mcp.json
{
"mcpServers": {
"pyrecrawl": {
"command": "uvx",
"args": [
"pyrecrawl",
"serve"
]
}
}
}Install PyreCrawl MCP in VS Code
.vscode/mcp.json
{
"servers": {
"pyrecrawl": {
"type": "stdio",
"command": "uvx",
"args": [
"pyrecrawl",
"serve"
]
}
}
}Install PyreCrawl MCP in VS Code CLI
$ code --add-mcp '{"name":"pyrecrawl","type":"stdio","command":"uvx","args":["pyrecrawl","serve"]}'Install PyreCrawl MCP in Claude Desktop
claude_desktop_config.json
{
"mcpServers": {
"pyrecrawl": {
"command": "uvx",
"args": [
"pyrecrawl",
"serve"
]
}
}
}Install PyreCrawl MCP in Codex
$ codex mcp add pyrecrawl -- uvx pyrecrawl serveInstall PyreCrawl MCP in Codex config.toml
~/.codex/config.toml
[mcp_servers.pyrecrawl] command = "uvx" args = ["pyrecrawl", "serve"]
Install PyreCrawl MCP in Gemini CLI
$ gemini mcp add -s user pyrecrawl uvx -- pyrecrawl serveInstall PyreCrawl MCP in Gemini settings.json
~/.gemini/settings.json
{
"mcpServers": {
"pyrecrawl": {
"command": "uvx",
"args": [
"pyrecrawl",
"serve"
]
}
}
}Started by HostedMCP on 16 Nov 2026
We ran this command for pyrecrawl 0.8.1 in an isolated container with no network and no keys. It answered and listed 13 tools.
PyreCrawl MCP tools 13
Read from the server with tools/listscrapeScrape a single URL → LLM-ready markdown. Use this when the user shares a URL and wants its content (read, analyze, summarize, extract). Auto-escalates through fast→stealth→llm when blocked. Args: url: Target URL (http/https). prefer: "auto" | "fast" | "stealth" | "llm". auto = fast first, escalate to stealth on block/short page. fast = cheap HTTP only (no JS). stealth = real Chromium + Cloudflare solver. llm = full Crawl4AI browser + BM25 fit-markdown. timeout: per-attempt timeout in seconds. include_html: include raw HTML in the response (large; off by default). js: (stealth only) JS expression evaluated against the live page after it settles. The value comes back in ``meta.js_result``. Use for data that lives in DOM *properties* (e.g. an input's ``.value``) rather than in serialized HTML. wait_for: (stealth only) JS predicate expression polled until truthy (bounded by ``timeout``). Use to wait for content that arrives asynchronously after ``network_idle``. Returns: {url, final_url, status, markdown, title, method, elapsed_ms, meta} or {error, url, method} on failure.
extractScrape + structured extraction using a CSS-based JSON schema. Use when the user wants structured data (tables, lists, product info) extracted from a page. Define a CSS schema to target specific elements. The schema is a JsonCssExtractionStrategy schema: { "name": "PageItems", "baseSelector": "div.item", "fields": [{"name": "title", "selector": "h2", "type": "text"}, ...] } Returns parsed JSON in `data`.
map_siteEnumerate all internal URLs reachable from `root`. Use when the user wants to map a site's structure or find all pages before crawling. Often paired with `crawl` or `batch_scrape`. Args: root: Website root (e.g. "https://example.com/docs"). include_pattern: Optional regex; only URLs matching are returned. limit: Hard cap on returned URLs.
crawlMulti-page crawl: discover URLs on `root`, then scrape each. Use this when the user wants to crawl an entire site section or docs, or needs multiple pages scraped in bulk. For single pages use `scrape`; for research questions use `deep_research`. Args: root: start URL. max_pages: hard cap on pages scraped. css_selector: scope each page's html/markdown to the matched element (non-llm: lxml re-scope of the fetched HTML; llm: native crawl4ai css_selector). prefer: "auto" | "fast" | "stealth" | "llm" (llm = Crawl4AI BFS deep-crawl). include_paths: regex — keep only URLs matching (matched against full URL). exclude_paths: regex — drop URLs matching (e.g. `/tag/|/page/\d+`). max_depth: 0 = flat harvest from the root page's links (default); >0 = true BFS up to that link depth, honoring the filters. Returns: {root, pages: [{url, markdown, title, ...}], count, discovered, elapsed_ms} or {error, root} on failure.
documentExtract text from a PDF/DOCX/PPTX URL → markdown (no browser). Use when the user shares a link to a document (PDF, Word, PowerPoint) and wants its text content. Also useful after `search_papers` to get full text from a paper's pdf_url. Content-type sniffed and routed to pypdf / python-docx / python-pptx. Optional deps — install with `pip install 'pyrecrawl[docs]'`.
searchWeb search via DuckDuckGo HTML (no API key required). Use this for targeted searches where you need anti-bot bypass (Cloudflare protection on DDG). For simple searches, the built-in web_search may suffice. For research questions, prefer `deep_research` (search + scrape + citations). Returns [{url, title, snippet}, ...]. The smart ladder bypasses DDG's bot detection if needed. Returns: {query, results: [{url, title, snippet}], count} or {error, query} on failure.
batch_scrapeScrape MANY URLs in ONE call (parallel, deduped, cache-aware). Use when the user provides multiple URLs or you have a list of pages to fetch. More efficient than calling `scrape` N times. Args: urls: Target URLs (deduped automatically; empties dropped). prefer: "auto" | "fast" | "stealth" | "llm". timeout: per-URL timeout in seconds. max_concurrency: parallel workers (default 4). include_html: include raw HTML per result (large; off by default). Returns {requested, unique, succeeded, failed, results[]}. Per-URL failures are isolated — other URLs still succeed. Returns: {requested, unique, succeeded, failed, results: [{url, markdown, ...}]}
deep_researchSearch the web, then pull the top sources as EVIDENCE (no LLM synthesis). PRIMARY RESEARCH TOOL — use when the user asks to research, investigate, deep-dive, fact-check, or learn about a topic. Returns a ``citations`` list with stable [n] numbers and an ``evidence`` list of per-source markdown — the agent does the synthesis from evidence. Multi-pass mode: set iterations=2-3 to auto-run additional searches with refined queries (alternatives, criticism, latest developments) and append deduplicated evidence. Each pass adds up to ``scrape_top`` new sources. Args: query: search string. limit: how many search results to fetch. scrape_top: how many of those to actually fetch content from. prefer: "auto" | "fast" | "stealth" | "llm". iterations: 1 (default, single pass), 2-3 (multi-pass with refined queries targeting evidence gaps). Each pass searches from a different angle and deduplicates by URL. Returns: {query, iterations_run, queries: [str], hits: [{url, title, snippet}], citations: [{url, title}], evidence: [{url, title, markdown}], scraped, used_engines, elapsed_ms} or {error, query, hint} on failure.
monitorTrack a URL over time and report meaningful content changes. Use when the user wants to watch a page for updates (price changes, new blog posts, status updates). Ask me to 'set up monitoring for <url>' for a guided setup playbook. Args: url: target URL. action: "check" | "history" | "forget". prefer: ladder preference, same as ``scrape``. css_selector: scope the diff to one element (so banner / nav changes don't trigger false positives). Snapshots persist under ``PYRECRAWL_MONITOR_DIR`` (default ``~/.pyrecrawl/monitors/``). ``check`` returns ``status`` of ``new`` | ``unchanged`` | ``changed`` | ``error`` and a unified diff when the page changed. Returns: {url, status: "new"|"unchanged"|"changed"|"error", diff?: str, snapshot_chars?: int, elapsed_ms?: int}
search_papersSearch academic papers via arXiv or Crossref — no API keys. Args: query: free-text search, e.g. "transformer attention scaling laws". limit: max results (1-25 arXiv / 1-20 crossref). source: "arxiv" (CS/physics/math preprints, default) or "crossref" (all fields, DOI-backed). category: optional arXiv category filter, e.g. "cs.LG", "cs.CV". Returns papers with id/url/pdf_url/title/authors/summary/published. Feed pdf_url into the `document` tool to extract full text. Returns: {query, source, papers: [{title, authors, abstract, url, pdf_url?, ...}]} or {error, query} on failure.
cacheInspect the response cache: stats, clear, enable, or disable. Use this to check cache hit rates before large batch jobs, or to clear stale cached responses when a site's content has changed. Args: action: "stats" (default) | "clear" | "disable" | "enable".
healthVerify PyreCrawl is working: engine versions, dependencies, update status. Use this at the start of a session or before a large scraping job to confirm all engines are installed and up to date.
Show all 13 toolsShow fewer tools
sessionDrive a persistent browser session — cookies & JS state kept across calls. Use for login walls and multi-step flows the one-shot ladder can't handle (e.g. user needs to log in first, then scrape a protected page). For simple pages use `scrape`; for research use `deep_research`. One-shot scrape has no session memory; here each action runs against the same live page. Args: session: named session; reuse the same name to keep state. action: "open" (url) — navigate, returns url/title/status "click" (selector) — click an element "fill" (selector, text) — type into an input "type" (key) — press a key, e.g. "Enter" "eval" (js) — run a JS expression, returns value "wait" (selector?) — wait for selector or sleep timeout_ms "content" — url/title/visible text of current page "screenshot" (full_page?) — returns png_base64 "cookies" — list session cookies "close" — destroy the session "list" — show live sessions Returns {session, action, ...result, elapsed_ms}; errors as {error}.
Configuration
PYRECRAWL_NO_TELEMETRY
Automated checks
Ran Nov 13, 2026Full report →Passed our automated checks. That is not a security audit: read the code and give the server the narrowest access it needs. How checks work
- Source repository linked and reachablesanggonboy/pyrecrawl is public and reachablegithub.com/sanggonboy/pyrecrawl ↗Pass
- Package names this repository as its sourcepyrecrawl names sanggonboy/pyrecrawl as sourcepypi.org/project/pyrecrawl/ ↗Pass
- Build provenance (Sigstore attestation)No build attestation published for this releaseNot checked
- Open source license (OSI)MITspdx.org/licenses/MIT.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-09-21github.com/sanggonboy/pyrecrawl ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNo popular package one edit away from pyrecrawlPass
- No critical advisory in the package or its dependencies115 resolved dependencies checked; no critical advisoryosv.dev ↗Pass
- No hidden instructions to the model0 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNot checked
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHub🔥 PyreCrawl — Web Browsing Superpowers for Your AI Agent
One command gives any AI agent the whole web. Scrape, extract, crawl, map, and search — self-hosted, no API keys, no rate limits, no subscription.
PyreCrawl speaks MCP (Model Context Protocol), the standard tool interface for Claude, Cursor, VS Code, Codex, OpenCode, Hermes, and any MCP-compatible agent.
A smart auto-fallback ladder always picks the cheapest method that succeeds:
fast HTTP
│ (403/503/Cloudflare challenge or empty body)
▼
stealth browser (real Chromium + Cloudflare solver)
│ (still blocked, or the page needs full JS rendering)
▼
deep processing (LLM-ready markdown, citations, structured extraction)
⚡ Tools exposed
| Tool | What it does |
|---|---|
scrape(url, prefer="auto") |
Single URL → LLM-ready markdown |
extract(url, schema) |
Scrape + structured extraction (JsonCss schema) |
map_site(root, include_pattern=None, limit=200) |
Enumerate all internal URLs |
crawl(root, max_pages=5, prefer="auto", include_paths=None, exclude_paths=None, max_depth=0) |
Multi-page crawl with path filters + true BFS depth |
document(url) |
PDF/DOCX/PPTX → markdown (no browser, optional [docs] extras) |
search(query, limit=10) |
Web search via DuckDuckGo HTML (no API key) |
search_papers(query, limit=8, source="arxiv", category=None) |
Academic search via arXiv + Crossref (no API key) — feed pdf_url into document |
batch_scrape(urls[], ...) |
Many URLs in ONE call — parallel, deduped, cache-aware |
deep_research(query, limit=5, scrape_top=3) |
Search → evidence pack with [n] citations (no LLM synthesis — your agent does that) |
monitor(url, action, css_selector=None) |
Change detection with persisted snapshots + unified diff |
session(session, action, ...) |
Persistent browser session (cookies kept) — login walls, multi-step flows, screenshots |
cache(action) |
Inspect/clear/enable/disable the HTTP response cache |
health() |
Versions + import sanity check |
MCP Resources (read-only state without a tool call):
pyrecrawl://cache/stats · pyrecrawl://sessions · pyrecrawl://monitors
MCP Prompts (ready-made playbooks): research(topic) · rag_ingest(site) · watch_page(url)
Env flags
| Variable | Default | Effect |
|---|---|---|
PYRECRAWL_CACHE |
off | 1 = in-memory LRU (128 pages), or a directory path (reserved for disk mode) |
PYRECRAWL_CACHE_TTL |
900 |
Cache entry lifetime in seconds |
PYRECRAWL_MONITOR_DIR |
~/.pyrecrawl/monitors |
Where monitor snapshots persist |
PYRECRAWL_NO_TELEMETRY |
off | 1 = disable the anonymous startup ping (also honors DO_NOT_TRACK=1) |
prefer options: "auto" (default ladder) · "fast" (HTTP only) · "stealth" (CF bypass) · "llm" (deep processing).
🚀 Install & Use (one-liner)
1. Install
UV (recommended — one command, zero Python setup)
UV is a fast Python package manager that handles Python itself — no need to install Python separately. Get it once:
# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows (PowerShell)
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
Then run PyreCrawl directly — no venv, no pip install, no Python download:
uvx pyrecrawl@latest
Or via uv tool install (persistent, recommended for regular use)
uv tool install pyrecrawl
Or via pipx (alternative)
pipx install pyrecrawl
Or via pip into a venv
pip install pyrecrawl
2. One-time browser engines
pyrecrawl setup
This installs Chromium + stealth browser engines (~2 min, one-time).
3. Register with your AI agent
# Auto-detect installed agents and write their MCP configs
pyrecrawl install
# Or target specific agents
pyrecrawl install claude-desktop cursor
# Dry-run to preview what would change
pyrecrawl install --dry-run
Supported agents: claude-desktop, claude-code, cursor, vscode, codex, opencode, hermes.
4. Start chatting
After installing + registering, restart your agent (or start a new session). Then ask:
"Scrape https://example.com and summarize it."
Available tools:
| Tool | What it does |
|---|---|
scrape |
Fetch a single URL → markdown (auto-escalates past Cloudflare) |
extract |
Scrape + structured extraction via CSS schema → JSON |
map_site |
Enumerate all internal URLs from a root |
crawl |
Multi-page crawl: discover + scrape in bulk |
batch_scrape |
Fetch many URLs in one parallel call |
search |
Web search via DuckDuckGo with anti-bot bypass |
search_papers |
Academic paper search (arXiv / Crossref) |
deep_research |
Search + scrape + citations in one call — primary research tool |
document |
Extract text from PDF/DOCX/PPTX URLs |
monitor |
Track a URL for content changes over time |
session |
Persistent browser session for login walls |
cache |
Inspect or clear the response cache |
health |
Verify engine availability + version |
Plus 3 guided prompts: research, rag_ingest, watch_page.
Quick examples
Ask your agent naturally — no special syntax needed:
| You say | Agent uses |
|---|---|
| "Scrape https://example.com and summarize it" | scrape → returns markdown → agent summarizes |
| "Research Rust memory safety vulnerabilities" | deep_research → search + scrape + citations |
| "Deep research on AI regulation worldwide" | deep_research(iterations=3) → multi-pass with refined queries |
| "Extract all product names and prices from this page" | extract → CSS schema → structured JSON |
| "Crawl https://docs.example.com and give me an overview" | crawl → multi-page → summary |
| "Monitor this page for price changes" | monitor → baseline snapshot → periodic diff |
| "Find papers about transformer attention" | search_papers → arXiv results |
| "What's the current cache hit rate?" | cache → stats |
🧠 Skills — Maximize Your Agent's Research Quality
PyreCrawl tools give your agent hands (scrape, crawl, search). But the agent still needs a brain — instructions on when to use which tool, how to chain research passes, and what anti-hallucination rules to follow.
That's what PyreCrawl Skills provides.
| MCP Tools (this repo) | Skills (pyrecrawl-skills) | |
|---|---|---|
| Role | Execute web operations | Tell the agent how to use them |
| Analogy | Hands | Brain |
| Example | deep_research(query, iterations=3) |
"Run 3 passes, check gaps after each, cite everything" |
| Required? | Yes (the engine) | Optional (but recommended for research quality) |
Quick setup:
# 1. Install the tools (you already have this)
uvx pyrecrawl@latest
# 2. Add the research skill to your project
git clone https://github.com/SanggonBoy/pyrecrawl-skills.git /tmp/pyrecrawl-skills
cp /tmp/pyrecrawl-skills/pyrecrawl-research/SKILL.md ./CLAUDE.md # or .cursorrules / AGENTS.md
Without skills: Your agent has powerful tools but improvises usage. With skills: Your agent follows a proven research protocol with anti-hallucination guardrails.
📚 Manual config (if pyrecrawl install doesn't match your setup)
Claude Desktop
Config file
- Linux:
~/.config/Claude/claude_desktop_config.json - macOS:
~/Library/Application Support/Claude/claude_desktop_config.json - Windows:
%AppData%\Claude\claude_desktop_config.json
{
"mcpServers": {
"pyrecrawl": {
"command": "uvx",
"args": ["--from", "pyrecrawl", "pyrecrawl", "serve"]
}
}
}
Claude Code
Config file: project-scoped .mcp.json
{
"mcpServers": {
"pyrecrawl": {
"command": "uvx",
"args": ["--from", "pyrecrawl", "pyrecrawl", "serve"]
}
}
}
Cursor
Config file: ~/.cursor/mcp.json
{
"mcpServers": {
"pyrecrawl": {
"command": "uvx",
"args": ["--from", "pyrecrawl", "pyrecrawl", "serve"]
}
}
}
VS Code / Copilot
Config file: .vscode/mcp.json (project-scoped)
{
"servers": {
"pyrecrawl": {
"command": "uvx",
"args": ["--from", "pyrecrawl", "pyrecrawl", "serve"],
"type": "stdio"
}
}
}
Codex CLI
Config file: ~/.codex/config.toml
[mcp_servers.pyrecrawl]
command = "uvx"
args = ["--from", "pyrecrawl", "pyrecrawl", "serve"]
OpenCode
Config file: ~/.config/opencode/opencode.json
{
"mcp": {
"pyrecrawl": {
"type": "local",
"command": ["uvx", "--from", "pyrecrawl", "pyrecrawl", "serve"],
"enabled": true
}
}
}
Hermes
Config file
- Linux/macOS:
~/.hermes/config.yaml - Windows:
%LocalAppData%\hermes\config.yaml
mcp_servers:
pyrecrawl:
command: uvx
args:
- --from
- pyrecrawl
- pyrecrawl
- serve
enabled: true
Windows note:
uvxmust be on PATH. If not, use the full path touvx.exe(e.g.C:\Users\<you>\AppData\Local\hermes\bin\uvx.exe).
🧠 How the ladder chooses
PyreCrawl runs each request through three tiers, stopping at the first one that returns a complete, LLM-ready result:
| Concern | Fast tier | Stealth tier | Deep tier |
|---|---|---|---|
| Static HTML page | ✅ ~200ms | — | — |
| Cloudflare-protected | ❌ | ✅ Turnstile solver | — |
| JS-heavy SPA | ❌ | ✅ real Chromium | — |
Live DOM data (input .value, JS state) |
❌ | ✅ js param |
— |
| LLM-ready markdown + citations | — | — | ✅ BM25, fit-markdown |
| Structured extraction (CSS schema) | — | — | ✅ |
| Deep crawl (BFS/DFS/BestFirst) | — | — | ✅ adaptive |
The agent never has to pick. prefer="auto" does it every call.
Live DOM data with js and wait_for
Some sites keep the data you want in a DOM property (e.g. an <input>'s .value)
that JS writes after an XHR — it never appears in the serialized HTML. The
scrape tool accepts two stealth-tier params for exactly this:
{
"url": "https://temp-mail.org/id",
"prefer": "stealth",
"wait_for": "document.getElementById('mail').value.includes('@')",
"js": "document.getElementById('mail').value"
}
wait_for— a JS predicate expression polled until truthy (bounded bytimeout). Use it instead of guessing a sleep for anything that arrives asynchronously.js— a JS expression evaluated once the page settles; the value comes back inmeta.js_result. Errors are captured inmeta.js_error(the page result is still returned, never a crash).
📊 Compared to Firecrawl (hosted)
| Firecrawl | PyreCrawl | |
|---|---|---|
| Cost | Free 1k/mo, then $16–333/mo | Free, self-hosted |
| Local LLM support | ❌ | ✅ Ollama / any LLM |
| Cloudflare bypass | ✅ (Fire-Engine, paid) | ✅ (free, built-in) |
| Markdown + BM25 | ✅ | ✅ |
| Self-host | ❌ | ✅ |
| Academic paper search | ❌ | ✅ arXiv + Crossref (search_papers) |
| Hosted search API | ✅ /search | ⚠️ DuckDuckGo HTML + arXiv/Crossref (no key) |
🔧 Development
git clone https://github.com/SanggonBoy/PyreCrawl.git
cd PyreCrawl
uv venv --python 3.12 .venv
source .venv/Scripts/activate # Windows; or .venv/bin/activate on macOS/Linux
uv pip install -e ".[dev]"
python -m playwright install chromium
scrapling install
Run tests
python scripts/selfcheck.py # real-network smoke test (13 tools + engines)
python scripts/probe_stdio.py # stdio JSON-RPC probe
python scripts/test_ladder_bug.py # SPA-shell ladder escalation regression
python scripts/test_js_eval.py # stealth js/wait_for params regression
python scripts/test_scope_selector.py # crawl css_selector/max_depth wiring
python scripts/test_link_harvest.py # map/BFS link purity regression
📦 Publish
Maintainers only:
git tag vX.Y.Z
git push origin vX.Y.Z
GitHub Actions builds + uploads to PyPI via trusted publishing.
🔔 Stay up to date
PyreCrawl checks PyPI on every startup and reports the latest version — your
MCP agent sees this automatically via the health() tool response and can
notify you inline.
To check manually:
pyrecrawl version
To upgrade:
pyrecrawl update # runs: uv tool upgrade pyrecrawl
Get notified of new releases: click Watch → Releases only at the GitHub repo to receive email notifications when a new version is published.
[!NOTE] PyreCrawl sends one anonymous usage ping per 24 h at server startup — see Privacy for exactly what's sent and how to opt out.
🔒 Privacy — anonymous usage ping
PyreCrawl phones home once per 24 h with a tiny anonymous ping when the MCP server starts, so we can count real users (DAU/MAU) instead of raw downloads.
| Sent (4 fields, ~100 bytes) | Never sent |
|---|---|
| Hashed machine id (SHA-256 of hostname+MAC — not reversible) | Your IP (not stored) |
| PyreCrawl version | Any URL you scrape |
| Python version | Any page content or search queries |
OS family (windows / linux / darwin) |
Anything else |
Client code: src/pyrecrawl/telemetry.py (~90 lines, stdlib only) ·
Collector: workers/telemetry/ — a self-hostable Cloudflare Worker + D1, no third-party analytics service.
Opt out any time:
export PYRECRAWL_NO_TELEMETRY=1 # or the industry-standard DO_NOT_TRACK=1
📜 Uninstall
# Remove from all agent configs
pyrecrawl uninstall
# Remove the package
uv tool uninstall pyrecrawl
🛡️ License
MIT — see LICENSE.
PyreCrawl MCP questions
How do I add PyreCrawl MCP to Claude Code?
Run claude mcp add pyrecrawl -- uvx pyrecrawl serve in your terminal. Add --scope user to make it available in every project.
How do I add PyreCrawl MCP to Cursor?
Open ~/.cursor/mcp.json and add the server under mcpServers, using the Cursor config shown in the install section above.
What tools does PyreCrawl MCP provide?
It exposes 13 tools, including scrape, extract, map_site, crawl, document.
How does PyreCrawl handle Cloudflare or bot protection?
It uses a smart auto-fallback ladder that can escalate from fast HTTP to a stealth browser with a real Chromium engine and Cloudflare solver.