fetchkit MCP server
Commit Nov 3, 2026Enables AI agents to autonomously fetch, scrape, extract, and download web content using realistic browser fingerprints and structured outputs.
Also known as pr1m8/pyfetcher
- Stars
- 2
- Tools
- 16
- Downloads, 30 days
- —
- Latest version
- —
- Last commit
- Nov 3, 2026
- License
- MIT
What is fetchkit MCP?
fetchkit enables AI agents to autonomously fetch, scrape, extract, and download web content using realistic browser fingerprints and structured outputs. It exposes 16 tools, such as fetch_url, fetch_multiple, scrape_css.
Summary generated from the project's README
About fetchkit MCP server
pyfetcher provides an agentic web infrastructure designed to give AI agents the ability to autonomously interact with the internet. It offers 16 specialized tools that allow agents to fetch, scrape, extract, and download web content using structured Pydantic outputs instead of raw HTML. This server is built for developers using clients like Claude Desktop, Claude Code, or LangChain, and can be run via stdio or as an HTTP server for remote agents.
The system distinguishes itself through realistic browser identity management, offering 11 different profiles with consistent headers to help bypass detection. It supports multiple backends including httpx, aiohttp, curl_cffi for TLS fingerprinting, and cloudscraper for Cloudflare bypass. For more complex workflows, it can be integrated with a full pipeline using Postgres for job queues and MinIO for object storage, enabling concurrent crawl, scrape, and download operations.
What you can do with it
- 1Autonomous Web ScrapingAn AI agent can use the provided tools to extract specific text or links from a webpage using CSS or XPath selectors.
- 2Large Scale Data PipelinesYou can set up a durable system to manage complex crawling and downloading tasks using a database and object storage.
Written from the project's README. Every use case is checked against a quote from it.
fetchkit MCP tools 16
Detected from the READMEfetch_urlfetch_multiplescrape_cssscrape_linksscrape_textscrape_metadatascrape_formsscrape_tablecheck_robotsparse_sitemapgenerate_headerslist_profiles
Show all 16 toolsShow fewer tools
random_user_agentextract_articleconvert_htmldownload_file
Configuration
FETCHKIT_REDIS_URL
Automated checks
Ran Nov 14, 2026Full report →Not verified yet. A server needs a score of 80, no serious finding, and enough data for us to check. How checks work · What to fix for more points
- Source repository linked and reachablepr1m8/pyfetcher is public and reachablegithub.com/pr1m8/pyfetcher ↗Pass
- Package names this repository as its sourceNot checked
- Build provenance (Sigstore attestation)Not checked
- Open source license (OSI)MITspdx.org/licenses/MIT.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-11-03github.com/pr1m8/pyfetcher ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNot checked
- No critical advisory in the package or its dependenciesNot checked
- No hidden instructions to the model16 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNo tool named like delete, drop or execPass
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubfetchkit
Agentic web infrastructure for autonomous fetching, scraping, and content acquisition.
Give AI agents the power to fetch, scrape, extract, and download anything on the web -- with realistic browser fingerprints, structured outputs, and a full crawl-scrape-download pipeline backed by Postgres and MinIO.
MCP Server | Quick Start | Pipeline | CLI | Documentation | Changelog | Examples
Why fetchkit?
The problem: AI agents need to interact with the web -- fetch pages, extract data, download files -- but existing tools aren't designed for autonomous operation. They lack structured outputs, realistic browser fingerprints, and pipeline orchestration.
fetchkit solves this by providing:
- MCP Server -- 16 tools that any AI agent (Claude, LangChain, LangGraph) can call directly. Structured Pydantic outputs, not raw HTML.
- Realistic browser identity -- 11 profiles with consistent UA + Client Hints + Sec-Fetch-* headers. TLS fingerprinting via curl_cffi. Cloudflare bypass.
- Full pipeline -- Concurrent crawl -> scrape -> download workers backed by Postgres job queues and MinIO object storage.
- Typed acquisition providers -- direct HTTP, yt-dlp, and gallery-dl behind capability discovery, provenance, credential references, and async progress events.
- Reusable durable jobs -- versioned pagination, media-collection, and site-search templates with portable artifacts; run locally or through optional Redis Streams and Temporal adapters.
- Typed page extraction -- chainable CSS/XPath selectors, explicit cardinality and conversion, filtered links, reversible URL patterns, responsive images, and gallery collections.
- URL interoperability -- one immutable HTTP/HTTPS value across Pydantic, urllib, HTTPX, yarl, tldextract, Requests/Scrapy-style objects, and structural adapters for other URL libraries.
Project status
fetchkit is in active alpha development. The typed request/response contracts,
HTTP transports, reusable clients, scraping helpers, and CLI are usable today.
The database pipeline, browser automation, downloader integrations, and MCP
server are broader optional layers and may evolve more quickly. The PyPI package
is named fetchkit; Python imports use pyfetcher.
The extension model deliberately separates ordinary HTTP transports, the optional Scrapy crawler-engine bridge, and acquisition providers such as yt-dlp and gallery-dl. See adapters and execution engines for the implemented transport, middleware, provider, and focused-skill boundaries.
pip install 'fetchkit[mcp]' # AI agent integration
pip install 'fetchkit[full]' # Everything
Highlights
pip install fetchkit # Core: fetch, scrape, headers
pip install 'fetchkit[mcp]' # + MCP server for AI agents
pip install 'fetchkit[pipeline]' # + Postgres job queue + MinIO storage
pip install 'fetchkit[full]' # Everything including yt-dlp, Playwright, etc.
Fetch with realistic browser headers
from pyfetcher import fetch
response = fetch("https://example.com")
print(response.status_code, response.ok)
# Sends Chrome-like headers with Client Hints, Sec-Fetch-*, UA rotation automatically
Reuse an async HTTP/2 connection pool
from pyfetcher import AsyncFetchClient, PoolPolicy
async with AsyncFetchClient(
http2=True,
pool=PoolPolicy(max_connections=100, concurrency=20),
) as client:
response = await client.get("https://example.com")
print(response.status_code, response.http_version)
Keep one client for the lifetime of a service. HTTP/2 is negotiated with the
origin and may fall back to HTTP/1.1; response.http_version reports what was
actually used.
Scrape anything
from pyfetcher.scrape import extract_links, extract_text, extract_readable_text
links = extract_links(html, base_url="https://example.com") # all links with internal/external tags
titles = extract_text(html, "h1") # CSS selector extraction
article = extract_readable_text(html) # strips scripts, nav, ads
For reusable typed extraction, install fetchkit[selectors]:
from pyfetcher import PageDocument, SelectorRule
page = PageDocument(html, url="https://example.com/gallery/42")
title = SelectorRule(
css="h1",
output="text",
cardinality="one",
transforms=("normalize_whitespace",),
).apply(page)
image_urls = page.xpath("//article//img/@src").getall()
4 HTTP backends -- pick the right one for the job
from pyfetcher import FetchRequest, fetch
response = fetch("https://example.com") # httpx (default, HTTP/2)
response = fetch(FetchRequest(url="https://example.com", backend="aiohttp")) # aiohttp (pure async)
response = fetch(FetchRequest(url="https://example.com", backend="curl_cffi")) # TLS fingerprinting
response = fetch(FetchRequest(url="https://example.com", backend="cloudscraper")) # Cloudflare bypass
Download media with yt-dlp & gallery-dl
from pyfetcher.downloaders.ytdlp import YtdlpDownloader
from pyfetcher.downloaders.gallerydl import GalleryDlDownloader
# Video/audio with progress tracking
yt = YtdlpDownloader()
info = await yt.extract_info("https://youtube.com/watch?v=...") # metadata only
results = await yt.download("https://youtube.com/watch?v=...", # full download
output_dir="./media", progress_callback=lambda p: print(p.status))
# Image galleries (170+ supported sites)
gdl = GalleryDlDownloader()
results = await gdl.download("https://imgur.com/gallery/...", output_dir="./images")
MCP Server -- give AI agents web superpowers
pyfetcher-mcp # stdio for Claude Desktop / Claude Code
pyfetcher-mcp --http 8000 # HTTP for LangChain / remote agents
# LangChain integration
from langchain_mcp_adapters import MultiServerMCPClient
client = MultiServerMCPClient({"pyfetcher": {"transport": "http", "url": "http://localhost:8000/mcp"}})
tools = await client.get_tools() # 16 structured tools ready for any agent
Features
Core Library
| Feature | Description |
|---|---|
| Browser Headers | 11 profiles (Chrome/Firefox/Safari/Edge) across 5 platforms. Consistent UA + Client Hints + Sec-Fetch-*. Market-share-weighted rotation. |
| 4 Backends | httpx (default, HTTP/2), aiohttp (async), curl_cffi (TLS fingerprint), cloudscraper (CF bypass) |
| Reusable Clients | Sync and async lifecycle clients with connection pooling, HTTP/2 negotiation, base URLs, and shared defaults |
| Streaming I/O | Sync/async response iterators plus bytes, text, sync-generator, and async-generator request content |
| Acquisition | Direct, yt-dlp, and gallery-dl providers with discovery, native options, provenance, credential references, and progress events |
| Middleware | Composable sync/async request-response layers shared by fetch and acquisition services |
| Job runtime | Versioned templates plus a service, status store, bounded worker, cancellation, retries, and dynamic child fan-out |
| Artifacts | Blob references and ordered gallery, playlist, search, and page collections without queueing large bytes |
| Contracts | Versioned JSON Schema, generated TypeScript clients, provider discovery, queues, protobuf, and gRPC service bindings |
| Rate Limiting | Per-domain + global token bucket with configurable burst |
| Retry | Exponential backoff via Tenacity with configurable status codes |
| Scraping | BeautifulSoup helpers plus typed Parsel CSS/XPath rules, URL/link filtering, images, and collections |
| Typed URLs | Pydantic, urllib, HTTPX, yarl, tldextract, request/response adapters, relative resolution, and canonical identity |
| Metadata | HTML meta, Open Graph, JSON-LD, microdata, RDFa, Dublin Core |
| CLI | Fetch, scrape, acquire, inspect templates, and export portable contracts |
| TUI | Interactive Textual terminal UI for building and inspecting requests |
Infrastructure (optional extras)
| Feature | Extra | Description |
|---|---|---|
| Pipeline | [pipeline] |
Concurrent Crawl -> Scrape -> Download workers over durable Postgres jobs |
| Database | [db] |
SQLAlchemy 2.0 async + Alembic. Jobs, pages, media, hosts, feeds, URL dedup |
| Object Store | [store] |
MinIO/S3 via aioboto3. Upload, download, presigned URLs |
| Downloaders | [downloaders] |
yt-dlp (progress hooks, info_dict) + gallery-dl (170+ sites) |
| Extractors | [extractors] |
trafilatura + readability-lxml fallback, html2text, markdownify |
| Media | [media] |
Audio (mutagen), video (pymediainfo), image (exifread), PDF (pypdf) |
| Browser | [browser] |
Playwright + stealth for JS-heavy sites |
| Scrapy | [scrapy] |
Optional request/response conversion and crawler-engine bridge |
| WebPoet | [web-poet] |
Page-object HTTP adapter and allowlisted standalone page execution |
| Scrapy-Poet | [scrapy-poet] |
Complete WebPoet dependency injection and rules in Scrapy crawls |
| Selectors | [selectors] |
Parsel-backed CSS/XPath page documents and typed extraction rules |
| Redis queue | [redis] |
Streams consumer groups, leases, dead letters, and delayed retries |
| Temporal | [temporal] |
Durable dynamic flows with side effects isolated in activities |
| Job API | [api] |
FastAPI control plane, worker health, cancellation, paging, and SSE events |
| Telemetry | [observe] |
OTLP traces and metrics, W3C propagation, Tempo and log correlation |
| Feeds | [feeds] |
RSS/Atom monitoring with adaptive polling |
| Crawler | [pipeline] |
URL frontier, spider + router, dedup, politeness, sitemap discovery |
Installation
pip install fetchkit
All optional extras:
pip install 'fetchkit[tui]' # Textual TUI
pip install 'fetchkit[curl]' # curl_cffi TLS fingerprinting
pip install 'fetchkit[cloudscraper]' # Cloudflare bypass
pip install 'fetchkit[db]' # Postgres + SQLAlchemy + Alembic
pip install 'fetchkit[store]' # MinIO/S3 object storage
pip install 'fetchkit[pipeline]' # db + store (full pipeline)
pip install 'fetchkit[downloaders]' # yt-dlp + gallery-dl
pip install 'fetchkit[extractors]' # trafilatura, readability, html2text
pip install 'fetchkit[media]' # Audio/video/image/PDF metadata
pip install 'fetchkit[browser]' # Playwright + stealth
pip install 'fetchkit[scrapy]' # Scrapy crawler-engine bridge
pip install 'fetchkit[web-poet]' # WebPoet page-object adapter
pip install 'fetchkit[scrapy-poet]' # WebPoet + full Scrapy integration
pip install 'fetchkit[selectors]' # Typed CSS/XPath page extraction
pip install 'fetchkit[url-tools]' # yarl, w3lib canonicalization, public-suffix domains
pip install 'fetchkit[redis]' # Redis queue + durable job status store
pip install 'fetchkit[grpc]' # gRPC job control plane + protobuf framing
pip install 'fetchkit[temporal]' # Temporal durable flow runtime
pip install 'fetchkit[api]' # FastAPI durable-job control plane
pip install 'fetchkit[observe]' # OpenTelemetry OTLP traces + metrics
pip install 'fetchkit[feeds]' # RSS/Atom feed parsing
pip install 'fetchkit[langchain]' # LangChain MCP client adapters
pip install 'fetchkit[full]' # Everything
Quick Start
Fetch
from pyfetcher import fetch, afetch, FetchRequest
import asyncio
# Sync
response = fetch("https://example.com")
print(response.status_code, response.ok)
# Async
response = asyncio.run(afetch("https://example.com"))
The functions are convenient for scripts. Long-running applications should reuse a client so keepalive and HTTP/2 connections remain pooled:
import asyncio
from pyfetcher import AsyncFetchClient
async def main() -> None:
async with AsyncFetchClient(base_url="https://api.example.com/v1/") as client:
response = await client.get("users")
print(response.status_code, response.http_version)
asyncio.run(main())
Streaming responses and request bodies
Response bodies can be consumed as normalized chunks, raw bytes, or incrementally decoded text:
from pyfetcher import AsyncFetchClient, StreamPolicy
async with AsyncFetchClient(stream=StreamPolicy(max_bytes=10_000_000)) as client:
async for data in client.iter_bytes("GET", "https://example.com/large-file"):
await destination.write(data)
Uploads accept ordinary generators and async generators without first joining the whole body in memory:
async def content():
yield b"first chunk\n"
yield b"second chunk\n"
async with AsyncFetchClient() as client:
response = await client.post("https://example.com/upload", content=content())
For a large or generated URL source, imap() keeps both source consumption and
network concurrency bounded:
async with AsyncFetchClient() as client:
async for response in client.imap(url_generator(), concurrency=20, ordered=False):
print(response.final_url, response.status_code)
FastAPI lifespan and dependency injection
Create one client in the FastAPI lifespan, store it in application state, and inject that same pooled instance into routes:
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
from typing import Annotated
from fastapi import Depends, FastAPI, Request
from pyfetcher import AsyncFetchClient
@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
async with AsyncFetchClient(http2=True) as client:
app.state.fetcher = client
yield
app = FastAPI(lifespan=lifespan)
def get_fetcher(request: Request) -> AsyncFetchClient:
return request.app.state.fetcher
Fetcher = Annotated[AsyncFetchClient, Depends(get_fetcher)]
@app.get("/title")
async def title(fetcher: Fetcher) -> dict[str, str | None]:
response = await fetcher.get("https://example.com")
return {"protocol": response.http_version, "html": response.text}
Provider-neutral acquisition
Select direct HTTP, yt-dlp, or gallery-dl explicitly, or allow conservative URL-based capability selection:
from pyfetcher import AcquisitionRequest, AcquisitionService
request = AcquisitionRequest(
url="https://www.youtube.com/watch?v=...",
provider="yt_dlp",
operation="metadata",
yt_dlp_options={"noplaylist": True},
)
async with AcquisitionService() as service:
async for event in service.aiter_events(request):
print(event.kind, event.provider)
Shared headers, cookies, proxies, output policy, credential references, and progress use common contracts. Provider-native yt-dlp and gallery-dl options remain isolated. See the acquisition guide.
Application downloaders can join the same API through
DownloaderAcquisitionProvider. Custom options are JSON-safe, namespaced by
stable provider name, validated by an optional Pydantic model, and exposed as
JSON Schema for generated TypeScript clients. Installed provider plugins are
loaded explicitly rather than during import.
Versioned contracts for TypeScript and RPC
Export the portable Python contracts as deterministic JSON Schema instead of maintaining a second set of request and result types by hand:
pyfetcher contracts list
pyfetcher contracts export ./generated/fetchkit --split
pyfetcher contracts export ./generated/fetchkit-ts --typescript
pyfetcher contracts proto ./generated/proto/jobs.proto
The manifest covers HTTP, extraction, acquisition, media, artifacts, search,
jobs, queues, and built-in template inputs. It also reports provider
capabilities, custom provider option schemas, and RPC service bindings. The
TypeScript export adds self-contained types, a typed control-plane client,
runtime descriptors, JSON schemas, and the packaged proto. Each
name@version contract has a stable $id and checksum suitable for CI drift
checks. Runtime callables, byte iterators, clients, and resolved credentials
remain Python-side. See the
cross-language contract guide.
Reusable pagination and media jobs
Jobs are JSON-safe data that reference an allowlisted template version. They can run in-process today or through the optional Redis Streams and Temporal runtimes:
from pyfetcher.jobs import JobRequest, PaginationInput, TemplateRef
job = JobRequest(
template=TemplateRef(name="web.pagination", version="1"),
target="https://example.com/archive?page=1",
payload=PaginationInput(
mode="enumerate",
page_link_selector="nav.pagination a",
visit_template=TemplateRef(name="article.extract", version="2"),
).model_dump(mode="json"),
)
Built-ins cover last-page discovery, page enumeration, next-link walking,
provider-neutral media/gallery/playlist acquisition, and pluggable site search.
Results use ArtifactRef, BlobRef, and ordered ArtifactCollection manifests,
so HTML, images, and videos stay out of queue messages and workflow histories.
See durable jobs and reusable flows.
Run a job or flow file directly, or host the same service through FastAPI:
pyfetcher jobs run flow.json --concurrency 8 --artifacts-dir ./artifacts
pyfetcher jobs serve --host 127.0.0.1 --port 8000 --concurrency 8
pyfetcher jobs grpc-serve --host 127.0.0.1 --port 50051
FETCHKIT_REDIS_URL=redis://localhost:6379/0 pyfetcher jobs serve
The default queue and result store are process-local. Install
fetchkit[api,redis] and use --redis-url or FETCHKIT_REDIS_URL to pair the
built-in Redis Streams queue with the durable Redis status/result store without
changing job templates. Queue deliveries remain at least once, so templates
with external side effects still need downstream idempotency.
Browser Profiles & Headers
from pyfetcher.headers.browser import BrowserHeaderProvider
from pyfetcher.headers.rotating import RotatingHeaderProvider
from pyfetcher.headers.ua import random_user_agent
from pyfetcher.fetch.service import FetchService
# Fixed profile (Chrome on Windows)
service = FetchService(header_provider=BrowserHeaderProvider("chrome_win"))
# Rotating profiles weighted by real-world market share
service = FetchService(header_provider=RotatingHeaderProvider())
# Just need a user-agent string?
ua = random_user_agent(browser="firefox", platform="macOS")
Scraping
from pyfetcher.scrape import (
extract_links, extract_text, extract_table,
extract_forms, extract_readable_text,
)
from pyfetcher.scrape.robots import parse_robots_txt, is_allowed
# CSS selectors
titles = extract_text(html, "h1.title")
rows = extract_table(html, "table.data")
# Links with internal/external classification
links = extract_links(html, base_url=url, same_domain_only=True)
# Forms with field extraction
forms = extract_forms(html, base_url=url)
print(forms[0].action, forms[0].to_dict())
# Robots.txt
rules = parse_robots_txt(robots_content)
allowed = is_allowed(rules, "/admin", user_agent="MyBot")
Rate-Limited Fetching
from pyfetcher.fetch.service import FetchService
from pyfetcher.ratelimit.limiter import DomainRateLimiter, RateLimitPolicy
limiter = DomainRateLimiter(
default_policy=RateLimitPolicy(requests_per_second=2.0, burst=5),
domain_policies={
"api.example.com": RateLimitPolicy(requests_per_second=0.5),
},
)
service = FetchService(rate_limiter=limiter)
Content Extraction
from pyfetcher.extractors.content import extract_article_text
from pyfetcher.extractors.convert import html_to_markdown, html_to_plaintext
# Article text (trafilatura with readability-lxml fallback)
article = extract_article_text(html, url="https://example.com/post")
# HTML -> Markdown
md = html_to_markdown(html)
yt-dlp & gallery-dl
from pyfetcher.downloaders.ytdlp import YtdlpDownloader
from pyfetcher.downloaders.gallerydl import GalleryDlDownloader
# yt-dlp with progress tracking
yt = YtdlpDownloader()
info = await yt.extract_info("https://youtube.com/watch?v=dQw4w9WgXcQ")
results = await yt.download(url, output_dir="./videos",
progress_callback=lambda p: print(f"{p.status}: {p.percent}"))
# gallery-dl for image galleries (170+ supported sites)
gdl = GalleryDlDownloader()
results = await gdl.download("https://imgur.com/gallery/...", output_dir="./images")
CLI
# Fetch with any backend
pyfetcher fetch https://example.com
pyfetcher fetch https://example.com -o json -b curl_cffi
# Preview generated headers
pyfetcher headers --profile chrome_win
pyfetcher headers --browser firefox -o json
pyfetcher headers --list
fetchkit MCP questions
What tools does fetchkit MCP provide?
It exposes 16 tools, including fetch_url, fetch_multiple, scrape_css, scrape_links, scrape_text.
What can AI agents do with fetchkit?
AI agents can use 16 structured tools to fetch pages, scrape data, extract content, and download files with realistic browser identities.