Provides a self-hosted, Tavily-compatible web search, extraction, crawling, and mapping API for AI agents.
Also known as khsarvar/siftdog
- Stars
- 3
- Tools
- 4
- Downloads, 30 days
- 109
- Latest version
- 0.3.0
- Last commit
- Oct 9, 2026
- License
- MIT
What is Siftdog MCP?
Siftdog provides a self-hosted, Tavily-compatible web search, extraction, crawling, and mapping API for AI agents. It exposes 4 tools, such as siftdog_search, siftdog_extract, siftdog_crawl.
Summary generated from the project's README
About Siftdog MCP server
Developers seeking a self-hosted alternative to the Tavily API can use Siftdog to provide web search, extraction, crawling, and mapping capabilities to AI agents. It functions as a drop-in replacement by supporting the same request and response shapes for endpoints like /search, /extract, /crawl, and /map. This allows existing code using the official Python SDK or LangChain integration to work simply by updating the base URL.
Users can deploy the server via Docker or pip, connecting it to a SearXNG metasearch instance for search functionality. For MCP clients like Claude Desktop, Cursor, or Claude Code, it can be accessed via a streamable HTTP connection at the /mcp endpoint or run locally over stdio using uvx. The system includes built-in SSRF protection to block private and internal addresses during web fetching. While it uses BM25 for relevance ranking instead of neural reranking, it offers advanced depth settings to fetch and return relevant content chunks.
What you can do with it
- 1Replace proprietary search APIsSwitch from Tavily to a self-hosted solution by pointing your existing Python or LangChain clients to a new base URL.
- 2Automate website crawlingPerform deep crawls of specific domains using depth, breadth, and regex filters to map out site structures.
Written from the project's README. Every use case is checked against a quote from it.
How to install Siftdog MCP
Formats verified 10 Nov 2026Add to CursorAdd to GooseInstall Siftdog MCP in Claude Code
$ claude mcp add siftdog -- uvx siftdog mcpInstall Siftdog MCP in Cursor
~/.cursor/mcp.json
{
"mcpServers": {
"siftdog": {
"command": "uvx",
"args": [
"siftdog",
"mcp"
]
}
}
}Install Siftdog MCP in VS Code
.vscode/mcp.json
{
"servers": {
"siftdog": {
"type": "stdio",
"command": "uvx",
"args": [
"siftdog",
"mcp"
]
}
}
}Install Siftdog MCP in VS Code CLI
$ code --add-mcp '{"name":"siftdog","type":"stdio","command":"uvx","args":["siftdog","mcp"]}'Install Siftdog MCP in Claude Desktop
claude_desktop_config.json
{
"mcpServers": {
"siftdog": {
"command": "uvx",
"args": [
"siftdog",
"mcp"
]
}
}
}Install Siftdog MCP in Codex
$ codex mcp add siftdog -- uvx siftdog mcpInstall Siftdog MCP in Codex config.toml
~/.codex/config.toml
[mcp_servers.siftdog] command = "uvx" args = ["siftdog", "mcp"]
Install Siftdog MCP in Gemini CLI
$ gemini mcp add -s user siftdog uvx -- siftdog mcpInstall Siftdog MCP in Gemini settings.json
~/.gemini/settings.json
{
"mcpServers": {
"siftdog": {
"command": "uvx",
"args": [
"siftdog",
"mcp"
]
}
}
}Started by HostedMCP on 16 Nov 2026
We ran this command for siftdog 0.3.0 in an isolated container with no network and no keys. It answered and listed 4 tools.
Siftdog MCP tools 4
Read from the server with tools/listsiftdog_searchRead-onlySearch the web and return ranked results with titles, URLs and relevant content. Use this to find sources for a question or to discover pages about a topic. If you already know the URLs, use siftdog_extract instead.
siftdog_extractRead-onlyFetch web pages and return their main content as clean markdown or text. Use this when you already have the URLs, for example from siftdog_search results.
siftdog_crawlRead-onlyCrawl a website from a starting URL and return the content of the pages found. Use this to read many pages of one site. For large sites, call siftdog_map first and narrow the crawl with select_paths.
siftdog_mapRead-onlyList the URLs of a website without fetching their content. Use this to see how a site is organised before extracting or crawling specific pages.
Configuration
SEARXNG_URLURL of a SearXNG instance with JSON output enabled. Needed only for siftdog_search; extract, crawl and map work without it.
Automated checks
Ran Nov 14, 2026Full report →Passed our automated checks. That is not a security audit: read the code and give the server the narrowest access it needs. How checks work
- Source repository linked and reachablekhsarvar/siftdog is public and reachablegithub.com/khsarvar/siftdog ↗Pass
- Package names this repository as its sourcesiftdog names khsarvar/siftdog as sourcepypi.org/project/siftdog/ ↗Pass
- Build provenance (Sigstore attestation)Sigstore attestation for siftdogpypi.org/project/siftdog/ ↗Pass
- Open source license (OSI)MITspdx.org/licenses/MIT.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-10-09github.com/khsarvar/siftdog ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNo popular package one edit away from siftdogPass
- No critical advisory in the package or its dependencies60 resolved dependencies checked; no critical advisoryosv.dev ↗Pass
- No hidden instructions to the model0 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNot checked
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubSiftdog — open-source, self-hosted Tavily alternative
Website · Benchmark · Blog · Hosted waitlist
Siftdog is an open-source, self-hosted web search API for AI agents and LLM apps, and a drop-in
replacement for the Tavily API. It serves the same /search,
/extract, /crawl and /map endpoints with the same request and response shapes, so code
written for Tavily (including the official Python SDK and the LangChain integration) works
against your own server by changing only the base URL. It needs no search API key.
- Search through a SearXNG metasearch instance, with no search API keys
- Extraction of clean markdown or text from web pages with trafilatura
- Ranking: BM25 relevance blended with the upstream engine's order;
advanceddepth fetches each page and returns its most relevant chunks - Answers (
include_answer) written by Claude from the retrieved results - Crawling and site maps with depth, breadth, limit and regex path/domain filters
- MCP server for Claude Code, Claude Desktop, Cursor and other MCP clients, over HTTP at
/mcpor stdio withsiftdog mcp - SSRF protection: private and internal addresses are blocked, including via redirects and DNS rebinding
- MIT licensed
Use it as a drop-in Tavily replacement
Point the official Tavily clients at your Siftdog server with api_base_url. The key can be any
string when auth is disabled, or one of your API_KEYS.
from tavily import TavilyClient
client = TavilyClient(api_key="your-siftdog-key", api_base_url="http://localhost:8000")
results = client.search("latest python release", search_depth="advanced", max_results=5)
pages = client.extract(urls=["https://en.wikipedia.org/wiki/Okapi_BM25"])
LangChain, through langchain-tavily:
from langchain_tavily import TavilySearch
search = TavilySearch(
max_results=5, tavily_api_key="your-siftdog-key", api_base_url="http://localhost:8000"
)
search.invoke({"query": "what is BM25 ranking"})
Tested with tavily-python 0.8.4 (search, extract, crawl, map; sync and async) and
langchain-tavily 0.2.18 (search, extract).
Quick start
Docker Compose (includes SearXNG)
The full stack, with a SearXNG instance for /search, using the published image:
git clone https://github.com/khsarvar/siftdog && cd siftdog
cp .env.example .env # optional: set API_KEYS and ANTHROPIC_API_KEY
docker compose up
Try a search:
curl -s localhost:8000/search -H "Content-Type: application/json" \
-d '{"query": "latest python release", "search_depth": "advanced"}'
If you set API_KEYS, add -H "Authorization: Bearer <key>". OpenAPI docs are served at
http://localhost:8000/docs.
Docker image only
docker run -p 8000:8000 ghcr.io/khsarvar/siftdog
/extract, /crawl and /map work on their own. For /search, point it at a SearXNG
instance with the JSON format enabled: -e SEARXNG_URL=http://your-searxng:8080. Images are
published for linux/amd64 and linux/arm64, tagged latest and by version (0.1.0, 0.1).
pip
pip install siftdog
SEARXNG_URL=http://your-searxng:8080 siftdog --port 8000
Configuration is read from environment variables or a .env file (see
Configuration).
Use with MCP clients (Claude, Cursor, ...)
Siftdog is also an MCP server with four read-only tools:
siftdog_search, siftdog_extract, siftdog_crawl and siftdog_map.
Connect to a running Siftdog server (Streamable HTTP at /mcp). With Claude Code:
claude mcp add --transport http siftdog http://localhost:8000/mcp \
--header "Authorization: Bearer <key>" # omit the header if API_KEYS is empty
Or run it locally over stdio with uv, no server needed.
SEARXNG_URL is only needed for siftdog_search:
claude mcp add siftdog -e SEARXNG_URL=http://your-searxng:8080 -- uvx siftdog mcp
Claude Desktop (claude_desktop_config.json), Cursor (.cursor/mcp.json) and most other clients
take the same command as JSON:
{
"mcpServers": {
"siftdog": {
"command": "uvx",
"args": ["siftdog", "mcp"],
"env": { "SEARXNG_URL": "http://your-searxng:8080" }
}
}
}
API keys work as for the REST API: send Authorization: Bearer <key>, or append
?api_key=<key> to the URL for clients that can't set headers (URLs can end up in logs, so
prefer the header). The HTTP endpoint only answers requests addressed to localhost unless you
list your hostname in MCP_ALLOWED_HOSTS.
Siftdog vs Tavily, Firecrawl and crw
| Siftdog | Tavily | Firecrawl | crw | |
|---|---|---|---|---|
| License | MIT | Proprietary (hosted service) | AGPL-3.0 | AGPL-3.0 |
| Self-hosted | Yes | No | Yes | Yes (also a managed API) |
| Tavily-compatible API | Yes | — | No (own API) | No (own API) |
| Search source | SearXNG metasearch | Proprietary | — | SearXNG (bundled) |
| JavaScript rendering | No (static HTML) | — | Yes | Yes (Lightpanda, Chrome fallback) |
| MCP server | Yes (HTTP and stdio) | Yes | Yes | Yes |
| Language | Python | — | TypeScript | Rust |
Measured results: Siftdog vs Tavily search benchmark
(reproducible with bench/).
When to pick something else: if you'd rather not run infrastructure, or you want Tavily's neural reranking, use hosted Tavily. If you need JavaScript-rendered pages or a scraping platform with more features, look at Firecrawl or crw. Siftdog is for teams that want the Tavily API on their own servers under a permissive license.
FAQ
What is Siftdog?
Siftdog is an open-source web search and extraction API for AI agents. It reproduces the Tavily
API (/search, /extract, /crawl, /map) on infrastructure you run yourself, using SearXNG
for search results, trafilatura for content extraction and BM25 for relevance ranking.
Is Siftdog a drop-in replacement for Tavily?
For the four core endpoints, yes. The request and response fields match Tavily's, and the
official tavily-python SDK and langchain-tavily work by setting api_base_url. The
differences: relevance scores come from BM25 rather than a neural reranker, instructions
(crawl/map) and include_image_descriptions (search) are accepted but ignored, and Tavily's
/research endpoint isn't implemented.
Do I need a search API key?
No. Search results come from SearXNG, which queries public search engines. The only optional
key is ANTHROPIC_API_KEY, used when a request sets include_answer.
Does it work with LangChain?
Yes, through the official langchain-tavily package. Pass api_base_url pointing at your
Siftdog server, as in the example above.
Does Siftdog have an MCP server?
Yes. The API server exposes MCP over Streamable HTTP at /mcp, and uvx siftdog mcp runs it
over stdio for local clients such as Claude Desktop and Cursor. See
Use with MCP clients.
Why does /search return 502 "failing engines"?
SearXNG gets its results by querying public search engines (Brave, DuckDuckGo, Google and
others), and those engines rate-limit or CAPTCHA an IP that sends many searches. When every
engine is failing, Siftdog returns 502 with the engines and reasons, for example
failing engines: brave (Suspended: too many requests), duckduckgo (CAPTCHA), rather than an
empty result list your agent would mistake for "nothing found". Engines recover on their own,
from minutes to about a day. If some engines still work, you get their results and the server
logs which engines were down. To reduce blocking, enable more engines in
docker/searxng/settings.yml and avoid bursts of identical searches.
Is it safe to expose Siftdog on a public server?
Set API_KEYS so only your clients can call it. /extract and /crawl fetch caller-supplied
URLs, so Siftdog refuses private, loopback and link-local addresses, checked on every redirect and
at connect time against the exact address used, which also stops DNS rebinding.
Can I run it without Docker?
Yes: pip install siftdog, then run siftdog with SEARXNG_URL pointing at any SearXNG
instance with the JSON output format enabled. /extract, /crawl and /map work without
SearXNG.
Local development
python3 -m venv .venv && .venv/bin/pip install -e ".[dev]"
docker compose up searxng -d # uncomment its `ports:` in docker-compose.yml first
.venv/bin/siftdog # http://127.0.0.1:8000, docs at /docs
.venv/bin/pytest # offline test suite
.venv/bin/ruff check . && .venv/bin/ruff format .
API
All endpoints take JSON POST bodies and a Authorization: Bearer <key> header (a legacy
api_key body field is also accepted). If API_KEYS is empty, auth is disabled.
| Endpoint | Purpose | Key parameters |
|---|---|---|
/search |
Web search | query, search_depth (basic/advanced), topic (general/news), time_range, max_results, chunks_per_source, include_answer, include_raw_content, include_images, include_domains, exclude_domains |
/extract |
Clean content from up to 20 URLs | urls, extract_depth, format (markdown/text), include_images |
/crawl |
Crawl a site and return page content | url, max_depth, max_breadth, limit, select_paths, exclude_paths, select_domains, exclude_domains, allow_external, format |
/map |
List a site's URLs without content | same traversal parameters as /crawl |
Differences from hosted Tavily: instructions (crawl/map) and include_image_descriptions are
accepted but ignored; relevance scores come from BM25, not a neural reranker.
Configuration
Environment variables (see .env.example): API_KEYS, SEARXNG_URL, ANSWER_ENABLED,
ANSWER_MODEL, ANSWER_EFFORT, FETCH_TIMEOUT, FETCH_CONCURRENCY, FETCH_MAX_BYTES,
ALLOW_PRIVATE_NETWORKS, CRAWL_MAX_LIMIT, MCP_ALLOWED_HOSTS.
MCP_ALLOWED_HOSTS lists the hostnames the /mcp endpoint answers besides localhost, for
example search.example.com,search.example.com:*. Requests addressed to any other host get
421, which protects a local server from DNS-rebinding attacks by web pages.
Security: /extract and /crawl make the server fetch caller-supplied URLs. Requests to
private, loopback and link-local addresses are blocked (including via redirects) unless
ALLOW_PRIVATE_NETWORKS=true. The check runs at connect time against the exact address being
connected to, so DNS rebinding can't get around it. Fetches of user URLs ignore
HTTP(S)_PROXY, since a proxy would hide the destination address. As defense in depth for
hostile multi-tenant deployments, also restrict egress at the network level.
License
Siftdog is released under the MIT License. The "Siftdog" name is covered separately by the trademark policy: use the code freely, but forks and hosted services need a different name.
Siftdog MCP questions
How do I add Siftdog MCP to Claude Code?
Run claude mcp add siftdog -- uvx siftdog mcp in your terminal. Add --scope user to make it available in every project.
How do I add Siftdog MCP to Cursor?
Open ~/.cursor/mcp.json and add the server under mcpServers, using the Cursor config shown in the install section above.
What tools does Siftdog MCP provide?
It exposes 4 tools, including siftdog_search, siftdog_extract, siftdog_crawl, siftdog_map.
Is Siftdog a drop-in replacement for Tavily?
Yes, for the four core endpoints, as the request and response fields match Tavily's and it works with official SDKs and LangChain.
Do I need a search API key to use Siftdog?
No, search results are obtained through a SearXNG metasearch instance which queries public search engines.