websearch MCP server
Commit Oct 31, 2026Performs multi-engine web searches, fetches content, and controls a browser to provide synthesized, grounded research answers.
Also known as SK-DEV-AI/websearch, skdev-websearch-mcp
- Stars
- 1
- Tools
- 18
- Downloads, 30 days
- 759
- Latest version
- 1.10.0
- Last commit
- Oct 31, 2026
- License
- MIT
What is websearch MCP?
websearch performs multi-engine web searches, fetches content, and controls a browser to provide synthesized, grounded research answers. It exposes 18 tools, such as doctor, ping, search.
Summary generated from the project's README
About websearch MCP server
Researchers and developers can perform deep, multi-engine web searches and automated browser interactions through this MCP server. It provides a wide array of tools for gathering information, including parallel searches across DuckDuckGo, Bing, Tavily, Wikipedia, and arXiv, as well as specialized vertical lanes for GitHub, Stack Exchange, and MDN. Users can fetch content as markdown or text, crawl entire documentation sites, or use a Chromium-based browser to handle JavaScript-heavy pages and interactive workflows.
The server is designed for local use, requiring a browser for persistent sessions and an optional OpenAI-compatible endpoint for in-tool distillation. It features a sophisticated synthesis engine that uses Groq to generate grounded answers, ensuring every claim is backed by a cited source. To manage large amounts of data, the fetch tool can compress long pages using a local model, while the browser_session tool allows for maintaining logins and cookies across multiple steps. It includes safety features like path traversal protection for downloads and strict adherence to robots.txt.
What you can do with it
- 1Deep research with citationsAsk complex research questions to receive a synthesized answer that explicitly identifies which source supports every claim.
- 2Large scale site crawlingRead through an entire documentation website by performing a breadth-first crawl with defined depth and page limits.
- 3Data extraction from tablesConvert HTML tables found on a webpage directly into structured JSON, CSV, or markdown formats.
Written from the project's README. Every use case is checked against a quote from it.
How to install websearch MCP
Formats verified 10 Nov 2026Add to CursorAdd to GooseInstall websearch MCP in Claude Code
$ claude mcp add --env 'GROQ_API_KEYS=<YOUR_GROQ_API_KEYS>' --env 'NV_KEY=<YOUR_NV_KEY>' --env 'S2_KEY=<YOUR_S2_KEY>' --env 'SE_KEY=<YOUR_SE_KEY>' --env 'TAVILY_KEYS=<YOUR_TAVILY_KEYS>' websearch -- uvx --from skdev-websearch-mcp websearch-mcpInstall websearch MCP in Cursor
~/.cursor/mcp.json
{
"mcpServers": {
"websearch": {
"command": "uvx",
"args": [
"--from",
"skdev-websearch-mcp",
"websearch-mcp"
],
"env": {
"GROQ_API_KEYS": "<YOUR_GROQ_API_KEYS>",
"NV_KEY": "<YOUR_NV_KEY>",
"S2_KEY": "<YOUR_S2_KEY>",
"SE_KEY": "<YOUR_SE_KEY>",
"TAVILY_KEYS": "<YOUR_TAVILY_KEYS>"
}
}
}
}Install websearch MCP in VS Code
.vscode/mcp.json
{
"servers": {
"websearch": {
"type": "stdio",
"command": "uvx",
"args": [
"--from",
"skdev-websearch-mcp",
"websearch-mcp"
],
"env": {
"GROQ_API_KEYS": "<YOUR_GROQ_API_KEYS>",
"NV_KEY": "<YOUR_NV_KEY>",
"S2_KEY": "<YOUR_S2_KEY>",
"SE_KEY": "<YOUR_SE_KEY>",
"TAVILY_KEYS": "<YOUR_TAVILY_KEYS>"
}
}
}
}Install websearch MCP in VS Code CLI
$ code --add-mcp '{"name":"websearch","type":"stdio","command":"uvx","args":["--from","skdev-websearch-mcp","websearch-mcp"],"env":{"GROQ_API_KEYS":"<YOUR_GROQ_API_KEYS>","NV_KEY":"<YOUR_NV_KEY>","S2_KEY":"<YOUR_S2_KEY>","SE_KEY":"<YOUR_SE_KEY>","TAVILY_KEYS":"<YOUR_TAVILY_KEYS>"}}'Install websearch MCP in Claude Desktop
claude_desktop_config.json
{
"mcpServers": {
"websearch": {
"command": "uvx",
"args": [
"--from",
"skdev-websearch-mcp",
"websearch-mcp"
],
"env": {
"GROQ_API_KEYS": "<YOUR_GROQ_API_KEYS>",
"NV_KEY": "<YOUR_NV_KEY>",
"S2_KEY": "<YOUR_S2_KEY>",
"SE_KEY": "<YOUR_SE_KEY>",
"TAVILY_KEYS": "<YOUR_TAVILY_KEYS>"
}
}
}
}Install websearch MCP in Codex
$ codex mcp add websearch --env 'GROQ_API_KEYS=<YOUR_GROQ_API_KEYS>' --env 'NV_KEY=<YOUR_NV_KEY>' --env 'S2_KEY=<YOUR_S2_KEY>' --env 'SE_KEY=<YOUR_SE_KEY>' --env 'TAVILY_KEYS=<YOUR_TAVILY_KEYS>' -- uvx --from skdev-websearch-mcp websearch-mcpInstall websearch MCP in Codex config.toml
~/.codex/config.toml
[mcp_servers.websearch] command = "uvx" args = ["--from", "skdev-websearch-mcp", "websearch-mcp"] [mcp_servers.websearch.env] GROQ_API_KEYS = "<YOUR_GROQ_API_KEYS>" NV_KEY = "<YOUR_NV_KEY>" S2_KEY = "<YOUR_S2_KEY>" SE_KEY = "<YOUR_SE_KEY>" TAVILY_KEYS = "<YOUR_TAVILY_KEYS>"
Install websearch MCP in Gemini CLI
$ gemini mcp add -s user -e 'GROQ_API_KEYS=<YOUR_GROQ_API_KEYS>' -e 'NV_KEY=<YOUR_NV_KEY>' -e 'S2_KEY=<YOUR_S2_KEY>' -e 'SE_KEY=<YOUR_SE_KEY>' -e 'TAVILY_KEYS=<YOUR_TAVILY_KEYS>' websearch uvx -- --from skdev-websearch-mcp websearch-mcpInstall websearch MCP in Gemini settings.json
~/.gemini/settings.json
{
"mcpServers": {
"websearch": {
"command": "uvx",
"args": [
"--from",
"skdev-websearch-mcp",
"websearch-mcp"
],
"env": {
"GROQ_API_KEYS": "<YOUR_GROQ_API_KEYS>",
"NV_KEY": "<YOUR_NV_KEY>",
"S2_KEY": "<YOUR_S2_KEY>",
"SE_KEY": "<YOUR_SE_KEY>",
"TAVILY_KEYS": "<YOUR_TAVILY_KEYS>"
}
}
}
}Started by HostedMCP on 16 Nov 2026
We ran this command for skdev-websearch-mcp 1.10.0 in an isolated container with no network and no keys. It answered and listed 18 tools.
websearch MCP tools 18
Read from the server with tools/listdoctorPreflight: report which lanes are usable RIGHT NOW and the exact command to enable the rest. Run this first, and whenever a feature returns empty results. Distinguishes 'broken' from 'needs optional setup' — the browser lanes (Google AI Mode, browser_action, bot-wall bypass) require a Chromium-based browser with --remote-debugging-port on the same host, so they are unavailable on headless servers; the keyless HTTP lanes work anywhere. Never prints key values, only which env var names are set. e.g. doctor()
pingLightweight connectivity check — verifies internet and key search endpoints are reachable. Use before expensive calls when connectivity is uncertain. No params needed. e.g. ping()
searchMulti-engine web search with dedup, reranking, and synthesis. Use for open research questions; for a known URL use fetch, for papers use arxiv, for encyclopedic facts use wikipedia. depth=1 returns snippets with relevance_score+fetch_relevance per result. depth>=2 fetches full pages + re-ranks. synthesize=True (default) returns Groq answer with [N] citations. google_ai_only skips all engines for Google AI Mode. e.g. search(query='latest AI models', depth=1)
fetchURL to markdown/text. Auto-fallback: direct fetch → CDP for blocked/JS pages. Supports PDF, EPUB, DOCX. SSRF-protected. Use focus="query" to filter content by relevance. Paginated via offset (response: next_offset). Results cached 1h; cache_ttl=0 fresh. For structured JSON extraction use `extract` instead. e.g. fetch(url='https://example.com')
screenshotARIA accessibility snapshot (LLM-readable page text). Use to see a page's interactive structure and what is actually visible; for readable article content use fetch. e.g. screenshot(url='https://example.com')
browser_actionDrive a real browser page (needs a Chromium-based browser with --remote-debugging-port on this host, e.g. chromium/google-chrome/brave/Helium, set via WEBSEARCH_CDP_URL; run doctor() to check): navigate, click, type, press a key, evaluate JS, read page HTML/text, get the accessibility tree, or capture a PNG. Use when a page needs interaction (login form, search box, 'load more' button) — for plain reading use fetch, for structure use screenshot. One call = one fresh tab, closed afterwards; pass steps[] to run a multi-step flow (focus -> type -> Enter -> read) in a single tab. URLs are SSRF-validated. e.g. browser_action(action='click', url='https://example.com', selector='#more')
browser_sessionPersistent browser tab you can reuse across calls (needs a Chromium with --remote-debugging-port; run doctor()). Without a session every browser_action call gets a fresh tab that is closed immediately, so logins, cookies and multi-step flows cannot survive. Create a session, then pass session_id to browser_action. action=create|list|info|close. Sessions are capped (8) and the least recently used is evicted; close when done so no tab leaks. e.g. browser_session(action='create', label='login', url='https://example.com') -> then browser_action(action='click', selector='#x', session_id='...')
fetch_manyFetch several URLs in one call. Replaces a loop of fetch calls or a throwaway requests script: dedups URLs, caps concurrency, honours robots.txt, shares the cache, and returns per-URL success/status/content plus counts. Reports `truncated` when the batch cap (50) is hit so you know to call again for the rest. Use when you have 2+ URLs; use fetch for a single known page.
crawlBreadth-first crawl of a site with hard caps. Replaces a bespoke scraper and makes politeness non-optional: per-page cap, depth cap, robots.txt, dedup, concurrency limit. Returns each page's title and trimmed content plus a stop_reason so you know whether the site was exhausted or a cap stopped it. Use for 'read this whole docs site'; for a known list of URLs use fetch_many.
extract_linksEvery link on a page with its anchor text, title and rel. Replaces hand-parsing HTML before crawling or finding a 'next' button. Resolves relative hrefs, drops javascript:/mailto:, dedups, and can stay on one domain. Use before crawl or to locate a specific navigation target.
extract_tablePull HTML tables into json, csv or markdown. Replaces a pandas/lxml script every time a page has a pricing or stats table. Reports tables_found so you can pick one by index, and marks truncated when max_rows is hit. Use extract(css) for non-tabular structure instead.
downloadSave a URL to disk and return the local path, byte count, sha256 and content type. Replaces curl/requests plus a filename guess. Writes only inside dest_dir (default ~/.cache/websearch/downloads) and refuses path traversal. Marks truncated if the byte cap stopped the transfer, so you know the file is incomplete.
| doctor | Preflight: report which lanes are usable RIGHT NOW and the exact command to enable the rest. Run this first, and whenever a feature returns empty results. Distinguishes 'broken' from 'needs optional setup' — the browser lanes (Google AI Mode, browser_action, bot-wall bypass) require a Chromium-based browser with --remote-debugging-port on the same host, so they are unavailable on headless servers; the keyless HTTP lanes work anywhere. Never prints key values, only which env var names are set. e.g. doctor() | |
| ping | Lightweight connectivity check — verifies internet and key search endpoints are reachable. Use before expensive calls when connectivity is uncertain. No params needed. e.g. ping() | |
| search | Multi-engine web search with dedup, reranking, and synthesis. Use for open research questions; for a known URL use fetch, for papers use arxiv, for encyclopedic facts use wikipedia. depth=1 returns snippets with relevance_score+fetch_relevance per result. depth>=2 fetches full pages + re-ranks. synthesize=True (default) returns Groq answer with [N] citations. google_ai_only skips all engines for Google AI Mode. e.g. search(query='latest AI models', depth=1) | |
| fetch | URL to markdown/text. Auto-fallback: direct fetch → CDP for blocked/JS pages. Supports PDF, EPUB, DOCX. SSRF-protected. Use focus="query" to filter content by relevance. Paginated via offset (response: next_offset). Results cached 1h; cache_ttl=0 fresh. For structured JSON extraction use `extract` instead. e.g. fetch(url='https://example.com') | |
| screenshot | ARIA accessibility snapshot (LLM-readable page text). Use to see a page's interactive structure and what is actually visible; for readable article content use fetch. e.g. screenshot(url='https://example.com') | |
| browser_action | Drive a real browser page (needs a Chromium-based browser with --remote-debugging-port on this host, e.g. chromium/google-chrome/brave/Helium, set via WEBSEARCH_CDP_URL; run doctor() to check): navigate, click, type, press a key, evaluate JS, read page HTML/text, get the accessibility tree, or capture a PNG. Use when a page needs interaction (login form, search box, 'load more' button) — for plain reading use fetch, for structure use screenshot. One call = one fresh tab, closed afterwards; pass steps[] to run a multi-step flow (focus -> type -> Enter -> read) in a single tab. URLs are SSRF-validated. e.g. browser_action(action='click', url='https://example.com', selector='#more') | |
| browser_session | Persistent browser tab you can reuse across calls (needs a Chromium with --remote-debugging-port; run doctor()). Without a session every browser_action call gets a fresh tab that is closed immediately, so logins, cookies and multi-step flows cannot survive. Create a session, then pass session_id to browser_action. action=create|list|info|close. Sessions are capped (8) and the least recently used is evicted; close when done so no tab leaks. e.g. browser_session(action='create', label='login', url='https://example.com') -> then browser_action(action='click', selector='#x', session_id='...') | |
| fetch_many | Fetch several URLs in one call. Replaces a loop of fetch calls or a throwaway requests script: dedups URLs, caps concurrency, honours robots.txt, shares the cache, and returns per-URL success/status/content plus counts. Reports `truncated` when the batch cap (50) is hit so you know to call again for the rest. Use when you have 2+ URLs; use fetch for a single known page. | |
| crawl | Breadth-first crawl of a site with hard caps. Replaces a bespoke scraper and makes politeness non-optional: per-page cap, depth cap, robots.txt, dedup, concurrency limit. Returns each page's title and trimmed content plus a stop_reason so you know whether the site was exhausted or a cap stopped it. Use for 'read this whole docs site'; for a known list of URLs use fetch_many. | |
| extract_links | Every link on a page with its anchor text, title and rel. Replaces hand-parsing HTML before crawling or finding a 'next' button. Resolves relative hrefs, drops javascript:/mailto:, dedups, and can stay on one domain. Use before crawl or to locate a specific navigation target. | |
| extract_table | Pull HTML tables into json, csv or markdown. Replaces a pandas/lxml script every time a page has a pricing or stats table. Reports tables_found so you can pick one by index, and marks truncated when max_rows is hit. Use extract(css) for non-tabular structure instead. | |
| download | Save a URL to disk and return the local path, byte count, sha256 and content type. Replaces curl/requests plus a filename guess. Writes only inside dest_dir (default ~/.cache/websearch/downloads) and refuses path traversal. Marks truncated if the byte cap stopped the transfer, so you know the file is incomplete. |
Show all 18 toolsShow fewer tools
diff_pageHas this page changed since the last time it was fetched? Compares the live page against the cached snapshot and returns changed/old_hash/new_hash plus the added and removed lines. Replaces a hand-rolled polling and hashing script. Returns had_previous=false on the first call for a URL: fetch it, then diff later. Useful for tracking docs, pricing and changelogs.
wikipediaFull Wikipedia API client: articles, summaries, categories, links, pageviews, revisions, backlinks, geosearch, recent changes, random pages. Use for encyclopedic facts; for open research use search. e.g. wikipedia(query='Python', action='summary')
arxivSearch arXiv academic papers (single-source; for multi-source paper search use codesearch papers). Use raw_query for boolean operators (AND, OR, ANDNOT), phrase search (ti:"exact phrase"), wildcards (au:smith*). e.g. arxiv(query='cs.AI transformer', count=5)
map_siteDiscover all pages on a website via sitemap XML (primary) and HTML link extraction (fallback). Use to survey a site's pages before fetching individually. Returns the base domain, source type, total count, and an array of discovered URLs with metadata (last_modified, priority, changefreq when available). e.g. map_site(url='https://example.com')
extractExtract structured JSON from a webpage using LLM, CSS, or regex strategies. Persistent cache — repeat calls free. For LLM strategy, describe what you want and get clean JSON back. For CSS strategy, provide field names (fastest, free). For raw page content, use `fetch` instead. Examples: extract(url='...', instruction='extract product name and price') or extract(url='...', fields=['name','price'], strategy='css')
pdf_extractPDF extraction: PyMuPDF fast text-layer path, Docling+OCR fallback for scanned/image-only PDFs (CPU). Formats: markdown (default), json, html, or comma combos like 'markdown,json'. e.g. pdf_extract(input_path='/path/to/doc.pdf', format='markdown')
| diff_page | Has this page changed since the last time it was fetched? Compares the live page against the cached snapshot and returns changed/old_hash/new_hash plus the added and removed lines. Replaces a hand-rolled polling and hashing script. Returns had_previous=false on the first call for a URL: fetch it, then diff later. Useful for tracking docs, pricing and changelogs. | |
| wikipedia | Full Wikipedia API client: articles, summaries, categories, links, pageviews, revisions, backlinks, geosearch, recent changes, random pages. Use for encyclopedic facts; for open research use search. e.g. wikipedia(query='Python', action='summary') | |
| arxiv | Search arXiv academic papers (single-source; for multi-source paper search use codesearch papers). Use raw_query for boolean operators (AND, OR, ANDNOT), phrase search (ti:"exact phrase"), wildcards (au:smith*). e.g. arxiv(query='cs.AI transformer', count=5) | |
| map_site | Discover all pages on a website via sitemap XML (primary) and HTML link extraction (fallback). Use to survey a site's pages before fetching individually. Returns the base domain, source type, total count, and an array of discovered URLs with metadata (last_modified, priority, changefreq when available). e.g. map_site(url='https://example.com') | |
| extract | Extract structured JSON from a webpage using LLM, CSS, or regex strategies. Persistent cache — repeat calls free. For LLM strategy, describe what you want and get clean JSON back. For CSS strategy, provide field names (fastest, free). For raw page content, use `fetch` instead. Examples: extract(url='...', instruction='extract product name and price') or extract(url='...', fields=['name','price'], strategy='css') | |
| pdf_extract | PDF extraction: PyMuPDF fast text-layer path, Docling+OCR fallback for scanned/image-only PDFs (CPU). Formats: markdown (default), json, html, or comma combos like 'markdown,json'. e.g. pdf_extract(input_path='/path/to/doc.pdf', format='markdown') |
Configuration
GROQ_API_KEYSSecretOptional. Groq keys for synthesis and query rewriting.
NV_KEYSecretOptional. NVIDIA embeddings for ranking.
S2_KEYSecretOptional. Semantic Scholar key (1 RPS).
SE_KEYSecretOptional. Stack Exchange key (10k/day instead of 300).
TAVILY_KEYSSecretOptional. Tavily search lane.
| Name | Description | |
|---|---|---|
| GROQ_API_KEYS | Optional. Groq keys for synthesis and query rewriting. | Secret |
| NV_KEY | Optional. NVIDIA embeddings for ranking. | Secret |
| S2_KEY | Optional. Semantic Scholar key (1 RPS). | Secret |
| SE_KEY | Optional. Stack Exchange key (10k/day instead of 300). | Secret |
| TAVILY_KEYS | Optional. Tavily search lane. | Secret |
Automated checks
Ran Nov 12, 2026Full report →Not verified yet. A server needs a score of 80, no serious finding, and enough data for us to check. How checks work · What to fix for more points
- Source repository linked and reachablePass
- Open source license (OSI)Pass
- Commit or release in the last yearPass
- Package name not close to a popular packagePass
- Remote endpoint uses HTTPSNot checked
- Destructive tools are labelledPass
- No hidden instructions to the modelInstruction aimed at the model found in the description or READMEFinding
- No critical advisory in the package or its dependenciesPass
README
View on GitHubwebsearch MCP Server
mcp-name: io.github.SK-DEV-AI/websearch
Companion agent skills: SK-DEV-AI/skills · npx skills add SK-DEV-AI/skills
Multi-engine web search + fetch MCP server. Parallel fan-out across DuckDuckGo, Bing Web/News RSS, Tavily, Wikipedia, arXiv, Google News RSS, Brave (last resort — rate-limited from some networks), Google AI Mode, and intent-gated vertical lanes (MDN, Stack Exchange, GitHub, HN, Semantic Scholar, OSV vuln, grep.app/Sourcegraph code, GDELT news) — with consensus-merge dedup, a local cross-encoder reranker, and Groq synthesis with grounding checks.
Tools (20)
- doctor — Run this first. Reports which lanes are usable right now and the exact command to enable the rest. Distinguishes "broken" from "needs optional setup". Never prints key values.
- search — Open research questions.
depthdefaults to 3 when a model lane is configured (more independent sources; measured) and 1 otherwise. Depth also sets how many pages are fetched:max(5, depth*2), ceiling 8.synthesizereturns a short, cited answer that never invents: it says which source backs each claim, surfaces disagreement instead of averaging it, and marks itselftruncatedwhen it ran out of budget. - fetch — URL to markdown/text. Direct fetch with auto-fallback to a Chromium CDP browser for blocked/JS pages. PDF/EPUB/DOCX supported. Paginated (
offset). Optionaldistill(see below). For structured JSON useextract. - fetch_many — Fetch several URLs in one call. Dedups, caps concurrency, honours robots.txt, shares the cache, returns per-URL success/status/content plus counts. Use when you have 2+ URLs; use
fetchfor one. - crawl — Breadth-first site crawl with hard caps (per-page cap, depth cap, robots.txt, dedup, concurrency). Returns each page plus a
stop_reasonso you know whether the site was exhausted or a cap stopped it. Use for "read this whole docs site". - extract_links — Every link on a page with anchor text, title and rel. Resolves relative hrefs, drops
javascript:/mailto:, dedups. Use beforecrawlor to find a "next" button. - extract_table — HTML tables to json, csv or markdown, by selector and index. Reports
tables_foundso you can pick one. - download — Save a URL to disk; returns the local path, byte count, sha256 and content type. Writes only inside
dest_dirand refuses path traversal. Markstruncatedif the byte cap stopped the transfer. - diff_page — Has this page changed since it was cached? Returns changed/old_hash/new_hash plus added and removed lines. First call for a URL returns
had_previous: false. - screenshot — ARIA accessibility snapshot (LLM-readable page text and interactive structure).
- browser_action — Drive a real browser page (navigate, click, type, press a key, evaluate JS, read HTML/text, ARIA tree, screenshot). Use when a page needs interaction; for plain reading use
fetch. Passsession_idto reuse a persistent tab. - browser_session — Create, list, inspect or close a persistent browser tab. Without one, every
browser_actioncall gets a fresh tab and closes it, so logins and cookies cannot survive. See below. - form_fill_submit — Fill many fields, submit, and read the result in one call. Verifies each element exists, reads back each typed value, and refuses to submit if a value did not stick. Reports
effect(navigated/content_changed/none) so a submit that silently did nothing is visible rather than assumed. - paginate — Click a "next" control, collecting a value per page, up to a cap. Stops honestly when the control is missing or has no effect, and reports
stop_reason. - wikipedia — Full Wikipedia API client: articles, summaries, categories, links, pageviews, revisions, backlinks, geosearch, recent changes, random pages.
- arxiv — arXiv paper search with boolean/phrase/wildcard query syntax (single-source; multi-source paper search lives in codesearch
papers). - extract — Structured JSON from a page via CSS (fast, free), LLM, or regex strategies. Repeat calls cached.
- pdf_extract — PDF to markdown/json/html. PyMuPDF fast path for text-layer PDFs, Docling+OCR fallback for scanned PDFs.
- map_site — Survey a site's pages via sitemap XML (primary) + HTML link fallback, before fetching individually.
- ping — Connectivity check for the internet and key search endpoints. Use before expensive calls when uncertain.
What replaced the throwaway scripts
Every one of these exists because an agent otherwise writes a one-off script:
| The task | The tool |
|---|---|
| Fetch 20 URLs | fetch_many |
| Read a whole docs site | crawl |
| Get every link on a page | extract_links |
| Pull a pricing table into CSV | extract_table |
| Save a PDF and get its path | download |
| "Did this page change?" | diff_page |
| Log in, then act across several calls | browser_session + browser_action(session_id=...) |
| Fill and submit a form | form_fill_submit |
| Walk "next" through pages | paginate |
Persistent browser sessions
browser_action opens one fresh tab per call and closes it afterwards. That is
right for a single read, but it makes any interactive workflow impossible: no
login, no cookies, no scroll position, no multi-step flow across calls.
A session is a named, long-lived tab:
browser_session(action="create", label="login", url="https://site.example/login")
# -> {"session_id": "login-1a2b3c4d", ...}
browser_action(action="type", selector="#email", text="me@example.com", session_id="login-1a2b3c4d")
browser_action(action="click", selector="#submit", session_id="login-1a2b3c4d")
browser_action(action="text", session_id="login-1a2b3c4d") # same tab, still logged in
browser_session(action="close", session_id="login-1a2b3c4d")
Sessions are opt-in: without session_id, behaviour is exactly as before. They
are capped at 8 with least-recently-used eviction, tabs are always closed
(including on eviction), and metadata is mirrored to
~/.cache/websearch/sessions.json so list can explain what survived a
restart. Sessions need a browser, so they are local-only.
In-tool distillation (optional, off by default)
A fetched page can be 50k characters. The calling model pays for all of it even
when it needed one fact. fetch(distill=true) runs a small model inside the
tool first and returns a compressed result:
fetch(url="https://example.com/long-report", distill=True,
distill_question="What are the pricing tiers?")
The result never pretends to be the page:
| Field | Meaning |
|---|---|
distill.distilled |
whether a summary was produced |
distill.raw_chars |
size of the original text |
distill.grounded |
whether every number and quoted span was found in the source |
distill.grounding.numbers_unverified |
figures in the summary that do not appear in the source |
distill.grounding.quotes_unverified |
quoted spans that are not verbatim |
distill.how_to_get_raw |
how to get the untouched text: distill=false |
Below the size threshold (default 4000 chars) distillation is skipped and the
raw text is returned, because a summary would cost more than it saves. On any
failure the raw text is returned with distill_skipped explaining why - a
broken summary is never returned in place of the page.
| Variable | Default | Meaning |
|---|---|---|
DISTILL_BASE_URL |
http://127.0.0.1:11435/v1 |
any OpenAI-compatible endpoint |
DISTILL_MODEL |
reflection/beam:free |
model id |
DISTILL_API_KEY |
none | bearer token; without it the lane stays off |
DISTILL_RETRIES |
3 |
attempts per call (1-10) |
DISTILL_MAX_TOKENS |
3000 |
initial completion budget |
DISTILL_MAX_TOKENS_CEILING |
12000 |
ceiling when escalating after empty content |
DISTILL_TIMEOUT |
90 |
seconds per request |
DISTILL_THRESHOLD_CHARS |
4000 |
below this, skip distillation |
The synthesis prompt (the answer you actually read)
search does not just rank results; it writes the answer. That answer is
generated from the retrieved text alone, under rules that exist to stop the
three failure modes of a search summary:
| Failure | What the prompt does |
|---|---|
| Invention | Nothing from the model's own knowledge. A claim with no citable source is deleted, not softened. If the results do not answer the question, saying so is the correct answer. |
| Bias | Rank is not evidence. The prompt says so explicitly, caps how many sources can come from one domain, requires disagreement between sources to be reported rather than averaged into a number that appears in no source, corrects false premises in the question, and labels vendor claims as vendor claims. |
| Injection | Results are wrapped in <results> and declared untrusted data. A page that says "ignore previous instructions" is data, not instruction; the model must exclude it and say it did. |
Concision is structural, not a wish: a sentence ceiling, an answer-first rule, and a named ban on filler phrasing ("it is important to note", "overall", ...).
Everything is verifiable rather than claimed:
- the result carries
prompt_version, so a quality change can be tied to a revision instead of guessed at temperatureis 0, so the same sources give the same textfinish_reasonandtruncatedare returned; a half sentence is labelled instead of looking like a complete thoughtdiversityreports the source mix and whether the per-domain cap had to relaxas_ofreports the newest publication date among the sources, and undated sources are labelled "Published: not stated", so freshness is never impliedduplicates_removedcounts syndicated copies collapsed before synthesis, andfigures_unverifiedlists numbers the answer contains that appear in no cited source (deterministic, no model call)unusable_sourcesnames the results that could not become evidence - a 404, an error page, or a page with under 400 chars of text - with the reason, andunreadable_sourcescounts the ones with no text at all- multi-hop: when the first answer admits a weakness (single source, a claim
it could not tie to a source, or a stated gap), one extra round runs on the
subquery that targets the gap. The response records
follow_upsand markssynthesis.round = 2. Bounded: once per call, only under a 75s budget, andfollow_up=falsedisables it - a second model pass (
grounded) re-reads the answer against the exact text the first pass saw and flags sentences no cited source supports. It is markedadvisorybecause it is another model's opinion, not a proof
Tool annotations
All 20 tools carry MCP annotations (readOnlyHint, destructiveHint,
idempotentHint, openWorldHint) so a client can tell a read from an action:
search and fetch are read-only, download writes a local file without being
destructive, browser_session changes local browser state only, and
browser_action / form_fill_submit are marked destructive because a click or a
form submit can have an effect that cannot be undone from here.
When a call takes too long (deadline, admission, partials)
MCP clients cut a tool call at their own timeout (jcode: 300 s) and report a
bare Request timeout after 300s with no payload. Measured over one day on a
single busy server: 314 calls, median 2.7 s, p90 123 s, max 225 s, and 33 hard
client timeouts — the tail is the problem, not the median. So the server bounds
its own work instead of hoping the client waits:
| Env | Default | Meaning |
|---|---|---|
WEBSEARCH_TOOL_DEADLINE |
240 |
Seconds one tool call may run. Keep it under the client timeout. 0 disables the bound |
WEBSEARCH_MAX_CONCURRENT |
8 |
Simultaneous tool calls. Extra callers wait, then get busy. 0 = unlimited |
WEBSEARCH_QUEUE_WAIT |
20 |
Seconds a call may wait for a free slot before being answered busy |
WEBSEARCH_SLOW_CALL_S |
30 |
A successful call at or above this logs SLOW <tool> took Ns to stderr |
WEBSEARCH_RERANK_TIMEOUT |
60 |
Reranker read timeout. On timeout the call degrades to engine ranking (see below) |
WEBSEARCH_RERANK_COOLDOWN |
20 |
After a reranker timeout, skip it for this long instead of paying the timeout again |
WEBSEARCH_RERANK_QUEUE_WAIT |
15 |
Longest a call waits behind other calls for the single rerank worker before answering with base order. 0 = wait without bound |
WEBSEARCH_OCR_IDLE_UNLOAD |
90 |
Idle seconds before the cached OCR converter (and its ~1 GB of models) is dropped |
WEBSEARCH_OCR_MIN_FREE_MB |
900 |
Refuse OCR below this much available memory (ocr_skipped: "low_memory") instead of swap-thrashing the whole server |
WEBSEARCH_DDG_TIMEOUT |
0 |
Optional per-call timeout for the blocking ddgs network work; 0 = no extra bound |
Three response shapes exist because of this, and each is deliberate:
{"success": false, "deadline_exceeded": true, "deadline_s": 240,
"elapsed_s": 241.3, "retryable": true,
"partial": {"success": true, "results": [...]},
"partial_note": "Partial result captured before the deadline: ..."}
partial— the stages that finished are still returned, because a cited source list is worth more than an empty retry.searchpublishes its snapshot as soon as ranking is done.deadline_exceeded— the call was cut by us, not by the client, and it says which knob to raise. Narrow the request (depth,count,synthesize=false) before raising the deadline: the deadline protects the server, it does not make the work cheaper.busy+retryable— admission was full forWEBSEARCH_QUEUE_WAIT. Retrying shortly is correct; queueing behind an unbounded swarm is what turns one slow call into many timed-out ones.
Reranking degrades honestly rather than failing: search returns a reranker
field only when ranking was not the requested path (worker unavailable, timeout,
cooldown, bad reply), plus the ordering that was used instead. On this box the
reranker's own p90 was 56 s, so a 60 s timeout trades ~10 % of reranked calls
for never stalling a swarm call for the full timeout.
The worker is one socket serving one request at a time, so a burst queues: with
8 calls in flight the last one waits for seven completions. Measured on 12 lanes
that queue alone pushed a search to 253 s, which is why the wait for the worker
is now capped (WEBSEARCH_RERANK_QUEUE_WAIT, 15 s). A call that cannot get the
worker in time answers with base order and says so, instead of turning a 4 s
ranking step into a 4-minute search.
OCR (scanned PDFs) runs on a dedicated single-slot pool with a cached converter
and a memory guard, and reports ocr_ms. asyncio.wait_for cannot cancel a
thread, so a per-call to_thread build kept loading docling models after the
request was abandoned; one cached converter plus one slot removes both the
model reload and the pile-up.
Why the answer is prose, not JSON
The tool envelope is already structured: answer, sources[], citations_used,
grounded, diversity, as_of. Asking the model to also emit JSON moves
structure from a place where it is generated deterministically into a place where
it is asserted, and measured behaviour is worse: a required key_facts array
gets padded with restatements of a single fact, which reads as corroboration.
JSON parsed 3/3 in testing, so this is not a reliability argument - it is a
content-quality and duplication argument. See exec/websearch-format-test/.
Which model writes the answer
The in-tool model comes first by default, because it is free at the point of use. Groq is the fallback if it fails. Both are plain OpenAI-compatible chat endpoints, so one call path serves both.
| Env | Default | Meaning |
|---|---|---|
WEBSEARCH_SYNTH_PROVIDER |
local |
local (in-tool model first), groq (Groq first), auto (either, in-tool preferred). Applies to every model call, not just synthesis |
WEBSEARCH_MODEL_FALLBACK |
on |
off = use only the preferred endpoint and report an error if it fails, instead of falling through to the other provider. Set it when the model choice is a requirement (cost, privacy, compliance), not a preference |
WEBSEARCH_MIN_MODEL_TOKENS |
512 |
Floor on the token budget of any model request. Reasoning models spend budget on reasoning_content first, so a 64-token label request comes back empty; the floor is what makes small helper calls work |
WEBSEARCH_DEFAULT_DEPTH |
auto | Pins the default depth. Auto = 3 when a model lane exists, else 1 |
WEBSEARCH_ENRICH_MAX_PAGES |
8 |
Ceiling on page fetches per search. The count itself is max(5, depth*2) |
WEBSEARCH_SYNTH_TOKENS |
1200 |
Answer budget. On finish_reason=length the result says truncated |
WEBSEARCH_SYNTH_MAX_SOURCES |
8 |
Ceiling on sources in the context |
WEBSEARCH_DOMAIN_CAP |
2 |
Max sources from one registrable domain |
WEBSEARCH_MIN_SOURCES |
3 |
Floor before the domain cap relaxes |
WEBSEARCH_SYNTH_SNIPPET_CHARS |
1500 |
Chars of each source given to the model |
doctor() reports all of these resolved (env.defaults) plus the active prompt
versions (env.prompts), and lists synthesis as its own lane, so "why is there
no answer?" has an answer instead of a blank field.
What a search costs in model calls
Search is not a model-free operation, and pretending otherwise made it
unpredictable. Every call that needs a model goes through llm.py, and every
response says where the tokens went:
"model_calls": {"rewrite": 1, "classify": 1, "decompose": 1, "synthesis": 1, "verify": 1}
| Call | When | Off switch |
|---|---|---|
rewrite |
the query depends on chat history (history argument) |
no history, no call |
classify |
always (biases engine selection, never suppresses search) | google_ai_only |
expand |
short query, depth 1 | query_expand=false |
decompose |
depth >= 2 | query_expand=false |
synthesis |
results exist | synthesize=false |
verify |
after synthesis | synthesize=false |
Two consequences worth knowing:
- One endpoint preference, everywhere. Before this was unified, synthesis
used the free in-tool model while rewrite/classify/decompose called Groq
directly and returned nothing without a Groq key. The result was a
model-written answer produced from an unplanned, literal query. Now the
in-tool model plans the search too, and Groq is only the fallback
(
WEBSEARCH_SYNTH_PROVIDER). - Small budgets are a trap. A reasoning model spends tokens on
reasoning_contentbefore it emitscontent, so a 64-token "output one label" request returns 200 with an empty message. Callers saw their fallback value and it looked like a real classification. Every request now gets at leastWEBSEARCH_MIN_MODEL_TOKENS(512), and an unfilled request is reported and logged instead of being mistaken for an answer.
What works out of the box, and what is optional
Everything keyless works immediately after pip install. No API keys, no browser, no GPU: search (DuckDuckGo, Bing RSS, Wikipedia, arXiv, Stack Exchange, GitHub, HN, grep.app, Google News RSS), fetch, map_site, pdf_extract, screenshot.
Four lanes need optional setup. Run doctor() and it tells you which are live and prints the exact command for the rest:
| Lane | Needs | On a headless server |
|---|---|---|
Google AI Mode (search(google_ai_only=True)) |
a Chromium-based browser with --remote-debugging-port + a signed-in Google account in that profile |
unavailable — no signed-in browser exists there |
| browser_action | same browser, no Google login needed | unavailable |
| CDP fallback for bot-walled pages | same browser | unavailable; those pages return a terminal verdict instead of hanging |
| Cross-encoder reranking | an optional local worker (separate download, ~500MB model) | works if installed; otherwise scoring falls back to blended relevance |
Enabling the browser lanes:
# any Chromium-family browser: chromium, google-chrome, brave, Helium, edge, vivaldi
chromium --remote-debugging-port=9222 --user-data-dir="$HOME/.mcp-browser"
Then open google.com in that window and sign in once (needed only for Google AI Mode). Point elsewhere with WEBSEARCH_CDP_URL=http://host:port.
# confirm
doctor() # per-lane report
search(query="...", google_ai_only=True) # explicit error + fix list if no browser
The browser lanes are local-only by design. On a server the HTTP lanes are unaffected, so the server is still fully useful — it just cannot query Google AI Mode. Without a browser, google_ai_only=True returns a structured error naming the cause and the fix, never an empty result set.
Setup
Requires Python 3.14+.
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp run.example run # fill in your API keys (run is gitignored — never commit it)
chmod +x run
Point your MCP client at run (stdio):
{ "websearch": { "command": "/path/to/websearch/run" } }
API keys (run)
| Key | Used for | Required? |
|---|---|---|
GROQ_API_KEYS |
fallback for every model call: rewrite, classify, expand, decompose, synthesis, grounding checks | Optional with an in-tool model (it becomes the fallback); required for those steps if no in-tool model is configured |
TAVILY_KEYS |
Tavily engine lane | Optional |
NV_KEY |
NVIDIA embedding endpoint | Optional |
Everything else (DDG, Bing RSS, Wikipedia, arXiv, RSS) is keyless. Brave's HTML scrape is currently rate-limited from some networks, so Bing RSS is the primary general-web door and Brave is a last resort.
Install from PyPI
pip install skdev-websearch-mcp # core install, small
pip install skdev-websearch-mcp[all] # + browser/OCR/extract extras (multi-GB)
The websearch-mcp console script speaks MCP over stdio:
{ "websearch": { "command": "websearch-mcp" } }
Docker
docker build -t websearch-mcp .
docker run --rm -i websearch-mcp
Keys are passed at run time, never baked in: -e GROQ_API_KEYS=....
Heavy extras: docker build --build-arg EXTRAS="[all]" -t websearch-mcp .
websearch MCP questions
How do I add websearch MCP to Claude Code?
Run claude mcp add --env 'GROQ_API_KEYS=<YOUR_GROQ_API_KEYS>' --env 'NV_KEY=<YOUR_NV_KEY>' --env 'S2_KEY=<YOUR_S2_KEY>' --env 'SE_KEY=<YOUR_SE_KEY>' --env 'TAVILY_KEYS=<YOUR_TAVILY_KEYS>' websearch -- uvx --from skdev-websearch-mcp websearch-mcp in your terminal. Add --scope user to make it available in every project.
How do I add websearch MCP to Cursor?
Open ~/.cursor/mcp.json and add the server under mcpServers, using the Cursor config shown in the install section above.
What tools does websearch MCP provide?
It exposes 18 tools, including doctor, ping, search, fetch, screenshot.
How do I check if the search lanes are working?
You should run the 'doctor' tool first. It reports which lanes are usable and provides the exact command to enable others.
How does the server handle long web pages to save tokens?
You can use the 'distill' option in the fetch tool, which runs a small model to return a compressed, grounded summary of the page.
What happens if a search result contains instructions to ignore previous prompts?
The server treats results as untrusted data wrapped in tags, ensuring the model treats them as data rather than instructions.