WebCrawlerAPI MCP server
Commit Oct 17, 2026Converts any web page into clean Markdown for AI assistants, handling JavaScript-rendered content and stripping boilerplate.
Also known as WebCrawlerAPI/mcp-server, mcp-server
- Stars
- 0
- Tools
- 1
- Downloads, 30 days
- —
- Latest version
- 0.1.3
- Last commit
- Oct 17, 2026
- License
- MIT
What is WebCrawlerAPI MCP?
WebCrawlerAPI converts any web page into clean Markdown for AI assistants, handling JavaScript-rendered content and stripping boilerplate. It exposes 1 tool, such as url.
Summary generated from the project's README
About WebCrawlerAPI MCP server
Converting web pages into clean Markdown for AI assistants is the primary function of WebCrawlerAPI. This remote MCP server allows developers to pull documentation, articles, and product pages directly into a chat or a knowledge base. By using the get_markdown tool, users can fetch any public URL up to 2048 characters and receive the main content without the distraction of ads, navigation menus, or boilerplate code.
Because the server runs on the WebCrawlerAPI infrastructure, it can successfully process pages that require JavaScript rendering. Setup involves connecting to a streamable HTTP transport via clients like Claude Desktop, Cursor, VS Code, or ChatGPT. Instead of managing manual API keys, the system uses OAuth 2.1 authentication. When you first connect, a browser window opens to let you sign in and approve access for your specific organization, which handles the billing for each request.
What you can do with it
- 1Extracting documentation for AI contextA developer can instruct an AI to fetch a specific technical manual and convert it into Markdown for better reasoning.
- 2Reading JavaScript-heavy websitesYou can request content from modern web applications that require script execution to display information.
- 3Cleaning web content for LLMsAn AI client can retrieve a webpage while automatically stripping away unnecessary elements like advertisements.
Written from the project's README. Every use case is checked against a quote from it.
How to install WebCrawlerAPI MCP
Formats verified 10 Nov 2026Add to CursorAdd to GooseInstall WebCrawlerAPI MCP in Claude Code
$ claude mcp add --transport http mcp-server https://mcp.webcrawlerapi.com/mcpInstall WebCrawlerAPI MCP in Cursor
~/.cursor/mcp.json
{
"mcpServers": {
"mcp-server": {
"url": "https://mcp.webcrawlerapi.com/mcp"
}
}
}Install WebCrawlerAPI MCP in VS Code
.vscode/mcp.json
{
"servers": {
"mcp-server": {
"type": "http",
"url": "https://mcp.webcrawlerapi.com/mcp"
}
}
}Install WebCrawlerAPI MCP in VS Code CLI
$ code --add-mcp '{"name":"mcp-server","type":"http","url":"https://mcp.webcrawlerapi.com/mcp"}'Install WebCrawlerAPI MCP in Codex
$ codex mcp add mcp-server --url https://mcp.webcrawlerapi.com/mcpInstall WebCrawlerAPI MCP in Codex config.toml
~/.codex/config.toml
[mcp_servers.mcp-server] url = "https://mcp.webcrawlerapi.com/mcp"
Install WebCrawlerAPI MCP in Gemini CLI
$ gemini mcp add -s user --transport http mcp-server https://mcp.webcrawlerapi.com/mcpInstall WebCrawlerAPI MCP in Gemini settings.json
~/.gemini/settings.json
{
"mcpServers": {
"mcp-server": {
"url": "https://mcp.webcrawlerapi.com/mcp",
"type": "http"
}
}
}WebCrawlerAPI MCP tools 1
Detected from the READMEurl
| url | — |
Automated checks
Ran Nov 14, 2026Full report →Not verified yet. A server needs a score of 80, no serious finding, and enough data for us to check. How checks work · What to fix for more points
- Source repository linked and reachablewebcrawlerapi/mcp-server is public and reachablegithub.com/webcrawlerapi/mcp-server ↗Pass
- Package names this repository as its sourceNot checked
- Build provenance (Sigstore attestation)Not checked
- Open source license (OSI)MITspdx.org/licenses/MIT.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-10-31github.com/webcrawlerapi/mcp-server ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNot checked
- No critical advisory in the package or its dependenciesNot checked
- No hidden instructions to the model1 tool description and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNo tool named like delete, drop or execPass
- Remote endpoint answers the MCP handshakeHTTPS endpoint answered 401 (sign-in required)Pass
README
View on GitHubWebCrawlerAPI MCP server
A remote Model Context Protocol server that lets AI assistants read any web page as clean Markdown. It runs on WebCrawlerAPI, so JavaScript-rendered pages work and navigation, ads and boilerplate are stripped. Use it to pull docs, articles and product pages into a knowledge base or into a chat.
- Remote URL:
https://mcp.webcrawlerapi.com/mcp - Transport: Streamable HTTP (stateless)
- Sign-in: OAuth 2.1 through
https://auth.webcrawlerapi.com, no API key to paste - Tool:
get_markdown
You need a WebCrawlerAPI account. Sign up at https://dash.webcrawlerapi.com.
Connect
One-click install: Add to Cursor · Add to VS Code
In every client below, the first request opens a browser window where you sign in and approve access for one of your organizations. Requests are billed to that organization.
Claude Code
claude mcp add --transport http webcrawlerapi https://mcp.webcrawlerapi.com/mcp
Then run /mcp inside Claude Code, pick webcrawlerapi and choose Authenticate.
Claude.ai and Claude Desktop
- Open Settings, then Connectors.
- Choose Add custom connector.
- Name:
WebCrawlerAPI. URL:https://mcp.webcrawlerapi.com/mcp. - Click Connect and approve access in the browser.
Cursor
Add this to ~/.cursor/mcp.json (or .cursor/mcp.json in a project):
{
"mcpServers": {
"webcrawlerapi": {
"url": "https://mcp.webcrawlerapi.com/mcp"
}
}
}
Open Cursor Settings, then MCP, and click Connect next to webcrawlerapi.
VS Code
Add this to .vscode/mcp.json (or run "MCP: Add Server" and choose HTTP):
{
"servers": {
"webcrawlerapi": {
"type": "http",
"url": "https://mcp.webcrawlerapi.com/mcp"
}
}
}
Start the server from the CodeLens above the entry and approve access in the browser.
MCP Inspector
npx @modelcontextprotocol/inspector
Pick transport "Streamable HTTP", URL https://mcp.webcrawlerapi.com/mcp, then use the OAuth flow in the Auth tab
and call get_markdown.
ChatGPT
- Open Settings, then Connectors, and turn on Developer mode (Advanced).
- Create a connector with URL
https://mcp.webcrawlerapi.com/mcpand authentication set to OAuth. - Enable the connector in a chat and approve access in the browser.
Menu names change between ChatGPT plans and releases. Custom connectors may not be available on every plan.
Tool reference
get_markdown
Fetch a public web page and return its main content as clean Markdown.
| Input | Type | Required | Description |
|---|---|---|---|
url |
string | yes | Absolute http:// or https:// URL, up to 2048 characters |
Returns the page as Markdown text. On failure the tool returns an error result with a short human-readable message (for example an unreachable site, a page blocked by robots.txt, a timeout, or an empty balance).
Annotations: read-only, idempotent, not destructive, open world (it reaches out to the public web).
Each call is billed to your WebCrawlerAPI organization. See https://webcrawlerapi.com/pricing.
Authentication
The server is an OAuth resource server. Clients discover the authorization server from
https://mcp.webcrawlerapi.com/.well-known/oauth-protected-resource, which points to https://auth.webcrawlerapi.com.
- Scope:
markdown:read(the only scope) - Tokens are bound to the MCP server URL (resource indicator) and to the organization you approve.
- You can review and revoke connected apps in the dashboard under Connected apps.
- The server never forwards your token to other services.
Privacy and terms
- Privacy policy: https://webcrawlerapi.com/privacy
- Terms of service: https://webcrawlerapi.com/tos
The server receives the URLs you ask it to fetch and returns the page content. It does not log tokens or full URLs with query strings.
Pricing
Usage is billed to your organization. Current prices: https://webcrawlerapi.com/pricing
Contributing
See CONTRIBUTING.md and the Code of Conduct. Changes are listed in CHANGELOG.md.
Support
support@webcrawlerapi.com. Security reports: see SECURITY.md.
Development
Requires Node 22 and pnpm.
pnpm install
cp .env.example .env # fill in the values
pnpm dev # tsx watch
pnpm lint && pnpm test && pnpm build
Configuration is via environment variables, see .env.example and CONTRIBUTING.md. Registry publishing
steps are in docs/registry-verification.md.
License
MIT, see LICENSE.
WebCrawlerAPI MCP questions
How do I add WebCrawlerAPI MCP to Claude Code?
Run claude mcp add --transport http mcp-server https://mcp.webcrawlerapi.com/mcp in your terminal. Add --scope user to make it available in every project.
How do I add WebCrawlerAPI MCP to Cursor?
Open ~/.cursor/mcp.json and add the server under mcpServers, using the Cursor config shown in the install section above.
What tools does WebCrawlerAPI MCP provide?
It exposes 1 tool, including url.
How do I authenticate with WebCrawlerAPI?
The server uses OAuth 2.1. When you make your first request, a browser window will open for you to sign in and approve access for your organization.
Can the server read pages that use JavaScript?
Yes, the server runs on WebCrawlerAPI, which ensures that JavaScript-rendered pages are processed correctly.
What happens if a page cannot be fetched?
The tool will return an error message explaining the failure, such as an unreachable site, a robots.txt block, a timeout, or an empty balance.
Does the server log my data?
The server receives requested URLs and returns content, but it does not log tokens or full URLs containing query strings.