CrawlBit MCP server
✓ VerifiedAllows an AI client to check if AI engines can find, read, and recognize a website through crawler access, entity recognition, and off-page gap analysis.
Also known as amati032-dev/crawlbit-mcp, crawlbit-mcp
- Stars
- 1
- Tools
- 4
- Downloads, 30 days
- —
- Latest version
- 1.0.1
- Last commit
- Aug 14, 2026
- License
- MIT
What is CrawlBit MCP?
CrawlBit allows an AI client to check if AI engines can find, read, and recognize a website through crawler access, entity recognition, and off-page gap analysis. It exposes 4 tools, such as crawlbit_crawler_watch, crawlbit_entity_check, crawlbit_offpage_gaps.
Summary generated from the project's README
About CrawlBit MCP server
CrawlBit provides a set of diagnostic tools designed to help developers and SEO specialists understand how AI engines interact with their websites. By using this MCP server, users can determine if specific bots like GPTBot, ClaudeBot, or Google-Extended can access their content. The server functions locally via Node.js and integrates with clients such as Claude Desktop, Claude Code, and Cursor.
Unlike many SEO tools, this server requires no accounts, no API keys, and no signups. It focuses on measuring the technical conditions necessary for AI visibility rather than running expensive live queries against LLMs. The four included tools allow for checking crawler access via robots.txt, verifying brand entity recognition, identifying off-page gaps, and inspecting structured data for AI shopping agents. Because it avoids running language models directly, the service remains free and avoids the rate-limiting issues common with more complex AI-driven audits.
What you can do with it
- 1Verify crawler accessCheck if specific AI bots like PerplexityBot or ClaudeBot are permitted to read a website.
- 2Assess brand entity recognitionDetermine if a brand is presented clearly enough for an AI to recognize it as a distinct entity.
- 3Audit AI shopping compatibilityInspect structured data to see if AI shopping agents can understand and recommend products.
Written from the project's README. Every use case is checked against a quote from it.
How to install CrawlBit MCP
Formats verified 10 Nov 2026Add to CursorAdd to GooseInstall CrawlBit MCP in Claude Code
$ claude mcp add crawlbit-mcp -- npx -y crawlbit-mcpInstall CrawlBit MCP in Cursor
~/.cursor/mcp.json
{
"mcpServers": {
"crawlbit-mcp": {
"command": "npx",
"args": [
"-y",
"crawlbit-mcp"
]
}
}
}Install CrawlBit MCP in VS Code
.vscode/mcp.json
{
"servers": {
"crawlbit-mcp": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"crawlbit-mcp"
]
}
}
}Install CrawlBit MCP in VS Code CLI
$ code --add-mcp '{"name":"crawlbit-mcp","type":"stdio","command":"npx","args":["-y","crawlbit-mcp"]}'Install CrawlBit MCP in Claude Desktop
claude_desktop_config.json
{
"mcpServers": {
"crawlbit-mcp": {
"command": "npx",
"args": [
"-y",
"crawlbit-mcp"
]
}
}
}Install CrawlBit MCP in Codex
$ codex mcp add crawlbit-mcp -- npx -y crawlbit-mcpInstall CrawlBit MCP in Codex config.toml
~/.codex/config.toml
[mcp_servers.crawlbit-mcp] command = "npx" args = ["-y", "crawlbit-mcp"]
Install CrawlBit MCP in Gemini CLI
$ gemini mcp add -s user crawlbit-mcp npx -- -y crawlbit-mcpInstall CrawlBit MCP in Gemini settings.json
~/.gemini/settings.json
{
"mcpServers": {
"crawlbit-mcp": {
"command": "npx",
"args": [
"-y",
"crawlbit-mcp"
]
}
}
}CrawlBit MCP tools 4
Detected from the READMEcrawlbit_crawler_watchcrawlbit_entity_checkcrawlbit_offpage_gapscrawlbit_shopping
Configuration
CRAWLBIT_BASE_URLOptional. Point the server at another CrawlBit instance. Defaults to https://www.crawlbit.app. No API key or account is needed to use this server.
Automated checks
Ran Nov 14, 2026Full report →Passed our automated checks. That is not a security audit: read the code and give the server the narrowest access it needs. How checks work
- Source repository linked and reachableamati032-dev/crawlbit-mcp is public and reachablegithub.com/amati032-dev/crawlbit-mcp ↗Pass
- Package names this repository as its sourcecrawlbit-mcp names amati032-dev/crawlbit-mcp as sourcenpmjs.com/package/crawlbit-mcp ↗Pass
- Build provenance (Sigstore attestation)No build attestation published for this releaseNot checked
- Open source license (OSI)MITspdx.org/licenses/MIT.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-08-14github.com/amati032-dev/crawlbit-mcp ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNo popular package one edit away from crawlbit-mcpPass
- No critical advisory in the package or its dependencies99 resolved dependencies checked; no critical advisoryosv.dev ↗Pass
- No hidden instructions to the model4 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNo tool named like delete, drop or execPass
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubCrawlBit MCP
Four free tools that answer one question: can AI engines find, read and recognise a website?
No account. No API key. No signup. Nothing here runs a language model, so nothing here costs you or us anything.
"Can ChatGPT read stripe.com?"
"Why does AI never mention my store?"
"Where is my brand missing outside my own site?"
Ask in plain language. Your AI client picks the right tool.
Install
Requires Node.js 18+.
Claude Code
claude mcp add crawlbit -- npx -y crawlbit-mcp
Claude Desktop
Add to claude_desktop_config.json:
{
"mcpServers": {
"crawlbit": {
"command": "npx",
"args": ["-y", "crawlbit-mcp"]
}
}
}
Cursor
Add to .cursor/mcp.json:
{
"mcpServers": {
"crawlbit": {
"command": "npx",
"args": ["-y", "crawlbit-mcp"]
}
}
}
Restart your client. Four tools appear.
The four tools
| Tool | What it answers |
|---|---|
crawlbit_crawler_watch |
Can GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended read this site? |
crawlbit_entity_check |
Does this brand read as a clear entity AI can recognise? |
crawlbit_offpage_gaps |
Where is this brand missing outside its own website? |
crawlbit_shopping |
Can AI shopping agents understand and recommend these products? |
Try these
Check whether AI crawlers can read example.com
Does example.com read as a clear brand entity to AI?
Where is example.com missing off-page, and which gap should we close first?
Compare example.com and competitor.com on AI crawler access
The tools compose. A useful sequence is: crawler access first (a blocked crawler makes everything else pointless), then the entity check, then the off-page gaps.
What this does NOT do
Written plainly, because an SEO tool that overstates itself is worse than no tool.
- It does not test whether AI actually cites a brand. That means running real buyer questions against ChatGPT, Perplexity and Claude, reading the answers, and reporting who gets named instead of you. It costs real money per run and is not part of this free surface. These four tools measure the conditions that make a citation possible, which is a smaller claim.
crawler_watchreadsrobots.txtonly. It cannot see blocking done at the CDN, firewall or rate-limit layer. A site can pass here and still refuse the crawler in practice.entity_checkreads what the site publishes about itself. It says nothing about the brand's reputation across the rest of the web.shoppinginspects published structured data. It cannot see a merchant feed submitted privately to a platform.
If a site scores well on all four and still is not cited, the answer is almost always the same and it is not technical: too few third-party pages mention it. These tools will tell you that honestly rather than sell you a fix that does not exist.
Why only four tools
CrawlBit also runs a full technical audit and generates llms.txt. Both use a language model,
so both cost money per run, and neither is exposed here.
That is a deliberate line rather than a teaser. A free tool whose bill grows with its popularity
gets rate-limited, degraded or withdrawn the moment it succeeds, and the people who installed it
are the ones who pay for that. Everything in this server costs nothing to run, so nothing here
has to be taken back later. The audit and the llms.txt generator live at
crawlbit.app if you want them.
Rate limit
The API rate-limits per IP. Normal conversational use does not come close, and the server runs on your machine, so the budget is yours rather than shared. If you do hit it, you get a plain sentence rather than a stack trace.
Configuration
| Variable | Default | Purpose |
|---|---|---|
CRAWLBIT_BASE_URL |
https://www.crawlbit.app |
Point the server at another instance |
Privacy
The server runs locally and stores nothing. The only data leaving your machine is the URL you ask about, sent to the CrawlBit API to be analysed. There is no account, so there is nothing to tie a request to a person.
About
Built by CrawlBit, which measures AI visibility and does the off-page work that earns citations. The free tools here are the measurement half. The paid work is the other half, and you are not required to look at it to use this.
MIT licensed. Issues and pull requests welcome.
CrawlBit MCP questions
How do I add CrawlBit MCP to Claude Code?
Run claude mcp add crawlbit-mcp -- npx -y crawlbit-mcp in your terminal. Add --scope user to make it available in every project.
How do I add CrawlBit MCP to Cursor?
Open ~/.cursor/mcp.json and add the server under mcpServers, using the Cursor config shown in the install section above.
What tools does CrawlBit MCP provide?
It exposes 4 tools, including crawlbit_crawler_watch, crawlbit_entity_check, crawlbit_offpage_gaps, crawlbit_shopping.
Does this tool require an API key?
No, the server requires no account, no API key, and no signup.
Can it detect if a site is blocked by a firewall or CDN?
No, the crawler watch tool only reads robots.txt and cannot detect blocking at the CDN, firewall, or rate-limit layer.
Does it test if AI engines actually cite a brand?
No, it measures the technical conditions that make a citation possible rather than running real buyer questions against LLMs.
Is my data stored by the server?
The server runs locally and does not store anything; only the URL you query is sent to the CrawlBit API for analysis.