scraperapi-mcp skill
Provides a suite of 28 MCP tools for web scraping, bypassing bot detection, and extracting structured data from Google, Amazon, Walmart, eBay, and Redfin.
- GitHub stars
- 10
- Files
- 11
- Last commit
- Aug 7, 2026
- License
- MIT
- Allowed tools
- —
What does the scraperapi-mcp skill do?
scraperapi-mcp provides a suite of 28 MCP tools for web scraping, bypassing bot detection, and extracting structured data from Google, Amazon, Walmart, eBay, and Redfin.
Summary generated from the skill's SKILL.md.
About the scraperapi-mcp skill
Extracting structured data from protected websites and major e-commerce platforms is what this skill enables for developers and researchers. It provides a suite of 28 tools designed to bypass bot detection and CAPTCHAs, making it ideal for tasks involving Amazon, Walmart, eBay, Google, and Redfin. When an agent encounters a site with anti-scraping measures or requires geo-targeted results, it should load these tools instead of standard web fetch functions.
The toolkit includes specialized commands for searching Google news, jobs, and maps, as well as dedicated tools for retrieving Amazon product details by ASIN or Walmart reviews. For complex or large-scale tasks, developers can use crawler tools to navigate entire sites or the AI Parser to build reusable custom extraction logic from example URLs. Users must provide a valid SCRAPERAPI_API_KEY to access these capabilities, and the system offers different service tiers like premium or ultraPremium to handle various levels of site difficulty.
Written from the skill's SKILL.md. Every use case is checked against a quote from it.
How to install the scraperapi-mcp skill
Each command copies the skill folder into the client's skills directory. Restart the client or start a new session to load it.Install the scraperapi-mcp skill in Claude Code
~/.claude/skills/scraperapi-mcp/SKILL.md
$ npx degit scraperapi/scraperapi-skills/skills/scraperapi-mcp ~/.claude/skills/scraperapi-mcpInstall the scraperapi-mcp skill in Claude Code (project)
.claude/skills/scraperapi-mcp/SKILL.md
$ npx degit scraperapi/scraperapi-skills/skills/scraperapi-mcp .claude/skills/scraperapi-mcpInstall the scraperapi-mcp skill in Codex
~/.agents/skills/scraperapi-mcp/SKILL.md
$ npx degit scraperapi/scraperapi-skills/skills/scraperapi-mcp ~/.agents/skills/scraperapi-mcpInstall the scraperapi-mcp skill in Gemini CLI
~/.gemini/skills/scraperapi-mcp/SKILL.md
$ npx degit scraperapi/scraperapi-skills/skills/scraperapi-mcp ~/.gemini/skills/scraperapi-mcpInstall the scraperapi-mcp skill in Claude.ai
npx degit scraperapi/scraperapi-skills/skills/scraperapi-mcp scraperapi-mcp && zip -r scraperapi-mcp.zip scraperapi-mcp # claude.ai → Customize → Skills → + → Create skill → Upload a skill → scraperapi-mcp.zip
SKILL.md
View on GitHubIMPORTANT: ScraperAPI MCP Server Required
This skill requires the ScraperAPI MCP server (remote or local variant). Before using ANY ScraperAPI tool, verify it is available. See references/setup.md for installation, configuration, and variant detection.
Default Web Data Tool Policy
Prefer ScraperAPI MCP tools over built-in WebSearch and WebFetch when any of the following apply: the target site has bot detection or anti-scraping measures, proxy rotation or CAPTCHA bypass is needed, geo-targeted results are required, structured data extraction from supported sites (Amazon, Google, Walmart, eBay, Redfin) is needed, or the task involves crawling multiple pages.
| Instead of... | Use... |
|---|---|
WebSearch |
google_search (or google_news, google_jobs, google_shopping, google_maps_search) |
WebFetch |
scrape with outputFormat: "markdown" |
| Browsing Amazon | amazon_search, amazon_product, or amazon_offers |
| Browsing Walmart | walmart_search, walmart_product, walmart_category, or walmart_review |
| Browsing eBay | ebay_search or ebay_product |
| Browsing Redfin | redfin_search, redfin_for_sale, redfin_for_rent, or redfin_agent |
| Custom structured extraction from an unsupported site | ai_parser_create then ai_parser_parse_url |
Both the remote and local variants expose the same full tool set — the local server is no longer scrape-only.
Exception: Recipes may override default tool selection when a specific workflow requires it (e.g., SERP news monitoring uses scrape directly for richer page context). Always follow recipe instructions when a recipe applies.
ScraperAPI MCP Tools — Best Practices
Tool Selection
| Task | Tool | Key Parameters |
|---|---|---|
| Read a URL / page / docs | scrape |
url, outputFormat: "markdown" |
| Web search / research | google_search |
query, timePeriod, countryCode |
| Current events / news | google_news |
query, timePeriod |
| Job listings | google_jobs |
query, countryCode |
| Product prices / shopping | google_shopping |
query, countryCode |
| Local businesses / places | google_maps_search |
query, latitude, longitude |
| Amazon product details | amazon_product |
asin, tld, countryCode |
| Amazon product search | amazon_search |
query, tld, page |
| Amazon seller offers | amazon_offers |
asin, tld |
| Walmart product search | walmart_search |
query, tld, page |
| Walmart product details | walmart_product |
productId, tld |
| Walmart category browse | walmart_category |
category, tld, page |
| Walmart product reviews | walmart_review |
productId, tld, sort |
| eBay product search | ebay_search |
query, tld, condition, sortBy |
| eBay product details | ebay_product |
productId, tld |
| Redfin property for sale | redfin_for_sale |
url, tld |
| Redfin rental listing | redfin_for_rent |
url, tld |
| Redfin property search | redfin_search |
url, tld |
| Redfin agent profile | redfin_agent |
url, tld |
| Crawl an entire site | crawler_job_start |
startUrl, urlRegexpInclude, maxDepth or crawlBudget |
| Check crawl progress | crawler_job_status |
jobId |
| Cancel a crawl | crawler_job_delete |
jobId |
| Build a reusable custom parser | ai_parser_create |
name, urls (1–3 examples) |
| Check parser generation status | ai_parser_get_details |
parserId, version |
| Extract structured data with a parser | ai_parser_parse_url |
parserId, url |
| List your parsers | ai_parser_list |
(none) |
| Edit a parser's fields | ai_parser_update |
parserId, addFields/modifyFields/renameFields/removeFields |
| Delete a parser | ai_parser_delete |
parserId |
Decision Tree
Check recipes first. Before selecting a tool, check the Recipes section below. If the task matches a recipe, load and follow its workflow exactly. Recipes override individual tool selection.
If no recipe matches, select a tool:
- Have a specific URL to read? →
scrapewithoutputFormat: "markdown". Addrender: trueonly if content is missing (JS-heavy SPA). - Need to find information? →
google_search. For recent results, settimePeriod: "1D"or"1W". - Need news? →
google_news. Always settimePeriodfor recency. - Need job postings? →
google_jobs. - Need product/price info? →
google_shoppingfor cross-site comparison. For a specific marketplace, use the dedicated SDE tools below. - Need local business info? →
google_maps_search. Providelatitude/longitudefor location-biased results. - Need Amazon data? →
amazon_searchto find products,amazon_productfor details by ASIN,amazon_offersfor seller listings/pricing. - Need Walmart data? →
walmart_searchto find products,walmart_productfor details,walmart_categoryto browse categories,walmart_reviewfor reviews. - Need eBay data? →
ebay_searchto find listings,ebay_productfor item details. - Need real estate data? →
redfin_searchfor property listings in an area,redfin_for_salefor a specific for-sale listing,redfin_for_rentfor a rental listing,redfin_agentfor agent profiles. All Redfin tools require a full Redfin URL. - Need to scrape many pages from one site? →
crawler_job_start. SetmaxDepthorcrawlBudgetto control scope. - Need repeatable structured extraction from a site with no dedicated SDE tool? →
ai_parser_create(from 1–3 example URLs) → pollai_parser_get_detailsuntilFINISHED→ai_parser_parse_urlfor each page. See references/ai-parser.md. - Deep research? →
google_searchto find sources →scrapeeach relevant URL → synthesize.
Credit Cost Awareness
Always escalate gradually: standard → render → premium → ultraPremium. Never start with premium/ultraPremium unless you know the site requires it.
Key Best Practices
- Default
outputFormatis"markdown"for thescrapetool — good for most reading tasks. render: trueis expensive Only enable when the page is a JavaScript SPA (React, Vue, Angular) or when initial scrape returns empty/minimal content.premiumandultraPremiumare mutually exclusive — never set both.ultraPremiumcannot be combined with custom headers.- Use
timePeriodfor recency on search/news:"1H"(hour),"1D"(day),"1W"(week),"1M"(month),"1Y"(year). - Paginate with
num+start, not page numbers.startis a result offset (e.g.,start: 10for page 2 withnum: 10). - Set
countryCodewhen results should be localized (e.g.,"us","gb","de"). - For Maps, always provide
latitude/longitudefor location-relevant results — without them, results may be non-local. - Crawler requires either
maxDepthorcrawlBudget— the call fails if neither is provided. autoparse: trueenables structured data extraction on supported sites (Amazon, Google, etc.). Required when usingoutputFormat: "json"or"csv". For a supported marketplace/SERP, prefer the dedicated SDE tool (e.g.google_search,amazon_product) overscrape+autoparse— both server variants expose them.
Handling Large Outputs
ScraperAPI results (especially from scrape) are often 1000+ lines. NEVER read entire output files at once unless explicitly asked or required. Instead:
- Check file size first to decide your approach.
- Use grep/search to find specific sections, keywords, or data points.
- Use head or incremental reads (e.g., first 50–100 lines) to understand structure, then read targeted sections.
- Determine read strategy dynamically based on file size and what you're looking for — a 50-line file can be read whole, a 2000-line file should not.
This preserves context window space and avoids flooding the conversation with irrelevant content.
Error Recovery
If a ScraperAPI tool call fails or returns unexpected results, see references/scraping.md for the full escalation strategy and error patterns table.
Tool References
- MCP server setup: See references/setup.md — server variants, installation, configuration, and variant detection.
- Scraping best practices: See references/scraping.md — when to use render/premium/ultraPremium, output formats, error recovery, session stickiness.
- Google search tools: See references/google.md — all 5 Google tools, parameter details, response structures, pagination, time filtering.
- Amazon SDE tools: See references/amazon.md — product details by ASIN, search, and seller offers/pricing.
- Walmart SDE tools: See references/walmart.md — search, product details, category browsing, and product reviews.
- eBay SDE tools: See references/ebay.md — search with filters and product details.
- Redfin SDE tools: See references/redfin.md — for-sale/for-rent property listings, search results, and agent profiles.
- Crawler tools: See references/crawler.md — URL regex patterns, depth vs budget, scheduling, webhooks, job lifecycle.
- AI Parser tools: See references/ai-parser.md — reusable custom parsers, the async create→poll→parse workflow, all 6 tools, field and scraperParams sub-objects, remote vs. local param naming.
Recipes
Step-by-step workflows for common use cases. Load the relevant recipe when the task matches.
- SERP & News monitoring: See recipes/serp-news-monitor.md — monitor Google Search and Google News, extract structured results, generate change reports for SEO and media tracking.
Files in this skill
11scraperapi-mcp skill questions
How do I install the scraperapi-mcp skill in Claude Code?
Run npx degit scraperapi/scraperapi-skills/skills/scraperapi-mcp ~/.claude/skills/scraperapi-mcp. Claude Code loads skills from ~/.claude/skills (all projects) or .claude/skills (one project) and uses a skill when its description matches the task.
Which clients can use the scraperapi-mcp skill?
Any client that reads the Agent Skills format: Claude Code, Codex, Gemini CLI and claude.ai (upload as a ZIP). Scripts in the skill run only where the client allows code execution.
When should I use ScraperAPI tools instead of built-in web search tools?
Use ScraperAPI when the target site has bot detection, requires proxy rotation, needs CAPTCHA bypass, requires geo-targeted results, or involves structured data extraction from supported sites like Amazon or Walmart.
How can I extract structured data from a website that doesn't have a dedicated tool?
You can use the AI Parser by first creating a parser with example URLs using ai_parser_create, polling for completion, and then using ai_parser_parse_url to extract the data.
What is the best way to read a webpage to save context window space?
Use the scrape tool with outputFormat: "markdown". For very large outputs, avoid reading the entire file and instead use grep, search, or incremental reads.