agentSearcher MCP server
Commit 27 Okt 2026Memungkinkan AI agent untuk melakukan pencarian web, mengekstrak konten, melakukan crawling website, dan memetakan struktur situs menggunakan proses self-hosted yang ringan.
Juga dikenal sebagai peakQuick/agentSearcher
- Stars
- 3
- Tools
- —
- Unduhan, 30 hari
- —
- Versi terbaru
- —
- Commit terakhir
- 27 Okt 2026
- Lisensi
- AGPL-3.0
Apa itu agentSearcher MCP?
Memungkinkan AI agent untuk melakukan pencarian web, mengekstrak konten, melakukan crawling website, dan memetakan struktur situs menggunakan proses self-hosted yang ringan.
Ringkasan dibuat dari README proyek
Tentang agentSearcher MCP server
Fungsi utama agentSearcher adalah memberikan kemampuan kepada AI agent untuk mencari, mengekstrak, dan melakukan crawling web tanpa bergantung pada API keys berbasis cloud. Solusi self-hosted ini berjalan sebagai satu proses Python tunggal, menjadikannya alternatif ringan dibandingkan setup kompleks yang melibatkan banyak layanan seperti SearXNG atau Firecrawl. Tool ini dirancang untuk berjalan di mesin lokal hanya dengan menggunakan CPU, serta mengelola memori secara otomatis dengan melakukan unloading pada browser dan model ranking setelah periode tidak aktif.
Developer dapat mengintegrasikan MCP server ini ke dalam workflow mereka melalui MCP menggunakan koneksi stdio atau streamable HTTP. Tersedia berbagai tool spesifik termasuk web_search, web_extract, web_crawl, dan web_map, yang dapat diubah namanya agar sesuai dengan format API Tavily untuk memudahkan migrasi. Sistem ini menangani halaman yang berat JavaScript melalui headless Chromium dan dapat membaca text layer dari file PDF. Dengan menggunakan multilingual cross-encoder dan ranking BM25, tool ini hanya memberikan potongan konten yang paling relevan untuk memastikan respons tetap sesuai dengan context window yang terbatas.
Apa yang bisa Anda lakukan dengannya
- 1Melakukan pencarian web yang tertargetMinta AI agent untuk menemukan informasi spesifik di berbagai search engine seperti DuckDuckGo atau Bing.
- 2Mengekstrak konten dari URL tertentuInstruksikan agent untuk mengambil teks relevan dari sebuah webpage atau dokumen PDF untuk dianalisis.
Ditulis berdasarkan README proyek. Setiap use case diperiksa terhadap kutipan dari README tersebut.
Konfigurasi
AGENTSEARCHER_API_KEYS
| Nama | Deskripsi | Tipe |
|---|---|---|
| AGENTSEARCHER_API_KEYS | — |
Pengecekan otomatis
Berjalan 14 Nov 2026Laporan lengkap →Belum diverifikasi. Sebuah server memerlukan skor 80, tidak ada temuan serius, dan data yang cukup untuk kami periksa. Cara kerja checks · Yang perlu diperbaiki untuk menambah poin
- Source repository terhubung dan dapat dijangkaupeakquick/agentsearcher is public and reachablegithub.com/peakquick/agentsearcher ↗Lolos
- Nama paket mencantumkan repositori ini sebagai sumbernyaTidak diperiksa
- Provenance build (atestasi Sigstore)Tidak diperiksa
- Lisensi open source (OSI)AGPL-3.0spdx.org/licenses/AGPL-3.0.html ↗Lolos
- Ada commit atau release dalam satu tahun terakhirLast commit or release 2026-10-27github.com/peakquick/agentsearcher ↗Lolos
- OpenSSF ScorecardTidak diperiksa
- Nama package tidak mirip dengan package populerTidak diperiksa
- Tidak ada advisory kritis pada paket atau dependensinyaTidak diperiksa
- Tidak ada instruksi tersembunyi untuk model0 tool descriptions and the README read; no override phrases, credential paths or hidden charactersLolos
- Tools yang bersifat destruktif telah diberi labelTidak diperiksa
- Endpoint remote merespons handshake MCPTidak diperiksa
README
Lihat di GitHubagentsearcher
Lightweight local web retrieval for AI agents: search, extract, crawl and map in one CPU-only process. No cloud keys.
An agent that works with the web needs to find pages, read them (including JavaScript pages and PDFs) and sometimes walk a whole site. agentsearcher does all of it in one Python process and returns only the passages that matter, sized for a small context window. It is available as an MCP server, an HTTP API (Tavily-compatible) and a CLI, with a dashboard in the browser.
Contents
Why
Self-hosted web retrieval usually means assembling several services: a metasearch engine such as SearXNG for search, a crawler such as Crawl4AI for page content, or a self-hosted Firecrawl, which runs as an API plus a browser service, Redis, RabbitMQ and Postgres. agentsearcher is one process instead:
- Light. ~75 MB idle; ~380 MB while ranking with the model; Chromium only starts for pages that need it. The model and the browser are unloaded after 2 minutes idle. Measured on a laptop i7-7500U with 7.5 GB RAM, no GPU (details).
- Search and reading together. One query fetches, renders, cleans and ranks the result pages, so the agent gets relevant passages instead of snippets or whole pages.
- Made for agents. Responses fit a small context window; reprints and near-duplicates are dropped; results the search engines agree on rank higher than SEO pages one engine pushes up.
- Easy to run.
pipx installordocker compose up. Nothing to configure to get started.
Features
- MCP server (stdio, or streamable HTTP at
/mcpon the API port) with the toolsweb_search,web_extract,web_crawlandweb_map. An env flag renames them totavily_*. - JavaScript pages: when a plain fetch isn't enough, the page is rendered in headless Chromium.
- PDF: the text layer is read. No OCR.
sitemap.xml,robots.txtandllms.txtare used by map/crawl, and.mdversions of pages by extract/crawl. Sites without these files work too.- Reprints and near-duplicate chunks are dropped (
dedup), and results can be capped per domain (max_per_domain). - SSRF protection: localhost and private networks are blocked by default.
- Response size is capped to fit a small context window. The model and the browser are unloaded after a period of inactivity.
- Pages and search results are cached for 15 minutes; optional block/lower/raise rules per domain.
- Dashboard at
/: request and page counters, CPU and memory (with the browser), recent requests, and a playground for search, extract, crawl and map. - Drop-in for Tavily clients:
/search,/extract,/crawl,/map,/usagerespond in the Tavily API format, so existing clients only change the base URL. Covered by tests with the officialtavily-pythonclient.
How it works
query ─► search engines (ddgs: duckduckgo, yahoo, yandex, brave, mojeek, bing; optional SearXNG)
─► fetch pages with curl_cffi (Chrome TLS fingerprint)
└─ empty page / SPA / anti-bot / text doesn't match snippet ─► patchright Chromium
─► trafilatura → markdown (PDF → text layer) ─► ~600-char chunks
─► BM25 ─► multilingual cross-encoder (ONNX int8, CPU)
─► dedup ─► only the best chunks of each page go into the response
Details: docs/how-it-works.md.
Installation
Docker (Chromium and the model are inside the image):
git clone https://github.com/peakQuick/agentSearcher.git && cd agentSearcher
docker compose up -d --build # API and MCP at http://127.0.0.1:8484, MCP at /mcp
Settings go under environment: in compose.yaml. On macOS and Windows, Docker Desktop runs a VM, which costs extra RAM; there, the pipx install below is lighter.
pipx (no venv to manage; pipx is in most package managers: apt install pipx, brew install pipx, pacman -S python-pipx):
pipx install git+https://github.com/peakQuick/agentSearcher
agentsearcher setup # one time: Chromium (~/.cache/ms-playwright) and the ~120 MB reranker (~/.cache/huggingface)
agentsearcher serve
Update with pipx upgrade agentsearcher. On a bare Linux server, agentsearcher setup --with-deps also installs Chromium's system libraries (needs root).
From source (development):
git clone https://github.com/peakQuick/agentSearcher.git && cd agentSearcher
python3 -m venv .venv
.venv/bin/pip install -e '.[dev]'
.venv/bin/agentsearcher setup
Below, agentsearcher means the installed command (.venv/bin/agentsearcher for a source install).
Quick start
CLI
agentsearcher search "what's new in python 3.14"
agentsearcher extract https://react.dev/learn -q "useState"
agentsearcher map https://docs.python.org/3/
HTTP API
agentsearcher serve # dashboard: open http://127.0.0.1:8484/
curl -s localhost:8484/search -H 'Content-Type: application/json' \
-d '{"query": "python asyncio tutorial", "max_results": 3}'
With tavily-python:
from tavily import TavilyClient
client = TavilyClient(api_key="local", api_base_url="http://127.0.0.1:8484")
print(client.search("python asyncio tutorial", max_results=3)["results"][0]["url"])
MCP
With serve or Docker running, HTTP clients and web UIs connect to http://127.0.0.1:8484/mcp:
claude mcp add --transport http agentsearcher http://127.0.0.1:8484/mcp
Or let the client start its own server over stdio:
claude mcp add agentsearcher -- agentsearcher mcp
Other clients: docs/mcp.md.
Configuration
The defaults work as is. Settings are environment variables with the AGENTSEARCHER_ prefix:
AGENTSEARCHER_API_KEYS=change-me agentsearcher serve # pipx / source
With Docker, put them under environment: in compose.yaml; for MCP over stdio, in the client's env block. GET /stats shows the values in effect (secrets hidden).
The ones people usually change: API_KEYS (before opening the server to a network), PROXY_URL (search engines block your IP), LLM_BASE_URL (answers with include_answer), MAX_REQUESTS / MAX_RENDERS (a machine with little RAM). All variables and ready-made recipes: docs/configuration.md.
Search depth
| mode | what it does | warm latency* |
|---|---|---|
ultra-fast |
search engine snippets only, no pages fetched | ~1 s |
fast |
pages without the browser, BM25 ranking | 3–4 s |
basic |
+ browser for JS and anti-bot pages, + cross-encoder | 6–9 s |
advanced |
+ more candidates, favor_recall, stricter about "empty" pages |
7–10 s |
* i7-7500U, 5 results. The first request takes longer: ~1.5 s to load the model and ~1 s to start the browser.
Documentation
| doc | what's inside |
|---|---|
| CLI | commands, flags and how to read the log |
| HTTP API | endpoints, parameters, responses and errors |
| MCP | tools and client setup |
| Configuration | environment variables |
| How it works | pipeline, browser, ranking, crawling, PDF, dedup, memory |
| Security | SSRF protection |
| Limitations | known limitations and possible problems |
Development
.venv/bin/ruff check .
.venv/bin/python -m pytest -m "not live" # offline: local fixture sites (SPA, challenge, PDF, sitemap...), tavily-python contract
.venv/bin/python -m pytest -m live # needs internet: tavily-python search at every search_depth
License
Copyright (C) 2026 peakQuick
GNU Affero General Public License v3.0. If you modify this project and let others use it over a network, you must make the source code of your modified version available to those users.
Pertanyaan tentang agentSearcher MCP
Apakah agentSearcher memerlukan API keys untuk pencarian web?
Tidak, tool ini dirancang untuk bekerja tanpa cloud keys dan menggunakan search engine seperti DuckDuckGo, Yahoo, dan Bing.
Apakah bisa menangani website yang berat JavaScript dan file PDF?
Ya, tool ini menggunakan headless Chromium untuk me-render halaman JavaScript dan dapat membaca text layer dari file PDF.
Berapa banyak memori yang digunakan?
Tool ini sangat ringan, menggunakan sekitar 75 MB saat idle dan sekitar 380 MB saat melakukan proses ranking dengan model.