MeshArc MCP server

mesharc

The Python client for the MeshArc API: a URL in, clean content out, and a record of what changed.

• Scrape one page or a batch — markdown, text, HTML, links, structured fields, a screenshot.
• Crawl a whole site with no project to set up first, and keep it as one if it turns out to be worth watching.
• Map what a site declares in its sitemaps before fetching any of it.
• Search the web for pages whose URLs you do not know, and read each result's page in the same call.
• Ask the agent a question: it searches, reads pages and answers, in the shape of a schema if you give one, with the pages its answer rests on. (Not live on mesharc.dev yet.)
• Monitor a search or a question: asked again hourly, daily or weekly, each answer compared with the last, a webhook told when it changed. (A question needs the agent, which mesharc.dev does not run yet.)
• Watch a site over time: projects, scheduled runs, and a change record — pages added, removed, modified, field by field.

Python 3.10 or newer. One dependency (httpx). Fully typed.

Install

pip install mesharc

Authentication

Every call needs an API key. Create one in the app under Settings → API keys — it is shown once — and give it to the client, or put it in MESHARC_API_KEY and construct the client with nothing:

from mesharc import MeshArc

arc = MeshArc("mesharc_...")
# or, with MESHARC_API_KEY in the environment:
arc = MeshArc()
# options: MeshArc(api_key, timeout=150.0, max_retries=2)

The key is only ever sent as a bearer header to api.mesharc.dev.

A key carries the scopes it was made with (read, write, admin), optionally a set of projects it may see, an expiry and a rate limit. A route the key may not use answers 403; a project it may not see answers 404.

Quick start

from mesharc import MeshArc

arc = MeshArc("mesharc_...")

page = arc.scrape("https://example.com/pricing")
print(page["markdown"])
print(page["verdict"], page["method"], page["credits"])   # ok crawler 1

scrape holds the request open until the page comes back (60 s by default), so there is nothing to poll for an ordinary page.

Reading pages

One page

page = arc.scrape("https://quotes.toscrape.com/js/", config={"render_js": "always"})

config is any setting a project takes, by its API name (render_js, only_main_content, formats, max_tier, wait_for_selector, actions, …). The full list, with defaults, is at mesharc.dev/docs/configuration.

page = arc.scrape_one(
    "https://example.com/",
    formats="markdown,text,cleanHtml",     # which bodies to return
    timeout_s=120,                         # how long the API holds the request (120 max)
    idempotency_key="pricing-2026-09-18",  # the same key returns the first answer for 24 h
)

Many pages

A list of URLs is a batch: grouped by host, fetched in parallel where the config allows, and returned as one row per URL.

batch = arc.scrape(["https://a.com/", "https://b.com/x"], config={"concurrency": 4})
for row in batch["pages"]:
    print(row["url"], row["httpStatus"], row["verdict"], row["credits"])

arc.scrape(urls, wait=False) returns the batch id at once; arc.batch(id, wait=True) finishes it later. Pass webhook_url= to be told instead of polling (batch.finished, signed with a secret returned once).

What a page looks like

Every page row carries the same fields, whether it came from a scrape, a crawl or a project:

Field	Meaning
markdown, text, cleanHtml, html, links, fields, screenshot	The bodies you asked for
httpStatus	The status the site answered with
verdict	ok, thin (short, but a page), blocked (refused, or a 404), skipped
errorCode	OK, or what went wrong: BLOCKED, NOT_FOUND, TIER_LIMIT, CAPTCHA, LOGIN_REQUIRED, RATE_LIMITED, TIMEOUT …
shape, warnings, signals	listing / table / form for a short page whose markup says what it is; short; why the judge decided as it did
method, tier, climbedTo	The engine that read it (crawler, tls, minted, browser, browser-residential, …), its tier, and how far a refused page climbed
credits, billedAs	What it cost; the rung it is priced at when not the one that fetched it
words, language, head, reason, crawledAt	Size, language, the head fields, the judge's sentence, when

A page the site refused costs 0, and so does a 404.

Crawling a site

job = arc.crawl(
    "https://docs.example.com",
    limit=200,                          # page budget
    maxDepth=3,                         # link hops from the seed
    includePaths=["/docs/*"],
    crawlMode="sitemap_first",          # what the sitemap declares first, then links
    maxTier="browser",                  # how far a refused page may climb
    scrapeOptions={"formats": ["markdown", "links"]},
    config={"crawl_delay_ms": 500},     # any project setting, directly
)

for page in job.pages():                # follows the cursor while the crawl runs
    print(page["url"], page["words"])
print(job.status, job.envelope["counts"], job.envelope["creditsUsed"])

crawl returns a handle immediately; job.pages() yields pages as they land and ends when the crawl does. wait=True blocks until it finishes; job.wait(), job.refresh(), job.cancel() do what they say; arc.get_crawl(id) reattaches to a crawl started elsewhere.

A one-shot crawl expires after 30 days. If the site is worth watching:

project = job.keep(name="Docs", schedule="weekly")

A webhook= in the options ({"url", "events", "metadata", "include"}) is told about crawl.started, crawl.page (fifty pages a message) and crawl.completed; its signing secret comes back once as job.webhook_secret. With "include": ["markdown"] the crawl.page messages carry each page's markdown too, up to about 1 MB a message: a page that does not fit, or was not stored, comes as "markdown": null with markdownOmitted (too_large or not_stored) and a contentUrl, an API path read with your key (job.page(url) reads the same). It needs crawl.page among the events, as it is when events is left out.

Mapping a site

for u in arc.map("https://docs.example.com"):
    print(u["url"], u["lastmod"])

details = arc.map_details("https://www.gov.uk/", search="visa", limit=500)
print(details["totals"], details["creditsUsed"])   # {'files': 29, 'urls': 508431, …} 29

A map costs one credit per sitemap file read — most sites are one file.

Searching the web

out = arc.web_search(
    "python packaging guide",
    limit=5,                            # 1 to 10 results
    freshness="month",                  # hour | day | week | month | year
    include_domains=["python.org"],     # exclude_domains= too; an exclude wins
    scrape=True,                        # each result's page as markdown as well
)

if out["status"] == "blocked":          # every engine refused; nothing was charged
    print(out["attempts"])
else:
    for hit in out["data"]:
        print(hit["position"], hit["title"], hit["url"], hit["snippet"])
        if hit.get("page"):
            print(hit["page"].get("markdown", "")[:200])
print(out["engine"], out["cached"], out["creditsUsed"])

again = arc.get_search(out["id"])       # a search started earlier, by id
for s in arc.searches(q="python"):      # the workspace's searches, newest first
    print(s["id"], s["query"], s["status"], s["resultCount"], s["creditsUsed"])

news = arc.web_search("python release", news=True, freshness="day")   # the engines' news results
for hit in news["data"]:
    print(hit["publisher"], hit["age"], hit["title"])                  # who published it, and how long ago ('20h')
more = arc.web_search("python packaging guide", page=2)               # results 11-20

web_search waits for the results and returns the whole envelope, the hits under data. A search is queued, running, done, blocked or error: blocked — every engine refused — is returned, not raised; error raises MeshArcError. A search that scrapes stays running until its pages land. scrape= also takes a dict, sent as given ({"formats": ["markdown", "links"], "maxCredits": 20}); country= and lang= say where and in what language to search; wait=False returns at once, and timeout_s is how long the API holds the request (60 s by default, 120 at most) before the client polls.

news=True searches the engines' news results rather than the web's (one or the other per search); each news hit also carries publisher and age, as the engine put it ("20h"). page= is which results page, 1 to 10: page 2 is results 11–20, and each page is its own search with its own charge.

A search needs a key that can write, and it spends credits. The results page costs the engine that read it; a results page every engine refused is free. An equal search — same query, news, page, country, lang, freshness and domains — within an hour of a finished one (ten minutes for news) comes from the cache (cached is true) with no charge for the results page. Scraped pages are charged, cached or not.

arc.search(project_id, q) is something else: it searches the pages a project has stored, not the web (see below).

Asking the agent

The agent does not run on mesharc.dev yet; web search does. The API has the agent, but a deployment runs agents only once it is set up for them, and until then a run is refused with 501 agents_unavailable -- as are follow-ups, agent tables and agent monitors. The client and the MCP tools are ready for it.

The agent takes a question, searches the web, maps sites and reads pages on its own, and answers with the pages it used. Give it a schema and the answer comes back in that shape:

run = arc.agent(
    "What does each plan on example.com cost per month?",
    urls=["https://example.com/pricing"],   # pages to start from, up to 20; omit to let it search
    allowed_domains=["example.com"],        # keep it to these sites, up to 20
    schema={                                # a JSON Schema of type object or array
        "type": "object",
        "properties": {"plans": {"type": "array", "items": {"type": "object", "properties": {
            "name": {"type": "string"}, "monthly": {"type": "number"}}}}},
    },
    max_credits=500,                        # the most the run may spend: 1 to 100,000, 2,000 by default
    max_steps=20,                           # the most steps it may take: 1 to 100, 40 by default
)

out = run.wait()                            # blocks until it finishes; the envelope
print(out["status"], out["creditsUsed"], out["steps"])
print(run.data)                             # {'plans': [...]}, matching the schema
for src in run.sources:                     # the pages the answer rests on
    print(src["url"], src["title"])
print(run.field_sources)                    # {'plans[0].monthly': {'url': ..., 'pageId': ...}, ...}

agent returns an AgentRun at once, queued or running; the prompt is up to 10,000 characters. timeout_s= asks the API to hold the request for the answer (120 s at most). Without a schema the answer is {"text": ...}. field_sources says where each value of the answer came from: its path ("plans[0].monthly", or "[2].name" for a list answer) to {url, pageId}, a page the run read.

run = arc.agent(
    "What does each plan on example.com cost per month?",
    webhook={"url": "https://hooks.example.com/mesharc", "events": ["agent.completed", "agent.failed"]},
    connection_id="conn_...",               # run on one of the workspace's own LLM connections
)
print(run.webhook_secret)                   # once: the secret the webhook's messages are signed with

webhook= is a URL, or {"url", "events", "metadata"}, told about agent.started, agent.action, agent.completed, agent.failed and agent.cancelled (all five by default); its signing secret comes back once, as run.webhook_secret; the handle continue_() returns carries the stopped run's secret on (the new run keeps the same webhook), and a run reopened with get_agent has none. connection_id= runs the agent on one of the workspace's own LLM connections — its model, its key, its bill.

To watch it work, follow the trace:

run = arc.agent("Who maintains the httpx library, and where is it developed?")
for event in run.trace():                   # follows the run until it ends
    print(event["seq"], event["kind"], event["text"])
print(run.wait()["data"])                   # {'text': '...'}

kind is start, resume, continue, model, search, fetch, render, map, select, extract, lookup (a data provider's record), propose, tool, busy or finish. trace(after=seq) resumes after an event already seen; follow=False yields what exists and returns.

run.cancel()                                # {'id': ..., 'status': 'cancelled'} if it was queued, 'cancelling' if running
run.refresh()                               # the run as it stands now

again = arc.get_agent(run.id)               # a run started earlier, by id
for r in arc.agent_runs(status="done"):     # the workspace's runs, newest first
    print(r["id"], r["status"], r["creditsUsed"], r["prompt"])
from datetime import date
for r in arc.agent_runs(model="openai:gpt-5.4-mini", since="2026-10-01", until=date.today()):
    print(r["id"], r["model"], r["createdAt"])

A run is queued, running, done, error, cancelled or credit_limit. wait() returns a run that is done, cancelled or stopped at credit_limit — the run reached max_credits, and what it had found is under data["partial"] — and raises MeshArcError 502 job_failed for error, with the API's text -- and for any other end it does not know, as the Node client does. A cancel stops a queued run at once and a running one before its next step; what it used stays charged. arc.cancel_agent(id) does the same by the run's id, with no handle needed.

agent_runs filters by status, by model (as runs name it) and by when the run was made: at or after since, before until, each an ISO 8601 date or date-time string or a date / datetime. A date means its midnight UTC; a date-time without a zone is read as UTC (so a naive datetime.now() is your local time read as UTC — pass datetime.now(timezone.utc) instead).

A run that stopped at credit_limit can be carried on with a new budget:

out = run.wait()
if out["status"] == "credit_limit":
    more = run.continue_(max_credits=1_000)   # a new run, on the same thread
    print(more.continues_run_id)              # run.id
    print(more.wait()["data"])                # the full answer, replacing the partial one

continue_() returns an AgentRun on the new run, which resumes from where the stopped one was, keeps the pages it read (not paid for again) and its webhook, whose secret the new handle's webhook_secret carries on. max_credits and max_steps are the new run's own; left out, they are the stopped run's. The stopped run's continued_by names the new one once refreshed. A run that did not stop at its credit limit, was already continued or has no saved progress raises MeshArcError 409 with code conflict; one past its keep date, 410 expired.

A run needs a key that can write, and it spends credits: each page it reads is charged as a read (a refused page is free), and the model's tokens at the model provider's price plus 20% — or 1 credit per 1,000 tokens on the workspace's own connection — all within max_credits. A run is kept 7 days; after that get_agent raises MeshArcError with status 410 and code expired, and trace() ends quietly.

Follow-ups and conversations

Needs agents running: mesharc.dev does not run agents yet, so everything in this section is refused there with 501 agents_unavailable.

first = arc.agent("Which plans does example.com sell?", urls=["https://example.com/pricing"])
first.wait()
nxt = first.follow_up("And which of them include SSO?")   # the same conversation
print(nxt.wait()["data"])

for c in arc.conversations(q="example.com"):              # the latest message first
    print(c["id"], c["messages"], c["title"])
thread = arc.conversation(first.id)                        # every run in it, oldest first, under "runs"

A follow-up answers after everything asked and found in the conversation before it, and may read again, free, any page the conversation read. Its request is its own — follow_up takes what agent takes — and it pays for its own work. Only the conversation's latest message can be followed, once it has finished: MeshArcError 409 conflict otherwise (the error names the latest), or past 30 messages.

Saved logins, data providers and proposals

Needs agents running: mesharc.dev does not run agents yet, so everything in this section is refused there with 501 agents_unavailable.

run = arc.agent(
    "How many employees does Stripe have, and what was Apple's revenue last year?",
    data_connections=["conn_pdl", "conn_fmp"],   # the workspace's data providers, one per provider
    logins="auto",                               # the saved logins it may sign in with (the default)
    proposals=True,                              # it may propose things to watch
)

data_connections= names up to two of the workspace's data-provider connections (arc.connections, below): People Data Labs for company data, Financial Modeling Prep for financial data. The agent may then look things up on your own key — the provider bills it, MeshArc charges no credits — and each record found is one of run.sources, as {url, title, provider}, url being the request it came from (your key is never in it). The agent reads pages signed out and signs in, page by page, where a page needs the account, with the workspace's saved login for that page's site (arc.logins); logins= says which it may use (every one by default, "none", or a list of ids), and login_id= is the older form of a list of one; logins are switched on per deployment. proposals=True lets the agent propose crawling a site or keeping a search or question on a schedule; nothing is made until a member approves it (arc.proposals).