Plainwright MCP server

Plainwright

Open-source, AI-native testing for web apps and AI products. Describe a test in plain English; an AI agent carries it out against the real thing and reports pass/fail (or a score) with steps, screenshots and a plain-English explanation. No selectors, no recorded scripts.

Kind	What it tests	How
Web UI	Flows in a real browser (sign-up, checkout, uploads)	An agent drives headless Chromium/Firefox/WebKit over Playwright MCP, reading the live page each step
API	REST endpoints	An agent sends real HTTP requests (optional OpenAPI spec as context)
Conversation	Chatbots and assistants	A simulated user with a persona and goal talks to your bot (website chat widget, HTTP API or A2A agent); a judge scores the transcript
Red team	Prompt injection, jailbreaks, prompt/PII leaks, off-topic, harmful content, tool misuse	Attacker conversations; each attack is judged resisted or breached
LLM eval / RAG eval	Answer quality over a dataset (CSV/JSONL)	Each row is sent to your endpoint and scored (correctness, relevance, safety, rubric · faithfulness, answer relevance, context precision/recall)
MCP server	Any MCP server	Automatic tool-contract check, then an agent exercises the tools
A2A agent	Agent-to-agent endpoints	Agent-card validation, then a conversation over A2A JSON-RPC
Site quality	Accessibility (axe-core), performance, broken links, SEO	Scripted browser checks — no AI involved

Around the tests: plain-English assertions and visual regression, saved logins, an encrypted secrets vault (the AI only ever sees {{secrets.NAME}}), one-time codes (TOTP / email), flaky-test detection with retries and quarantine, failure triage with issue filing (Jira, Linear, GitHub — by hand or automatically), imports from Jira/Linear/GitHub/Figma/Notion, AI root-cause analysis and run chat, a CI gate, scheduling, per-run token and cost tracking, analytics, team roles, and Google/GitHub sign-in. The full list is in docs/features.md.

It works with Anthropic, OpenAI, Gemini, OpenRouter, Groq or a local Ollama model — every provider drives the same tool-calling agent, and "Test connection" checks that the chosen model can call tools before it is used.

How it compares

	Open source	Self-host	Plain-English web tests	Self-healing	AI-product testing¹	MCP / A2A tests	Own / local model	CI gate
Plainwright	✅	✅	✅	✅	✅	✅ / ✅	✅ / ✅	✅
Auto Playwright	✅	✅	◐	◐	?	? / ?	◐ / ?	?
e2e by TesterArmy	✅	✅	✅	✅	?	? / ?	✅ / ✅	✅
Shortest (antiwork)	✅	✅	✅	◐	?	? / ?	◐ / ?	✅
Browser Use	✅	✅	◐	◐	?	? / ?	✅ / ✅	?
Stagehand (Browserbase)	✅	✅	◐	✅	?	? / ?	✅ / ◐	?
DeepEval (Confident AI)	✅	◐	?	?	◐	◐ / ?	✅ / ✅	✅
LangWatch Scenario	✅	✅	?	?	◐	? / ?	✅ / ◐	✅
promptfoo	✅	✅	✗	✗	✅	✅ / ◐	✅ / ✅	✅
Applitools	✗	◐	✅	✅	?	? / ?	? / ?	✅
Katalon	✗	◐	◐	✅	?	? / ?	✅ / ?	✅
mabl	✗	◐	◐	✅	◐	? / ?	? / ?	✅
Momentic	✗	◐	✅	✅	?	? / ?	? / ?	✅
QA Wolf	✗	◐	✅	◐	◐	◐ / ?	? / ?	✅
Testim (Tricentis)	✗	◐	◐	✅	?	? / ?	? / ?	✅
testRigor	✗	?	✅	✅	◐	? / ?	? / ?	✅
Testsigma	✗	◐	✅	✅	?	? / ?	✅ / ?	✅
Octomind (likely defunct)	✗	◐	✅	◐	?	? / ?	? / ?	✅

✅ yes · ◐ partial · ✗ no · ? not documented (no evidence found — not necessarily missing). ¹ Conversation (agent-to-agent) tests, red teaming, LLM evals and RAG evals: ✅ = all four, ◐ = some.

Checked 2026-10-21 from each product's own docs, pricing pages and repositories. The full comparison covers 23 features, including pricing, with a source link for every cell. Corrections are welcome — edit docs/comparison.json.

Quick start

git clone https://github.com/SaiBolisetty/Plainwright.git && cd Plainwright
cp .env.example .env          # PowerShell: Copy-Item .env.example .env
# In .env, set SECRET_KEY to a long random value:
#   python -c "import secrets; print(secrets.token_urlsafe(48))"
# Optional: ANTHROPIC_API_KEY, the fallback model when nobody has added an AI provider
docker compose up --build

Open http://localhost:8080, create an account, and follow the guided setup: add an AI provider, create a project, pick starter tests from the templates, and watch the first run live. The first start takes a few minutes while the browser containers install Playwright MCP and their browsers.

SECRET_KEY signs sign-ins and encrypts every stored credential. Left empty, it falls back to a public development key: with PLAINWRIGHT_ENV=development (the default) the app starts, logs CRITICAL and shows project owners a warning; with PLAINWRIGHT_ENV=production it refuses to start. Once you have data, never just edit the key — rotate it with python -m app.cli rotate-secret-key (docs/security.md).

How a run works

"Check that a returning customer can pay with a saved card"
        │  POST /api/runs/trigger/{test}        (or Run all · CI webhook · schedule · retry)
        ▼
backend ──► Redis queue ──► worker
                              │  resolves the AI provider (project → owner default → server key)
                              │  runs the engine for the test's kind, e.g. for Web UI:
                              │    browser_snapshot → browser_click(ref) → browser_type …
                              ▼
                   pass/fail · steps · screenshots · scores · transcript · cost
                              │  assertions, visual diff, CAPTCHA detection, retries, alerts
                              ▼
                Postgres  ──►  dashboard (live view over SSE)

The agent reads the page through the accessibility snapshot on every step, so renamed buttons, pop-ups and redirects don't break tests. Secrets are swapped in only when a tool runs and scrubbed from everything that is stored or shown.

Services

Service	Port	Role
nginx	8080	Public entry point: /api → backend, / → frontend
frontend	127.0.0.1:3000	React + Vite + Tailwind build
backend	127.0.0.1:8000	FastAPI REST API, auth, Alembic migrations on start, /metrics
worker	—	RQ workers (WORKER_REPLICAS, default 2) running the test engines
scheduler	—	Cron-scheduled tests and the stuck-run reaper
playwright-mcp (+ -firefox, -webkit)	127.0.0.1:8931 (Chromium)	Headless browsers behind Playwright MCP
postgres, redis	127.0.0.1:5433, 127.0.0.1:6379	Data and job queue

Only nginx is published on every interface; everything else is loopback-only or internal. The browsers live on a separate browser network shared only with the worker and backend, so a page under test cannot reach Postgres, Redis or the platform itself. The backend runs without --reload; for live reload use docker-compose.dev.yml (see docs/deploy.md).

Deploying to a VPS, Render (blueprint in render.yaml, or the Deploy to Render button above), Railway or Fly.io, S3/R2 storage, Sentry, Prometheus, OAuth, SMTP and scaling: see docs/deploy.md.

Security

• Provider API keys, vault secrets, integration configs and saved logins are encrypted at rest with a key derived from SECRET_KEY, and are never returned in full by the API.
• Sign-in tokens are revocable: changing or resetting a password, or "Sign out other sessions" in Settings, signs out every other session. Sign-in and registration are rate-limited, and Google/GitHub sign-in and project invites only attach to verified email addresses.
• Every server-side request to a user-supplied URL — API, eval, MCP and A2A targets, browser navigation, webhooks, trackers — goes through an SSRF guard that refuses cloud-metadata, loopback and the stack's own services (ALLOW_INTERNAL_TARGETS=true turns it off for a trusted single-user install).

The security model, the operator checklist and the key-rotation steps are in docs/security.md. Please report vulnerabilities privately, as described there.

Run from CI

Each project has a webhook and secret (Project → Settings → General → Run from CI). A pipeline triggers every active test and waits for the gate:

RESP=$(curl -fsS -X POST "$PLAINWRIGHT_URL/api/runs/webhook/$PLAINWRIGHT_PROJECT_ID" \
  -H "X-Webhook-Secret: $PLAINWRIGHT_WEBHOOK_SECRET" -H "Content-Type: application/json" \
  -d '{"triggered_by": "ci"}')
# then poll  $PLAINWRIGHT_URL$(echo "$RESP" | jq -r .status_url)  until .gate is "pass" or "fail"

.github/workflows/test.yml is a ready-made GitHub Actions job that does this and prints a summary table. Details: docs/ci.md.

Documentation

Where	What
docs-site/	The full documentation (Astro Starlight), published at https://saibolisetty.github.io/Plainwright/: getting started, every test kind, reliability, integrations, self-hosting, configuration and the REST API reference. Run it with cd docs-site && npm install && npm run dev (http://localhost:4321); .github/workflows/docs.yml publishes it to GitHub Pages.
docs/features.md	Everything Plainwright does, on one page
docs/testing-ai.md	Testing chatbots, agents, LLM/RAG endpoints, MCP servers and A2A agents
docs/ci.md	The CI webhook and quality gate
docs/deploy.md · docs/security.md	Deployment and the security model
docs/comparison.md	The full competitor comparison, one source link per cell
CONTRIBUTING.md · CHANGELOG.md · ROADMAP.md	Working on Plainwright, what changed, what's next

Benchmark

bench/ is a reproducible benchmark: 23 tasks on public practice sites (15 web UI, 2 API, 3 site-quality, and 3 controls that must fail) measuring pass rate, false-pass rate, flakiness, speed and cost per model, with adapters for running the same tasks in other tools. A dry run prints the plan and a cost estimate without calling anything:

backend/.venv/Scripts/python bench/run_bench.py --dry-run --models claude-haiku-5-5 --trials 3

No results are published yet: a real run spends money on an AI provider, so it waits for an explicit go-ahead. Method, fairness rules and flags: bench/README.md.

Development

# Backend (Python 3.12; 3.11 also runs the tests)
cd backend
python -m venv .venv && .venv/Scripts/pip install -r requirements-dev.txt   # .venv/bin on Linux/macOS
.venv/Scripts/python -m pytest -q tests          # unit tests: SQLite, fake LLM, no network
uvicorn app.main:app --reload --port 8000        # needs Postgres + Redis
rq worker --url redis://localhost:6379 test_runs

# Frontend
cd frontend
npm install
npx vitest run                                    # component tests
npm run dev                                       # Vite dev server

Unit tests never call a real LLM, network service or Docker. Schema changes go through Alembic (backend/migrations/); the backend upgrades the database on start. Setup details, the docs site, the bench and the pull-request checklist are in CONTRIBUTING.md.

License

Apache License 2.0 © 2026 Plainwright contributors. Third-party attributions (axe-core is bundled under MPL-2.0) are in NOTICE. Contributions are accepted under the same license and signed off under the Developer Certificate of Origin (git commit -s).