A read-only QA agent that tracks test findings, manages flaky test quarantines, and provides auditable regression reports.
Also known as ArtJack/verdict, verdict, verdict-qa-mcp
- Stars
- 4
- Tools
- 10
- Downloads, 30 days
- 2,298
- Latest version
- 0.93.1
- Last commit
- Nov 15, 2026
- License
- MIT
What is Verdict QA MCP?
Verdict QA is a read-only QA agent that tracks test findings, manages flaky test quarantines, and provides auditable regression reports. It exposes 10 tools, such as list_projects, get_verdict, get_findings.
Summary generated from the project's README
About Verdict QA MCP server
Tracking test findings and managing flaky test quarantines is what Verdict QA enables for developers using Claude Code or other coding agents. This read-only QA agent maintains a persistent state to compare current test results against previous runs, identifying whether issues are new, still open, resolved, or regressed. It uses a stable ID and age for every finding to ensure that regressions are always ranked first in reports.
Users interact with the agent through specific commands like /verdict:run or by installing it as a pip package via verdict-qa-mcp. The system classifies failures into five distinct categories, such as REAL_DEFECT or STALE_EXPECTATION, and manages flaky tests by moving them into a quarantine ledger with a mandatory expiry. Unlike other agents, it lacks an Edit tool and employs a write-scope guard to ensure it remains an independent auditor that never modifies your source code.
To use the server, you need Python 3.10 or newer for the MCP SDK. The agent works across various languages including Python, TypeScript, and Go by interfacing with your existing test runners. It provides auditable reports that include coverage on changed files and explicitly lists what was not tested during a run.
What you can do with it
- 1Identify regressed defectsAsk the agent to compare the current test run against the stored baseline to find reintroduced bugs.
- 2Audit test coverage gapsRequest a report that highlights which parts of the codebase were not covered by the current test suite.
- 3Verify specification requirementsHave the agent review a specification for contradictions or ambiguities before any code is written.
Written from the project's README. Every use case is checked against a quote from it.
How to install Verdict QA MCP
Formats verified 10 Nov 2026Add to CursorAdd to GooseInstall Verdict QA MCP in Claude Code
$ claude mcp add verdict -- uvx --from verdict-qa-mcp verdict-mcpInstall Verdict QA MCP in Cursor
~/.cursor/mcp.json
{
"mcpServers": {
"verdict": {
"command": "uvx",
"args": [
"--from",
"verdict-qa-mcp",
"verdict-mcp"
]
}
}
}Install Verdict QA MCP in VS Code
.vscode/mcp.json
{
"servers": {
"verdict": {
"type": "stdio",
"command": "uvx",
"args": [
"--from",
"verdict-qa-mcp",
"verdict-mcp"
]
}
}
}Install Verdict QA MCP in VS Code CLI
$ code --add-mcp '{"name":"verdict","type":"stdio","command":"uvx","args":["--from","verdict-qa-mcp","verdict-mcp"]}'Install Verdict QA MCP in Claude Desktop
claude_desktop_config.json
{
"mcpServers": {
"verdict": {
"command": "uvx",
"args": [
"--from",
"verdict-qa-mcp",
"verdict-mcp"
]
}
}
}Install Verdict QA MCP in Codex
$ codex mcp add verdict -- uvx --from verdict-qa-mcp verdict-mcpInstall Verdict QA MCP in Codex config.toml
~/.codex/config.toml
[mcp_servers.verdict] command = "uvx" args = ["--from", "verdict-qa-mcp", "verdict-mcp"]
Install Verdict QA MCP in Gemini CLI
$ gemini mcp add -s user verdict uvx -- --from verdict-qa-mcp verdict-mcpInstall Verdict QA MCP in Gemini settings.json
~/.gemini/settings.json
{
"mcpServers": {
"verdict": {
"command": "uvx",
"args": [
"--from",
"verdict-qa-mcp",
"verdict-mcp"
]
}
}
}Started by HostedMCP on 16 Nov 2026
We ran this command for verdict-qa-mcp 0.93.1 in an isolated container with no network and no keys. It answered and listed 10 tools.
Verdict QA MCP tools 10
Read from the server with tools/listlist_projectsRead-onlyList every project with a Verdict baseline in the solo state root.
get_verdictRead-onlyThe last recorded verdict, release blockers, and report path for a project.
get_findingsRead-onlyFindings for a project. status: 'open' (default), 'all', 'accepted' (the maintainer's accepted risks), or one of NEW / STILL_OPEN / RESOLVED / REGRESSED / ACCEPTED. REGRESSED sort first, then by severity, then by age (oldest first shown last run's pressure).
get_quarantineRead-onlyThe flaky-test quarantine ledger, with an expired flag per entry.
get_questionsRead-onlyThe questions the tester parked for a person — parked ones with their age, and answers no run has read yet. Answer with `verdict-answer`.
get_historyRead-onlyRun history, oldest first. Machine-native from runs.jsonl where it exists; the INDEX.md markdown parse remains as the legacy fallback for history that predates the file.
get_reportRead-onlyContent of a QA report. Default: the last run's report; pass `report` (a path relative to the QA root, as returned by get_verdict/get_history) for an earlier one. Reports over 512KB are truncated with a flag.
get_profileRead-onlyThe project's QA profile — isolation rules, risk areas, real test commands — plus the lessons ledger (judgment corrections) when one exists.
get_trendsRead-onlyRun-over-run trajectory: per-run test counts and verdicts parsed from the INDEX, plus the current pressure picture — open findings by severity, age distribution, quarantine size. Direction is the signal; cells the INDEX writes as prose come back raw with parsed numbers where possible.
get_stateRead-onlyThe full raw state.json for a project (escape hatch; schema in docs/state-schema.md).
Automated checks
Ran Nov 14, 2026Full report →Passed our automated checks. That is not a security audit: read the code and give the server the narrowest access it needs. How checks work
- Source repository linked and reachableartjack/verdict is public and reachablegithub.com/artjack/verdict ↗Pass
- Package names this repository as its sourceverdict-qa-mcp names artjack/verdict as sourcepypi.org/project/verdict-qa-mcp/ ↗Pass
- Build provenance (Sigstore attestation)Sigstore attestation for verdict-qa-mcppypi.org/project/verdict-qa-mcp/ ↗Pass
- Open source license (OSI)MITspdx.org/licenses/MIT.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-11-12github.com/artjack/verdict ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNo popular package one edit away from verdict-qa-mcpPass
- No critical advisory in the package or its dependencies37 resolved dependencies checked; no critical advisoryosv.dev ↗Pass
- No hidden instructions to the model0 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNot checked
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubVerdict
CI · verdict on itself · eval: 8/8 seeded defects · pinned rules: 527/527 killed · PyPI: verdict-qa-mcp · OpenSSF Best Practices: passing · MIT license
Your test suite is green. Verdict found a defect that had lived 4,595 days.
Verdict is a QA agent that does not fix, does not flatter, and does not forget. It measures
before it judges — the harness runs your gates, hashes every line a finding cites, re-runs
the guarding test at the old commit and the new one — and it keeps a memory: every run is a
delta against the last, findings age, regressions rank first, and the tester's own misses
are published beside its hits. The contract it runs under is immutable and hashed into every
verdict; what it learns lives beside the contract, dated and auditable, and never edits it.
The number above is real: FilePerms in a 7k★ Python library could not revoke a permission
bit since 2014-02-07, and every one of its 625 tests was green the day Verdict filed it —
the run, and the misses, are in the ledger.
/plugin marketplace add ArtJack/verdict # Claude Code
/plugin install verdict@verdict
/verdict:run
npx skills add ArtJack/verdict # every other coding agent
Most AI "QA agents" are a paragraph of enthusiasm with a checklist. They audit your repo from scratch every time, re-report the same 20 findings until you stop reading, call flaky tests "failures", call stale tests "failures", and end with "LGTM! 🎉".
Verdict is a Claude Code plugin built the way QA is actually practiced:
- It remembers. A state file carries the baseline. Every finding gets a stable ID and
an age; every run reports
NEW / STILL_OPEN / RESOLVED / REGRESSED— regressions ranked first, always. - A red test means something. Every failure is classified —
REAL_DEFECT,STALE_EXPECTATION(which needs a citation proving the change was intended),BRITTLE_TEST,ENVIRONMENT, orFLAKY(confirmed by re-runs, quarantined with an expiry). The classification most likely to excuse a regression carries the highest evidence bar. - A verdict you can defend. Every run ends in exactly one of
pass | pass with risks | blocked | fail— an open Blocker forcesfail,blockedis a legitimate outcome, and apassalways names what was not tested. - It never fixes your code. There is no
Edittool, a hook confines its writes to the QA root, and a strict-mode Bash guard closes the shell's write channels — a tester that patches what it judges isn't independent. The guard is a heuristic, not a sandbox, and the README says so. - It is tested, and it tests itself. A scored eval suite with the misses published, a hash-chained run history a copied or edited state fails, a track record the harness computes rather than the tester — and an audit of its own releases: 78 findings filed against itself in its first 15 runs — 59 fixed with the fix verified, one an accepted risk on the record, two still open, and 16 closed without proof, which its own ledger counts as unknown, not as wins. Every fixed harness rule is pinned as a mutant the suite must kill.
What a run looks like: a real delta report — Verdict on its own code, run 14.
Who pays for the model? You do, with the Claude subscription you already have: the
plugin runs inside your own session, nothing routes through anyone else, and everything
below the model — the state, the gate, the harness, the eval scorer — is stdlib Python
(the optional MCP server adds the mcp SDK)
that runs for free. Works on Python, TypeScript, Go, or anything with a test runner; the
eval fixtures cover Python and TypeScript.
Read next: Install · What installs, and when it runs · Quickstart · Why another QA agent · The tested tester · CI gate · Accepting a risk · FAQ
Install
/plugin marketplace add ArtJack/verdict
/plugin install verdict@verdict
Any other coding agent — Cursor, Codex, OpenCode and the rest of the agent skills ecosystem — gets the same doctrine as five skills, and the same harness as a pip package:
npx skills add ArtJack/verdict # release risk · verify a fix · flaky triage · root cause · spec review
pip install verdict-qa-mcp # verdict-facts, verdict-finalize, verdict-gate, verdict-accept, verdict-answer, verdict-replay, verdict-dupes
Listed on skills.sh.
The skills restate the contract for an agent that cannot run the verdict agent; the
hooks that enforce the read-only guarantee exist only in Claude Code, so there the
guarantee is the agent's own discipline plus the harness's refusals. AGENTS.md and
llms.txt at the repository root are for agents that read before they act.
Python 3.9 or newer, whatever your python3 resolves to — the hooks and the
fact harness are stdlib-only and are invoked by that name, which on a stock Mac is
/usr/bin/python3 (3.9). The optional MCP server is a pip install and needs 3.10+,
which is what requires-python in pyproject.toml refers to. The floor is tested:
a module that would fail to import on 3.9 fails CI instead
(tests/test_interpreter_floor.py) — because the
failure it prevents was silent. The Bash guard once raised on import there while the
write guard beside it kept working, so a strict session looked armed with half its
controls missing.
What installs, and when it runs
Installing a plugin means letting its code run in your sessions, so here is exactly
what this one does — measured from hooks/hooks.json, not summarised from memory.
Six hook registrations; each starts a python3 (tens of milliseconds) when its
event fires:
| Event | Fires on | Script | Silent when |
|---|---|---|---|
PreToolUse |
Write/Edit/MultiEdit/NotebookEdit |
write-scope guard | always, unless VERDICT_STRICT=1 or the caller is the verdict agent itself — and for one path in any session: a QA root's accepted.json / answers.json, which only verdict-accept / verdict-answer write |
PreToolUse |
Bash |
bash-scope guard | always, unless VERDICT_STRICT=1 or the caller is the verdict agent itself |
PostToolUse |
Write/Edit/MultiEdit |
state validator | unless the written file is a QA root's state.json or findings/<ID>.json |
Stop / SubagentStop |
end of turn | run-contract check | unless a QA run in this session left hand-written state, or a Verdict agent of this session measured the facts and is stopping without verdict-finalize (the run marker names the session; another session's marker, or another agent stopping beside a run in progress, says nothing) — each blocks at most once, never loops |
SessionStart |
session open | findings banner, or a one-time first-run hint | unless the repository has QA state — or, for the hint, unless it is a fresh interactive start in a git repository with a test suite Verdict has never seen (once per repository, three repositories per person, never with VERDICT_STRICT, VERDICT_NO_HINT=1 or a headless claude -p) |
Every hook fails open: malformed input, missing files, or an exception mean
exit 0 and silence — a broken hook must never brick a session. VERDICT_STRICT=1
is what arms the scope guards, and you set it only for dedicated QA sessions
(headless, CI, the nightly); in ordinary interactive work the guards are no-ops.
Prefer not to install globally? Everything works per-repository: copy
agents/verdict.md into <repo>/.claude/agents/ and the
hooks/hooks.json entries into <repo>/.claude/settings.json,
with ${CLAUDE_PLUGIN_ROOT} replaced by a checkout path. That is exactly how the
eval harness provisions its scratch projects — eval/run_eval.py
is the reference implementation.
Quickstart
/verdict:run # first run: profile + isolation rules + baseline
/verdict:run the payment retry change # every later run: a delta against the stored state
Every later run is a delta against the stored state. A repeat run returns something like:
VERDICT: fail
Scope: 2c67f47..b4e2943 (4 commits, 16 files) · run 4 (delta)
Isolation check: pass (no .env present; no live service touched)
Findings — REGRESSED first:
REGRESSED PRICER-F-002 Critical/P0 round_cents uses banker's rounding again (pricer.py:17)
resolved 08-19, reintroduced by b4e2943 — this forces the verdict
NEW PRICER-F-007 Major/P1 quarantine graveyard: test_listable_at_floor_exactly
skipped 114 days with no expiry — it is the test that would catch F-001
STILL_OPEN PRICER-F-001 Critical/P0 age 6d is_listable rejects a price exactly at the floor
FLAKY test_bulk_discount_applies — fails 3/6 runs with no code change; quarantined
until 2026-09-07, excluded from this verdict, listed until re-evaluated
Delta gates: tests 213 → 213 · duration +0.4% · coverage on changed files: no decrease
Release blockers: PRICER-F-002 (regressed), PRICER-F-001
Not tested: concurrency under parallel checkout — no harness present
Fix order: 1) F-002 2) F-001 (unskip its test first, watch it fail red) 3) F-007 expiry
Artifact: .qa/reports/2026-08-24-pricer-review.md
Why another QA agent
Typical qa-expert.md |
Verdict | |
|---|---|---|
| Remembers the last run | no — every run is a fresh audit | state file; NEW/STILL_OPEN/RESOLVED/REGRESSED with ages |
| Flaky tests | "keep flakes under 1%" (prose) | quarantine ledger with mandatory expiry; flakes excluded from the verdict, never from the report |
| Release decision | "go/no-go" appears as a checklist word | four-verdict contract; an open Blocker forces fail |
| Red test triage | "investigate failures" | five-class taxonomy; STALE_EXPECTATION requires an intent citation |
| Quality gates | ">90% coverage" absolutes | direction gates: coverage on changed files must not decrease; 0 tests collected ≠ 1 test failing |
| Test design | "test edge cases" | 24-technique catalog with risk triggers — incl. property-based, metamorphic (for ML/LLM output), MC-DC, contract tests (docs/test-design.md) |
| Can edit your code | nothing stops it | no Edit tool + write-scope hook + strict-mode Bash guard |
| Security | ignored, or oversold | opt-in report-only pass: dependency audit + diff secret scan; pentest explicitly out of scope |
| Risk prioritisation | "focus on high-risk areas" | the ranking is computed from the project's own finding history (severity-weighted, paths merged across citation depths), and the report must show the ranking, the cutoff, and everything below it — which lands in not-tested |
| Root cause | "investigate the failure" | a four-link chain with a citation per link, a mandatory class check (is this an instance or a pattern?), and causation proven by flipping the cause in a scratch copy — with its own scored fixture built around a decoy |
| Requirements review | never — code only | /verdict:spec judges the spec before code exists (contradictions, unmeasurables, boundary ambiguities, history conflicts) — with its own scored eval fixture |
| "No bugs found!" | frequently | never — coverage, gaps, and residual risk instead |
| AI-authored code | same checklist as human code | provenance measured (trailer census over the range, profile authorship); a pattern catalog with a procedure and evidence bar per entry (docs/ai-authored-code.md); deterministic censuses for hallucinated imports, placeholders and swallowed errors feed judgment — and a scored fixture proves the behaviours |
| Its own accuracy | unmeasured, and unmeasurable after the fact | every finding states a confidence when filed; the outcome is computed from what the finding did, kept in a permanent ledger, and reported as a track record the harness computes rather than the tester |
| Tested itself | — | scored eval suite: baseline + delta-memory + adversarial-honesty fixtures, deterministic scorer, published answer keys (eval/) |
| State consumable by other tools | — | verdict-mcp: read-only MCP server over the state — works from Cursor, Codex, CI, any MCP client |
The tested tester
A QA agent that was never tested is exactly the kind of claim it should reject.
eval/ is a scored eval suite with a deterministic scorer —
score.py reads the state file, not the prose — and eight fixtures, the four that carry the headline claims:
- Baseline (fixtures/pricer): 8 seeded issues covering all five failure classifications, including a boundary defect hidden behind a "temporarily" skipped test and a stale expectation whose intent citation sits in the CHANGELOG. Answer key.
- Delta (fixtures/pricer_rev_b): scores the flagship —
a run against an authored run-2 history must produce
REGRESSED(ranked first),NEW,STILL_OPEN,RESOLVED, and release an expired quarantine, while a CHANGELOG decoy tries to launder the new defect as intended. Answer key. - Liar (fixtures/liar): adversarial honesty — a test script that prints "ALL TESTS PASSED" unconditionally, a conftest that skip-marks the whole suite, a mock asserting its own return value, a tautological assertion. Scores whether the verdict takes output at face value.
- Spec (fixtures/refund-spec): shift-left — a draft PRD
with a seeded contradiction, an unmeasurable requirement, an exactly-at-the-boundary
ambiguity, a silent failure-path gap, and a CHANGELOG that contradicts the spec. Scores
/verdict:specfinding them all before any code exists.
python3 eval/run_eval.py --mode seeded|live|baseline runs it all in an isolated scratch
repo and scratch state home. Results are published as measured; misses — and any answer-key
amendment — stay in the table (eval/README.md).
Every one of those keys was written by the hands that wrote the prompt. The external key
(eval/swebench.py) is one nobody here chose: SWE-bench Verified
instances — real defects, each fixed by its own maintainers with a test that fails before the
fix — every instance from the five smallest repositories in the set (pytest, pylint, requests,
seaborn, flask; 40). The checkout's history ends at the bug's base commit, the environment is
the one the maintainers had that week, the withheld test never enters the tree, and the issue
text is the whole charter. The score is location, deterministic: does a path:line the
finding cites fall in the file the fix touched, inside its hunk, in the same function? Rate,
time, tokens and every miss are in the
ledger.
State modes
- Solo (default): state lives in
~/.claude/verdict/<repo-name>/— nothing added to your repo. - Team: create
.qa/in the repo (/verdict:baseline team) and commit it — your teammates and CI share the same baseline, and QA reports travel with the code. Commit the record, not what a single run passed through:.qa/.gitignorelistsfacts.json,judgment.json,state.json.prev,run-in-progress.json,findings.prev/andusage.jsonl. The last is the bill — what each run spent on the machine that ran it, and for the local tier the address of the gateway it called./verdict:baseline teamwrites the file; a.qa/created by an earlier version needs it added by hand.
The state schema is documented in docs/state-schema.md — versioned, forward-compatible, human-readable JSON.
Commands
Plugin commands are namespaced by the plugin and must be typed in full — /verdict:run,
/verdict:status. There is no short form: a bare /verdict is an unknown command,
measured rather than assumed.
Start here: /verdict:run — the front door. It reads the tester's memory and picks the
right pass itself: no state yet → a baseline; state present → today's delta; arguments
given → a delta narrowed to what you named. It says which it chose and why. Every command
below is the same machinery aimed at one specific job, for when you already know which
job you want.
| Command | What it does |
|---|---|
/verdict:run |
Front door — routes to baseline, delta, or a scoped review from the stored state |
/verdict:baseline |
Initialize the QA root, project profile, and baseline state |
/verdict:regression |
Regression checklist: changed area → adjacent flows → integrations |
/verdict:release |
Release gate with the four-verdict contract |
/verdict:bug |
Turn a symptom/log/complaint into a classified, structured bug report |
/verdict:flake |
Classify an intermittent failure: ≥3 reproductions, mechanism hunt → BRITTLE_TEST fix task, or FLAKY quarantine with expiry |
/verdict:status |
Read-only status from the stored state — no run, no writes, no agent spin-up |
/verdict:spec |
Shift-left: judge a spec/issue/PRD for testability before code exists — contradictions, unmeasurables, undefined boundaries, silent gaps, history conflicts, plus Given/When/Then criteria |
/verdict:cause |
Trace a failure to its root cause: symptom → mechanism → origin → class, each link cited, causation proven by counterfactual rather than narrated; trigger, cause, and latent condition kept apart |
/verdict:charter |
Timeboxed exploratory charter with a risk focus seeded from the profile's incident history; observations captured as evidence, discoveries converted to bug reports and regression candidates |
The tester's memory, over MCP (optional)
Verdict's state isn't locked inside the agent. verdict-mcp is a small read-only MCP
server over the same state files, so anything that speaks MCP can consult your QA memory —
an orchestrator gating a merge, a Cursor or Codex session, a CI step commenting a PR:
| Tool | Returns |
|---|---|
get_verdict(project) |
last verdict, release blockers, report path, not-tested list |
get_findings(project, status) |
open (default), all, or NEW / STILL_OPEN / RESOLVED / REGRESSED — REGRESSED ranked first |
get_quarantine(project) |
the flaky ledger, each entry with a computed expired flag |
get_questions(project) |
the questions the tester parked for a person, each with its age, and answers no run has read yet — answer with verdict-answer |
get_history(project) |
run-over-run trend parsed from the report INDEX |
get_report(project, report?) |
full report content (default: last run's) — path-guarded to the QA root, so a CI step can quote the evidence, not just link it |
get_profile(project) |
the project's QA profile: isolation rules, risk areas, real test commands — plus the lessons ledger when one exists |
get_trends(project) |
run-over-run trajectory from the INDEX, the current pressure picture (open by severity, age distribution, quarantine size, duration), and hotspots — where this project's defects actually cluster, computed from its own findings and severity-weighted, with the number of runs behind the ranking |
list_projects() / get_state(project) |
everything with a baseline / the raw state |
claude mcp add verdict -- uvx --from verdict-qa-mcp verdict-mcp
The distribution is verdict-qa-mcp — the console script and the import package are
still verdict-mcp / verdict_mcp; only the name PyPI indexes differs, because
verdict-mcp there belongs to an unrelated project. Installing straight from the
repository also works and needs no release:
uvx --from git+https://github.com/ArtJack/verdict verdict-mcp
project is a key from the solo root (~/.claude/verdict/, override with VERDICT_HOME)
or a repo path in team mode (resolves <repo>/.qa/). Every tool carries a read-only
annotation and the server never writes — the tester's memory is public API; the tester's
pen is not. Needs uv (or pipx install verdict-qa-mcp); the plugin itself still has
zero dependencies and works without the server.
Verdict QA MCP questions
How do I add Verdict QA MCP to Claude Code?
Run claude mcp add verdict -- uvx --from verdict-qa-mcp verdict-mcp in your terminal. Add --scope user to make it available in every project.
How do I add Verdict QA MCP to Cursor?
Open ~/.cursor/mcp.json and add the server under mcpServers, using the Cursor config shown in the install section above.
What tools does Verdict QA MCP provide?
It exposes 10 tools, including list_projects, get_verdict, get_findings, get_quarantine, get_questions.
Does Verdict QA edit my code?
No, it is a read-only agent with no Edit tool and strict-mode guards to ensure it remains an independent tester.
What are the possible verdict outcomes?
Every run results in one of four specific outcomes: pass, pass with risks, blocked, or fail.