cap-evolve MCP server
Commit Nov 13, 2026Optimizes an AI agent's prompts, tools, MCP surfaces, and skills against your own evaluations using a validation-gated evolve loop.
Also known as skillberry-ai/cap-evolve
- Stars
- 70
- Tools
- —
- Downloads, 30 days
- —
- Latest version
- —
- Last commit
- Nov 13, 2026
- License
- Apache-2.0
What is cap-evolve MCP?
cap-evolve optimizes an AI agent's prompts, tools, MCP surfaces, and skills against your own evaluations using a validation-gated evolve loop.
Summary generated from the project's README
About cap-evolve MCP server
Improving an agent's system prompt, tool code, MCP tool surface or skill package against an evaluation you already own is the job cap-evolve takes on. You supply the agent and its eval; the tool runs an evaluate, diagnose, propose, gate and commit loop, editing what the agent reads rather than model weights. It is aimed at developers who have failed trajectories on hand and want one honest score plus an inspectable record of each change.
Installation goes through pip on the core package (Python 3.10 or newer, zero runtime dependencies). The CLI offers init to scaffold a project and write capevolve.yaml, doctor as a readiness check that names the fix under each failing row, algorithms to list the five options with the spec lines for picking one, diff to compare the seed against the winning candidate or a candidate against its parent, and watch, replay or run --tui for live views. A run also writes a self-contained dashboard.html. Setup paths cover a Claude Code plugin, other coding-agent hosts via install.sh --host, or a manual adapter; the bundled toy_calc example needs no API key.
What sets it apart is the acceptance rule: a validation-only significance gate (delta greater than k standard errors), with the test split sealed and scored once. Every candidate is a git commit, and the dashboard shares one projection with the terminal, so costs, diffs, lineage and a tasks by iterations heatmap cannot disagree with the log.
What you can do with it
- 1Check readiness before a paid runAsk the client to run the readiness check and report which prerequisites are missing plus the command that fixes each one, before any evaluation budget is spent.
- 2Read the edit that moved the numberHave the client diff the seed prompt against the winning candidate so you can see exactly which lines the optimizer changed and in which files.
- 3Pick an algorithm from the specAsk which of the five algorithms fits your setup and get the specific spec lines needed to select it.
- 4Optimize tool code and prompts jointlyHand over an agent whose failures mix missing rules and weak tool code, and have both capabilities edited together in one loop.
- 5Inspect a run without an API keyLet the client replay a bundled recording to confirm the dashboard layout and gate reasons before wiring your own adapter.
Written from the project's README. Every use case is checked against a quote from it.
Automated checks
Ran Nov 14, 2026Full report →Not verified yet. A server needs a score of 80, no serious finding, and enough data for us to check. How checks work · What to fix for more points
- Source repository linked and reachableskillberry-ai/cap-evolve is public and reachablegithub.com/skillberry-ai/cap-evolve ↗Pass
- Package names this repository as its sourceNot checked
- Build provenance (Sigstore attestation)Not checked
- Open source license (OSI)Apache-2.0spdx.org/licenses/Apache-2.0.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-11-13github.com/skillberry-ai/cap-evolve ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNot checked
- No critical advisory in the package or its dependenciesNot checked
- No hidden instructions to the model0 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNot checked
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubcap-evolve improves an AI agent's prompts, tools, and skills by learning from failed evaluation traces.
You bring the agent and the eval you already have. cap-evolve runs the loop — evaluate → diagnose the failures → propose an edit → keep it only if it beats a held-out split by a significant margin → commit — and reports one honest number. It optimizes what your agent reads, not its weights.
Why cap-evolve
- Optimize more than prompts. System prompts, executable tool code, MCP tool surfaces, and whole skill packages — pick one or several and optimize them jointly.
- Learn from real agent failures. Every iteration reads full trajectories and per-task causal feedback (which task ids a prior edit broke and fixed), so edits are large and don't regress the wins.
- Keep evaluation honest. Acceptance is a val-only significance gate (Δ > k·SE); the test split is sealed and scored exactly once. Both live in the core, not in editable docs.
- Inspect every change. Each candidate is a git commit; the dashboard shows costs, timing, diffs, lineage, and a tasks × iterations pass/fail heatmap.
Try it in two minutes — no API key required
toy_calc is a deterministic stand-in agent that only answers correctly when its system
prompt contains a [CALC] marker. The mock optimizer adds it, so the score provably
rises — no model is called.
git clone https://github.com/skillberry-ai/cap-evolve.git
cd cap-evolve
python3 -m venv .venv && source .venv/bin/activate
pip install ./core # package: cap-evolve-core · CLI: cap-evolve · zero runtime deps
bash benchmarks/toy/examples/calc/run.sh
Expected — the seed prompt scores 0.0 on val; the optimized prompt is gate-accepted and
scores 1.0 on the sealed test split:
baseline_val 0.0 -> test_reward 1.0 (gate-accepted, test sealed) + dashboard.html
Open the printed dashboard.html in any browser. Full walkthrough:
Getting started.
The CLI
Start with no arguments — cap-evolve prints a branded home screen with the golden path and
every command grouped by what it's for.
cap-evolve # home: the 3-step path + all commands
cap-evolve init # scaffold a project and write capevolve.yaml
cap-evolve doctor # readiness check: what's missing + the command that fixes it
cap-evolve algorithms # the five algorithms and the exact spec lines to pick one
cap-evolve help <command> # full help with copy-paste examples
doctor is the one to run before spending anything. Every failing row names the fix:
See what actually changed
Every candidate is a snapshot, so you can read the edit that moved the number — unified below 120 columns, side-by-side above:
cap-evolve diff --best # seed → the winning candidate
cap-evolve diff cand_0003 # against its parent
cap-evolve diff cand_0003 --stat # just the per-file +/- counts
Watch a run, live
cap-evolve watch # live view of the newest run
cap-evolve replay --demo # no API key, no config — replays a bundled recording
cap-evolve run --tui # the live view instead of the line log
cap-evolve watch --diff # …and show what each accepted candidate changed
The dashboard
cap-evolve dashboard # live, over a base dir of runs
Every run gets the same tabs whatever algorithm produced it — Overview, Candidates, Gate,
Tasks, Cost, Logs, Diffs, Trajectories, Memory, Files — and an algorithm that has extra signal
gets an extra tab rather than a different dashboard. GEPA's minibatch-vs-full-val gates and
Pareto selection, SkillOpt's epochs and edit-budget schedule, and
agent-optimize's free-form rounds are all read from events the engine already emitted.
The dashboard and the terminal are the same projection (events.jsonl → reduce_run), so they
cannot disagree about what happened. run also writes a self-contained dashboard.html that
needs no server.
Choose your path
| Path | Use it when | Start |
|---|---|---|
| Claude Code plugin | You use Claude Code and want slash commands + honesty hooks | claude --plugin-dir ./plugins/cap-evolve then follow RUN.md |
| Another coding-agent host | Codex, Gemini, opencode, Cursor, Droid, Copilot, Kimi, Pi, Antigravity, openclaw, IBM Bob, bare | ./install.sh --host <name> then follow RUN.md |
| Manual adapter + CLI | You want to wire the adapter yourself and drive cap-evolve directly |
Optimize your own agent |
Each path shares the same core install and the same honesty guarantees. Full setup, credentials, and the optional dashboard: Installation.
What can cap-evolve optimize?
| Capability | What the optimizer may change |
|---|---|
| System prompts | Rewrite / consolidate / add rules, examples, output contracts — never drop a needed rule |
| Tool implementations | Edit tool code for deterministic enforcement; add/wrap/swap tools (never bare-remove) |
| MCP tool surfaces | Safe edits only — tool docs, in-description examples, and which tools are exposed |
| Skill packages | An Agent Skill dir — SKILL.md bodies, references, and executable scripts |
Combine them, e.g. [system-prompt, tools]. See Architecture.
Results
Each result is labeled fit metric (no holdout) or held-out (test scored once on ids the optimizer never saw). Full detail, models, task/trial counts, commits, and costs: docs/RESULTS.md. Every row is cross-checked against a committed run artifact except RH-SWE-bench, whose artifact is not in this repo — see the caveats in docs/RESULTS.md before quoting it.
| Benchmark | Split | Baseline → Optimized | Gain |
|---|---|---|---|
| RH-SWE-bench (skill-package + system-prompt, Harbor) | val — fit metric (119 tasks) | 0.580 → 0.765 |
+0.185 / +31.9% |
| toy_calc (zero-API) | sealed test | 0.0 → 1.0 |
deterministic proof |
| τ²-bench airline (policy + tools) | val — fit metric | 0.536 → 0.712 |
+0.176 / +32.8% |
| τ²-bench airline, held-out 30(=val)/20 | sealed test | 30.0 → 47.5 |
+17.5 pp / +58.3% |
| SkillsBench (skill package) | sealed test (held-out) | 0.556 → 0.667 |
+0.111 / +20.0% |
At a glance — baseline → optimized across all benchmarks:
reward × 100
─────────────────────────────────────────────────────────────────────
RH-SWE-bench (119 tasks, fit metric) ●────────────● 58.0 → 76.5 +18.5 pp / +31.9%
τ²-bench airline (50 tasks, fit metric) ●──────────● 53.6 → 71.2 +17.6 pp / +32.8%
τ²-bench airline (20 tasks, held-out) ●──────────● 30.0 → 47.5 +17.5 pp / +58.3%
SkillsBench (3 tasks, held-out) ●──────● 55.6 → 66.7 +11.1 pp / +20.0%
─────────────────────────────────────────────────────────────────────
○ = baseline (seed) ● = optimized (best candidate)
Not an apples-to-apples leaderboard. For how the held-out τ²-bench result sits next to external tool-optimization work (EvoTool on the original τ-Bench, and Evolutionary Context Search), with defined criteria and caveats, see docs/COMPARISON.md.
How it works
flowchart LR
A[Prompt, tools, MCP, or skills] --> B[Run evaluation]
B --> C[Diagnose failures]
C --> D[Generate candidate]
D --> E[Validation gate]
E -->|Accepted| F[Git-versioned best candidate]
E -->|Rejected| C
F --> G[Final evaluation and report]
Each iteration receives the current best capability, its failed trajectories, per-task impact (what previous edits broke and fixed), and the history of previous attempts. It proposes one bold, multi-part candidate, evaluates it on val, and records whether the gate accepted it. The pipeline is intake → implement-and-check → baseline → algorithm → finalize → report; the exact optimizer-context files, run-dir layout, and honesty guarantees are in Architecture and Honest evaluation.
Use it with your own agent
Wire one small adapter — three required methods (plus optional hooks):
tasks(split) -> list[Task] # your eval cases for 'train'|'val'|'test'|'all'
run_target(task, ctx, *, seed) -> Rollout # run your agent with the candidate LIVE as ctx
score(task, rollout) -> Score # reward in [0,1] + feedback (never leak the gold)
Everything else — splits, trials, gating, pass^k, the sealed test, memory, and the dashboard — is provided by the core. Two ways to get there:
- Let your coding agent build it — open the agent you already use at the repo root and
tell it to follow
RUN.md. It runsintake, asks for anything missing, writes the adapter, passescap-evolve check, then runs the loop. - Do it yourself — implement the adapter and drive the CLI.
Both are walked through in docs/OPTIMIZE_YOUR_OWN.md; the contract is in docs/ADAPTER_CONTRACT.md. For common cases, don't write an adapter from scratch — copy a ready-made adapter template (JSONL, HuggingFace, tau2-bench, SWE-bench, SkillsBench) and switch providers with a one-line env change: docs/ADAPTER_TEMPLATES.md.
Examples
| Example | What it shows | Needs | Run |
|---|---|---|---|
toy_calc |
The full loop, deterministically | nothing | bash benchmarks/toy/examples/calc/run.sh |
tau2_airline |
Onboard a real benchmark from one prompt; optimize policy + tool code | gateway creds, Claude Code | bash benchmarks/tau2/examples/airline/setup.sh && bash benchmarks/tau2/examples/airline/run.sh |
skillsbench |
Optimize a skill package; agent runs in Docker | Docker, uv, Claude creds |
bash benchmarks/skillsbench/examples/shared_skills/setup.sh && bash benchmarks/skillsbench/examples/shared_skills/run.sh |
Each example's paste-to-agent brief is its PROMPT.md, its narrative is DEMO.md, and its
committed run is under run_full/. See the full interactive dashboard for the tau2 run with
no backend: cd benchmarks/tau2/examples/airline/run_full/ui && python3 -m http.server 8000. Reproduce
from zero: tau2 · SkillsBench.
Documentation
| Document | Use it when |
|---|---|
| Site (home) | You want the interactive site — hero, results, and doc navigation in one place |
| Getting started | You want your first successful run |
| Installation | You need host-specific setup, credentials, or the dashboard |
| Optimize your own agent | You want to integrate your agent or benchmark |
| Adapter templates | You want a copy-and-run adapter (JSONL, HuggingFace, tau2, SWE-bench, SkillsBench) |
| Adapter contract | You are implementing an adapter |
| Architecture | You want to understand the pipeline and optimizer context |
| Agent orchestration | You want the agent to drive the loop itself (orchestration_mode: agent, agent-optimize) |
| Honest evaluation | You need details on splits, gates, and sealing |
| Results | You want the full experiments and artifacts |
| Comparison | You want positioning vs other tools and external results |
| Extending cap-evolve | You are adding a capability, optimizer, or algorithm |
| Troubleshooting | Installation or a run failed |
| Roadmap | You want planned work |
| How-to guides | You want a specific harness + benchmark recipe |
| Docker/Podman on CCC | You are running on a cluster with no root, no sudo, no subuid range (IBM CCC), or submitting cap-evolve through LSF |
Project status and support
Beta (0.x). Contributions welcome — see CONTRIBUTING.md and the
Code of Conduct. Report security issues via SECURITY.md.
Changes are tracked in CHANGELOG.md.
Citation
@software{cap-evolve,
title = {cap-evolve: a skills-native, host-agnostic harness for honestly
optimizing AI-agent capabilities},
year = {2026},
note = {https://github.com/skillberry-ai/cap-evolve}
}
Acknowledgements. cap-evolve includes no third-party code — the gepa and
skillopt skills are independent implementations of the GEPA (arXiv:2507.19457) and
SkillOpt (arXiv:2605.23904) papers, and it draws on ideas from DSPy and Anthropic's
Agent Skills standard. The bundled example uses
tau2-bench (MIT). Full citations:
docs/sources.bib.
License
Apache-2.0.
cap-evolve MCP questions
What can cap-evolve optimize?
It can optimize system prompts, tool implementations, MCP tool surfaces, and whole skill packages, individually or jointly.
Does cap-evolve require an API key to try it?
No. The toy_calc demo uses a deterministic mock optimizer and calls no model.
How does cap-evolve keep evaluation honest?
Candidates must beat a held-out validation split by a significance margin, and the test split is scored only once.
How do I see what changed between candidates?
Use the diff command to compare the seed against the winning candidate or a candidate against its parent.
What can I run before spending money?
Run the doctor command for a readiness check; each failing row names the command that fixes it.