mispick MCP server

mispick

Find out which of your MCP tools an AI model mixes up.

TL;DR

• The problem. An AI assistant picks which tool to use by reading each tool's short description. If two descriptions sound alike, it quietly picks the wrong one, and nothing tells you.
• What mispick does. It writes realistic test requests for every tool, asks a real model to pick a tool for each one, and counts what it picked. You get a table of which tools get mistaken for which (a confusion matrix), plus a 0–100 score.
• It's safe. It only reads the tool list. It never runs a tool, so you can point it at a server that deletes things.
• It can suggest fixes, and it re-tests each suggested rewrite. It keeps only the ones that measurably helped.
• Free by default. It runs on a local model through Ollama. Cloud models are optional.
• Status: v0.2.0 is on PyPI. A public benchmark of real MCP servers is running now. A research study built on it is pre-registered, with first results in.

uvx mispick run --cmd "python -m my_server"

How to read the report. Each row of the table is a tool. Each column is what the model actually picked when that tool was the right answer. A perfect server has every count on the diagonal. Anything off the diagonal is a mix-up worth fixing. Every percentage comes with a range (a 95% confidence interval), because with few test requests a single number can mislead.

---

What the benchmark found

We ran mispick on 14 real MCP servers from the official registry: 3,636 test requests on Gemini 3.5 Flash-Lite. Full results: docs/study.md.

• Most requests go to the right tool, but not all: 85% did.
• Mistakes are concentrated: only 107 of 6,024 tool pairs were ever mixed up, so a server usually has a few specific problem pairs rather than general confusion.
• Results are repeatable: 97% of requests got the same answer in all three runs.
• Overlapping names and similar descriptions go with more mix-ups, but with 13 servers that evidence is fragile.
• The clearest tip for server authors: say when to use each tool. Tools whose description does were less often mistaken for others.
• One server could not be used with Gemini at all, because of a broken schema in its tool list.

Three real servers, as confusion matrices. Each row is the tool a request was meant for, and each column is what the AI picked. Numbers on the diagonal are correct.

• ai.wavespeed/mcp, 60% correct. Every request to run a model went to list_models, and so did most requests to get a price.

• Memory (MCP reference server), 67% correct. Edit and delete requests often went to search_nodes first. The model looked up the entry before changing it, which is sensible, but mispick scores the first call, so it counts as a miss.

• Acquisition.gov, 100% correct. Five clearly separated tools and no mix-ups. This is what a good tool surface looks like.

Every number here comes from results/: headline.json, analysis.json, and the frozen run in analysis/data/benchmark-v1/. Results are for one model on one date; see the limitations below.

Why this rather than a static linter

mcp-lint, mcpolish, oh-my-mcp, mcp-conform and mcp-cve-lint check tool definitions against rules, and they are good at it — run one. But a description can pass every rule and still be indistinguishable from the tool next to it. Rule-based linters predict problems; this measures them, by putting your tool list in front of a real model and counting what it picks. Two other projects do measure selection — whichtool (local models, but no rewrites, one server per run) and toolfit (rewrites, but cloud-only, one server per run). Neither looks across servers. The full comparison, with what each one actually does, is in docs/research.md.

How it works

1. Connect to your server and read tools/list (never tools/call).
2. For each tool, generate N realistic requests — plus hard negatives written to sit as close as possible to its nearest neighbour, and some requests no tool should answer.
3. Ask the model to choose a tool for each request, K times, showing it the real tool definitions a client would send.
4. Score it: accuracy, per-tool precision and recall, a confusion matrix, argument validity, stability across the K runs — every rate with a Wilson 95% interval.
5. Propose rewrites for the worst pairs, re-run the same requests against them, and keep only the ones that measurably won.

Install

uvx mispick --help          # no install
uv tool install mispick     # or keep it
pip install mispick

The default backend is a local model through Ollama, so it costs nothing and needs no account:

ollama pull qwen3.5:4b

Cloud models are optional, with the matching key in the environment:

mispick run --snapshot tools.json --model openai/gpt-4.1-mini            # OPENAI_API_KEY
mispick run --snapshot tools.json --model gemini/gemini-3.5-flash-lite   # GEMINI_API_KEY
mispick run --snapshot tools.json --model anthropic/claude-haiku-4-5-20251001

# anything else that speaks the OpenAI dialect - vLLM, OpenRouter, LM Studio
MISPICK_BASE_URL=http://localhost:8000/v1 mispick run --snapshot tools.json \
  --model compatible/my-model

On cost. Every selection call re-sends the whole tool list, so a cloud run is priced by tool-list size × queries × runs, not by the number of servers. Measuring one real 64-tool server at N=8 K=3 is about 30M input tokens on its own. --dry-run is not implemented yet; until it is, read the token estimate mispick prints and multiply.

Use

# measure a local server, a remote one, or a captured snapshot
mispick run --cmd "python -m my_server"
mispick run --url https://example.com/mcp
mispick run --snapshot tools.json

# capture a snapshot so later runs are offline and reproducible
mispick snapshot --cmd "python -m my_server" -o tools.json

# every server in your config at once, and what they do to each other
mispick run --config ~/Library/Application\ Support/Claude/claude_desktop_config.json

# rewrites for the worst pairs, each one re-tested
mispick fix --cmd "python -m my_server" --patch fix.diff

# reports and CI
mispick run --snapshot tools.json --format html --out report.html --badge badge.svg
mispick run --snapshot tools.json --fail-under 80          # exit 1 below the threshold
mispick compare base.json head.json --max-drop 5           # exit 1 on a regression

Generated requests are cached in .mispick/queries.yaml. Read that file and edit it — it is your test set, and it only regenerates for a tool whose name, description or schema changed. Commit it.

Proven fixes

mispick fix rewrites both descriptions in a confused pair, then re-runs the same requests against the rewrite and reports before/after with an exact McNemar test — the trials are paired, so an unpaired test would be the wrong one. Rewrites that fail are shown too.

That re-test is the whole point. Hasan et al. rewrote tool descriptions with an LLM across 231 tasks and found success rose by a median 5.85 points — but regressed in 16.67% of cases (arXiv 2602.14878). An unmeasured rewrite is a one-in-six chance of making things worse. mispick also caps rewrite length, so a rewrite cannot win by crowding out its neighbours, and it never edits your source — it prints a suggestion.

Expect rejections, especially on a small local model. In our own runs against ollama/qwen3.5:4b, two of three proposed rewrites measurably made selection worse — one of them took a tool from 78% to 58% accuracy, breaking 14 trials and fixing none, and it replicated when re-run with three times the data. Its stated reasoning read perfectly sensibly. The third improved by 10 points but over only two changed trials, which is not enough to call, so mispick reported it as inconclusive rather than banking it. A 4B model is good enough to find confusion and often not good enough to write its way out of it; fix mode is more productive with a larger model. Either way the verdicts are the product — a run where nothing is accepted has told you something true.

Across servers

Point --config at a whole claude_desktop_config.json and mispick loads every server at once, because that is how a client sees them. It reports which bare tool names collide across servers and — the part that matters — how often a request for one server's tool is answered by another server's. On the two-server fixture, 17% of trials cross servers and workspace:search_docs falls to 0% recall: every request for it is answered by wiki:search_docs. The MCP spec names this problem and warns that serverInfo.name is not unique across servers, so mispick keys on your config label instead.

In CI

- uses: actions/checkout@v4
- uses: msc35/mispick-action@v1
  with:
    snapshot: tools.json
    model: anthropic/claude-haiku-4-5-20251001
    fail-under: '80'
    compare-base: 'true'
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

It posts the report as a sticky PR comment, and with compare-base the comment reads 84 → 91 (▲7) and names the confusions that appeared or disappeared. With no model key it skips with an explanation instead of failing — a check that goes red over a missing secret is a check people learn to ignore. See examples/github-action. The Action also lives on its own in msc35/mispick-action, for the GitHub Marketplace.

Benchmark

benchmark/ measures real servers from the official MCP registry that start without an API key, and builds a static leaderboard from the results.

• Now running. A daily GitHub Actions job measures 15 servers on Gemini 3.5 Flash-Lite's free tier. It needs about 11 days because of the free tier's daily limit, and it resumes where it stopped each day.
• Reproducible. The captured tool lists are committed in benchmark/snapshots/.
• Safe. Only initialize and tools/list are ever sent.
• Honest about coverage. The page also lists the servers it could not measure, so coverage is not overstated.

See benchmark/README.md and the run plan in docs/benchmark-plan.md.

Research study

The benchmark data also feeds a small study. Its analysis plan was written and committed before looking at any results, so the questions can't be bent to fit the answers. The study asks three questions:

1. What makes two tools easy to confuse? For example, similar descriptions, overlapping names, or missing "use this when…" guidance.
2. Does a cheap model's confusion predict a stronger model's?
3. Do generated test requests give the same answers as hand-written ones?

First results are in. Similar tool names go with more mix-ups, but with 13 servers the evidence is fragile. The clearest signal: a tool whose description says when to use it gets its requests taken by other tools less often. Plan: docs/analysis-plan.md. Results: docs/study.md.

Limitations

Read these before quoting a number.

• Results depend on the model. A score is one model's reading of one tool surface on one day, not a property of your server. Every report says so, and states the model, N, K and the date.
• Small local models are noisy. That is why every rate carries a Wilson 95% interval and why --runs exists. With K=3 the intervals are wide; read them, not the point estimate.
• The requests are generated, so they are not your traffic. They are a stand-in for it. Edit .mispick/queries.yaml to make them yours.
• It measures one selection turn. Not multi-step agent behaviour, not tool output quality, not recovery after a bad call. Arguments are checked against inputSchema and no further.
• A low score can be correct. Two tools may genuinely overlap. mispick tells you the model cannot separate them; whether they should be separable is your call.
• mispick fix often accepts nothing, and that is working as intended. See the note above.
• Token counts are estimates, and lower bounds — a real tokenizer and the provider's framing both add to them.
• Reasoning models spend heavily on hidden thinking. mispick turns thinking off by default for Ollama (measured: 89s → 2.2s per call on qwen3.5:4b, same answers); --think restores it.

Development

uv sync --all-extras                  # --all-extras so mypy also checks the Anthropic backend
uv run pytest                         # offline; no model or network needed
uv run ruff check . && uv run mypy src
uv run mispick run --snapshot tests/fixtures/confusing_tools.json --model mock

--model mock is a deterministic offline backend. The test suite uses it, so pytest passes with no Ollama, no keys and no network.

License

MIT. See LICENSE.