# evalpair MCP server

> Run LLM evaluation suites against local or hosted models and compare performance runs with statistical noise floor analysis.

- Repository: https://github.com/Srikarmk/evalpair
- Page: https://hostedmcp.org/servers/srikarmk/evalpair
- Transport: stdio
- Stars: 0
- License: MIT
- Latest version: 0.1.1

## Install

### Claude Code

```sh
claude mcp add --env 'EVALPAIR_API_KEY=<YOUR_EVALPAIR_API_KEY>' evalpair -- uvx evalpair
```

### Cursor (`~/.cursor/mcp.json`)

```json
{
  "mcpServers": {
    "evalpair": {
      "command": "uvx",
      "args": [
        "evalpair"
      ],
      "env": {
        "EVALPAIR_API_KEY": "<YOUR_EVALPAIR_API_KEY>"
      }
    }
  }
}
```

### VS Code (`.vscode/mcp.json`)

```json
{
  "servers": {
    "evalpair": {
      "type": "stdio",
      "command": "uvx",
      "args": [
        "evalpair"
      ],
      "env": {
        "EVALPAIR_API_KEY": "<YOUR_EVALPAIR_API_KEY>"
      }
    }
  }
}
```

### VS Code CLI

```sh
code --add-mcp '{"name":"evalpair","type":"stdio","command":"uvx","args":["evalpair"],"env":{"EVALPAIR_API_KEY":"<YOUR_EVALPAIR_API_KEY>"}}'
```

### Claude Desktop (`claude_desktop_config.json`)

```json
{
  "mcpServers": {
    "evalpair": {
      "command": "uvx",
      "args": [
        "evalpair"
      ],
      "env": {
        "EVALPAIR_API_KEY": "<YOUR_EVALPAIR_API_KEY>"
      }
    }
  }
}
```

### Codex

```sh
codex mcp add evalpair --env 'EVALPAIR_API_KEY=<YOUR_EVALPAIR_API_KEY>' -- uvx evalpair
```

### Codex config.toml (`~/.codex/config.toml`)

```toml
[mcp_servers.evalpair]
command = "uvx"
args = ["evalpair"]

[mcp_servers.evalpair.env]
EVALPAIR_API_KEY = "<YOUR_EVALPAIR_API_KEY>"
```

### Gemini CLI

```sh
gemini mcp add -s user -e 'EVALPAIR_API_KEY=<YOUR_EVALPAIR_API_KEY>' evalpair uvx -- evalpair
```

### Gemini settings.json (`~/.gemini/settings.json`)

```json
{
  "mcpServers": {
    "evalpair": {
      "command": "uvx",
      "args": [
        "evalpair"
      ],
      "env": {
        "EVALPAIR_API_KEY": "<YOUR_EVALPAIR_API_KEY>"
      }
    }
  }
}
```

## Environment variables

| Name | Required | Secret | Description |
|---|---|---|---|
| `EVALPAIR_API_KEY` | no | yes | Bearer token for an OpenAI-compatible server, if it needs one (OPENAI_API_KEY also works). |
| `EVALPAIR_HOME` | no | no | Folder for the call cache, saved runs and your own suites (default ~/.evalpair). |

## Tools (5)

- `list_suites`: Use to see which eval suites can be run: name, item count, checker and what each measures. (read-only)
- `list_runs`: Use to find earlier runs to show or compare: run id, suite, model, item count, whether it has a repeat pass. (read-only)
- `run_eval`: Use to run a suite against one model and save the run. Returns its score with an interval and its first failures. Calls the model; cached replies are reused.
- `show_run`: Use to see a saved run's score with its interval, the noise floor if it has a repeat, and the end of failed replies. (read-only)
- `compare_runs`: Use to compare two runs of the same suite item by item: score difference with a paired interval, flips each way, and whether answers moved more than the noise floor. (read-only)

## FAQ

### How can I run an evaluation using evalpair?

You can use the `run_eval` tool via MCP or use the CLI command `uvx evalpair run <suite> -m <model>` to execute a suite against a specific model.

### Can I create my own evaluation suites?

Yes, you can create custom suites by creating a folder in ~/.evalpair/suites/ containing a suite.toml file and a JSONL file with items.

### Does evalpair use LLMs to grade the answers?

No, it uses deterministic code-based checkers to ensure objective scoring without LLM bias.
