# eval-runner MCP server

> Provides a standardized testing harness to run regression suites and validate MCP server tool calls using YAML fixtures.

- Repository: https://github.com/dbsectrainer/mcp-eval-runner
- Page: https://hostedmcp.org/servers/dbsectrainer/mcp-eval-runner
- Transport: stdio
- Stars: 0
- License: MIT
- Latest version: 1.0.1

## Install

### Claude Code

```sh
claude mcp add --env 'YOUR_API_KEY=<YOUR_YOUR_API_KEY>' mcp-eval-runner -- npx -y mcp-eval-runner
```

### Cursor (`~/.cursor/mcp.json`)

```json
{
  "mcpServers": {
    "mcp-eval-runner": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-eval-runner"
      ],
      "env": {
        "YOUR_API_KEY": "<YOUR_YOUR_API_KEY>"
      }
    }
  }
}
```

### VS Code (`.vscode/mcp.json`)

```json
{
  "servers": {
    "mcp-eval-runner": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "mcp-eval-runner"
      ],
      "env": {
        "YOUR_API_KEY": "<YOUR_YOUR_API_KEY>"
      }
    }
  }
}
```

### VS Code CLI

```sh
code --add-mcp '{"name":"mcp-eval-runner","type":"stdio","command":"npx","args":["-y","mcp-eval-runner"],"env":{"YOUR_API_KEY":"<YOUR_YOUR_API_KEY>"}}'
```

### Claude Desktop (`claude_desktop_config.json`)

```json
{
  "mcpServers": {
    "mcp-eval-runner": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-eval-runner"
      ],
      "env": {
        "YOUR_API_KEY": "<YOUR_YOUR_API_KEY>"
      }
    }
  }
}
```

### Codex

```sh
codex mcp add mcp-eval-runner --env 'YOUR_API_KEY=<YOUR_YOUR_API_KEY>' -- npx -y mcp-eval-runner
```

### Codex config.toml (`~/.codex/config.toml`)

```toml
[mcp_servers.mcp-eval-runner]
command = "npx"
args = ["-y", "mcp-eval-runner"]

[mcp_servers.mcp-eval-runner.env]
YOUR_API_KEY = "<YOUR_YOUR_API_KEY>"
```

### Gemini CLI

```sh
gemini mcp add -s user -e 'YOUR_API_KEY=<YOUR_YOUR_API_KEY>' mcp-eval-runner npx -- -y mcp-eval-runner
```

### Gemini settings.json (`~/.gemini/settings.json`)

```json
{
  "mcpServers": {
    "mcp-eval-runner": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-eval-runner"
      ],
      "env": {
        "YOUR_API_KEY": "<YOUR_YOUR_API_KEY>"
      }
    }
  }
}
```

## Environment variables

| Name | Required | Secret | Description |
|---|---|---|---|
| `YOUR_API_KEY` | yes | yes | Your API key for the service |

## Tools (10)

- `run_suite`: Execute all fixtures in the fixtures directory. Returns a pass/fail summary. Execution modes: Live mode: fixture has a 'server' block with command/args — spawns the server and calls tools via real MCP stdio transport. Simulation mode: no server block — assertions run against expected_output fields. Assertion types supported in fixture expect blocks: output_contains: "substring" output_not_contains: "substring" output_equals: "exact text" output_matches: "regex" tool_called: "tool_name" latency_under: 500 schema_match: { type: "object", required: ["result"], properties: { result: { type: "string" } } } Step output piping: use {{steps.<step_id>.output}} in input values to reference the output of a previous step.
- `run_case`: Run a single named test case from the fixtures directory. Assertion types supported in fixture expect blocks: output_contains: "substring" output_equals: "exact text" tool_called: "tool_name" latency_under: 500 schema_match: { type: "object", required: ["id"], properties: { id: { type: "number" } } }
- `list_cases`: Enumerate all available fixtures with their step counts and assertion types. (read-only)
- `create_test_case`: Create a new YAML fixture file in the fixtures directory. Supported assertion types in each step's expect block: output_contains: "substring" — checks output includes this text output_equals: "exact" — checks exact output match tool_called: "tool_name" — verifies which tool was called latency_under: 500 — asserts response time under N ms schema_match: { type: 'object', ... } — validates output JSON against schema
- `regression_report`: Compare current state to the last run and return what changed (regressions, fixes, new cases). (read-only)
- `compare_results`: Diff two named run results by run ID. Shows regressions, fixes, new and removed cases. (read-only)
- `generate_html_report`: Generate a full single-file HTML report for a completed run. The report includes: suite summary (pass/fail counts, duration), per-case drill-down with per-assertion pass/fail details, and color-coded status (green=pass, red=fail, yellow=error). Use run_suite or run_case first, then pass the returned run_id here. (read-only)
- `scaffold_fixture`: Generate a boilerplate YAML fixture file in the fixtures directory. Each tool name provided becomes a documented step with placeholder input parameters and all supported assertion types pre-filled as comments. Example: scaffold_fixture({ name: 'search_test', tool_names: ['search', 'summarize'] }) Supported assertion types (pre-filled as TODOs in the generated file): output_contains, output_equals, tool_called, latency_under, schema_match
- `evaluate_deployment_gate`: CI gate — evaluate whether recent eval runs meet a minimum pass rate threshold. Fails (passed: false) if the pass rate of recent runs drops below min_pass_rate. Use this in CI pipelines to block deployments on regressions. (read-only)
- `discover_fixtures`: Discover fixture files across one or more directories. Returns a list of fixtures with their names, paths, and step counts. If no dirs are provided, uses the configured fixtures directory and FIXTURE_LIBRARY_DIRS env var (colon-separated list of paths). (read-only)

## FAQ

### What kind of assertions are supported?

Supported assertions include checking if output contains or does not contain text, exact matches, regex patterns, tool call verification, latency thresholds, and JSON schema validation.
