결정론적 규칙과 선택 사항인 LLM judge를 사용하여 에이전트 출력의 품질, 안전성 및 비용을 평가합니다.
다른 이름 iris-eval/mcp-server, mcp-server, @iris-eval/mcp-server
- 스타
- 10
- Tools
- 17
- 다운로드, 30일
- 1,137
- 최신 버전
- 0.21.0
- 마지막 커밋
- 2026년 11월 13일
- 라이선스
- MIT
Iris MCP란?
Iris은(는) MCP server입니다. 결정론적 규칙과 선택 사항인 LLM judge를 사용하여 에이전트 출력의 품질, 안전성 및 비용을 평가합니다. 17개 도구를 제공합니다. 예: log_trace, evaluate_output, get_traces.
프로젝트 README에서 생성된 요약
Iris MCP server 정보
Iris는 에이전트 출력의 품질, 안전성 및 비용을 점수화하여 에이전트 성능을 감사할 수 있는 방법을 제공합니다. 개발자는 수동 점검에 의존하는 대신 28가지 내장 결정론적 규칙을 사용하여 PII 유출, 프롬프트 인젝션, 환각(hallucination) 및 비용 임계값 초과와 같은 문제를 감지할 수 있습니다. 시스템은 모든 에이전트 실행 기록을 로컬 SQLite 데이터베이스에 저장하므로, OpenTelemetry 엔드포인트, webhook 또는 LLM judge를 명시적으로 활성화하지 않는 한 데이터의 프라이버시가 보장됩니다.
사용자는 Claude Desktop, Cursor, VS Code, Windsurf를 포함한 다양한 MCP 클라이언트에 Iris를 통합할 수 있습니다. 결정론적 규칙은 무료이며 로컬에서 실행되지만, 의미론적 평가 작업을 처리하기 위해 자체 API key를 제공하여 LLM judge를 선택적으로 활성화할 수 있습니다. 또한 localhost:6920에서 제공되는 대시보드를 통해 실패 사례를 시각화하고 서로 다른 에이전트 실행 기록을 비교할 수 있습니다. Iris는 자율 에이전트를 평가할 때 주관적인 인상보다 감사 가능한 수치가 필요한 개발자를 위해 설계되었습니다.
활용 사례
- 1에이전트 안전성 및 품질 감사에이전트에게 작업을 기록하도록 요청하고 결과 출력이 특정 규칙을 위반하는지 평가합니다.
- 2비용 및 환각 위험 모니터링결정론적 규칙을 사용하여 에이전트가 비용 임계값을 초과하거나 환각 마커를 생성하는지 확인합니다.
- 3HTTP를 통한 트레이스 수집 자동화LLM을 거치지 않고 POST 요청을 사용하여 에이전트 트레이스를 서버로 직접 전송합니다.
프로젝트의 README를 바탕으로 작성되었습니다. 모든 활용 사례는 해당 내용의 인용문과 대조하여 확인합니다.
Iris MCP 설치 방법
형식 확인: 2026년 11월 10일Cursor에 추가Goose에 추가Claude Code에 Iris MCP 설치
$ claude mcp add --env 'IRIS_ANTHROPIC_API_KEY=<YOUR_IRIS_ANTHROPIC_API_KEY>' --env 'IRIS_API_KEY=<YOUR_IRIS_API_KEY>' --env 'IRIS_OPENAI_API_KEY=<YOUR_IRIS_OPENAI_API_KEY>' --env 'IRIS_WEBHOOK_SECRET=<YOUR_IRIS_WEBHOOK_SECRET>' mcp-server -- npx -y @iris-eval/mcp-serverCursor에 Iris MCP 설치
~/.cursor/mcp.json
{
"mcpServers": {
"mcp-server": {
"command": "npx",
"args": [
"-y",
"@iris-eval/mcp-server"
],
"env": {
"IRIS_ANTHROPIC_API_KEY": "<YOUR_IRIS_ANTHROPIC_API_KEY>",
"IRIS_API_KEY": "<YOUR_IRIS_API_KEY>",
"IRIS_OPENAI_API_KEY": "<YOUR_IRIS_OPENAI_API_KEY>",
"IRIS_WEBHOOK_SECRET": "<YOUR_IRIS_WEBHOOK_SECRET>"
}
}
}
}VS Code에 Iris MCP 설치
.vscode/mcp.json
{
"servers": {
"mcp-server": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"@iris-eval/mcp-server"
],
"env": {
"IRIS_ANTHROPIC_API_KEY": "<YOUR_IRIS_ANTHROPIC_API_KEY>",
"IRIS_API_KEY": "<YOUR_IRIS_API_KEY>",
"IRIS_OPENAI_API_KEY": "<YOUR_IRIS_OPENAI_API_KEY>",
"IRIS_WEBHOOK_SECRET": "<YOUR_IRIS_WEBHOOK_SECRET>"
}
}
}
}VS Code CLI에 Iris MCP 설치
$ code --add-mcp '{"name":"mcp-server","type":"stdio","command":"npx","args":["-y","@iris-eval/mcp-server"],"env":{"IRIS_ANTHROPIC_API_KEY":"<YOUR_IRIS_ANTHROPIC_API_KEY>","IRIS_API_KEY":"<YOUR_IRIS_API_KEY>","IRIS_OPENAI_API_KEY":"<YOUR_IRIS_OPENAI_API_KEY>","IRIS_WEBHOOK_SECRET":"<YOUR_IRIS_WEBHOOK_SECRET>"}}'Claude Desktop에 Iris MCP 설치
claude_desktop_config.json
{
"mcpServers": {
"mcp-server": {
"command": "npx",
"args": [
"-y",
"@iris-eval/mcp-server"
],
"env": {
"IRIS_ANTHROPIC_API_KEY": "<YOUR_IRIS_ANTHROPIC_API_KEY>",
"IRIS_API_KEY": "<YOUR_IRIS_API_KEY>",
"IRIS_OPENAI_API_KEY": "<YOUR_IRIS_OPENAI_API_KEY>",
"IRIS_WEBHOOK_SECRET": "<YOUR_IRIS_WEBHOOK_SECRET>"
}
}
}
}Codex에 Iris MCP 설치
$ codex mcp add mcp-server --env 'IRIS_ANTHROPIC_API_KEY=<YOUR_IRIS_ANTHROPIC_API_KEY>' --env 'IRIS_API_KEY=<YOUR_IRIS_API_KEY>' --env 'IRIS_OPENAI_API_KEY=<YOUR_IRIS_OPENAI_API_KEY>' --env 'IRIS_WEBHOOK_SECRET=<YOUR_IRIS_WEBHOOK_SECRET>' -- npx -y @iris-eval/mcp-serverCodex config.toml에 Iris MCP 설치
~/.codex/config.toml
[mcp_servers.mcp-server] command = "npx" args = ["-y", "@iris-eval/mcp-server"] [mcp_servers.mcp-server.env] IRIS_ANTHROPIC_API_KEY = "<YOUR_IRIS_ANTHROPIC_API_KEY>" IRIS_API_KEY = "<YOUR_IRIS_API_KEY>" IRIS_OPENAI_API_KEY = "<YOUR_IRIS_OPENAI_API_KEY>" IRIS_WEBHOOK_SECRET = "<YOUR_IRIS_WEBHOOK_SECRET>"
Gemini CLI에 Iris MCP 설치
$ gemini mcp add -s user -e 'IRIS_ANTHROPIC_API_KEY=<YOUR_IRIS_ANTHROPIC_API_KEY>' -e 'IRIS_API_KEY=<YOUR_IRIS_API_KEY>' -e 'IRIS_OPENAI_API_KEY=<YOUR_IRIS_OPENAI_API_KEY>' -e 'IRIS_WEBHOOK_SECRET=<YOUR_IRIS_WEBHOOK_SECRET>' mcp-server npx -- -y @iris-eval/mcp-serverGemini settings.json에 Iris MCP 설치
~/.gemini/settings.json
{
"mcpServers": {
"mcp-server": {
"command": "npx",
"args": [
"-y",
"@iris-eval/mcp-server"
],
"env": {
"IRIS_ANTHROPIC_API_KEY": "<YOUR_IRIS_ANTHROPIC_API_KEY>",
"IRIS_API_KEY": "<YOUR_IRIS_API_KEY>",
"IRIS_OPENAI_API_KEY": "<YOUR_IRIS_OPENAI_API_KEY>",
"IRIS_WEBHOOK_SECRET": "<YOUR_IRIS_WEBHOOK_SECRET>"
}
}
}
}Iris MCP 도구 17
README에서 감지됨log_traceevaluate_outputget_traceslist_rulesdeploy_ruledelete_ruledelete_traceevaluate_with_llm_judgeverify_citationscompare_runscompare_tracesevaluate_runs
| log_trace | — | |
| evaluate_output | — | |
| get_traces | — | |
| list_rules | — | |
| deploy_rule | — | |
| delete_rule | — | |
| delete_trace | — | |
| evaluate_with_llm_judge | — | |
| verify_citations | — | |
| compare_runs | — | |
| compare_traces | — | |
| evaluate_runs | — |
도구 17개 모두 보기도구 수 줄이기
scorenametypeconfigweight
| score | — | |
| name | — | |
| type | — | |
| config | — | |
| weight | — |
구성
IRIS_ANTHROPIC_API_KEY비밀Enables evaluate_with_llm_judge and verify_citations with an Anthropic model. Optional: the deterministic rules never need a key
IRIS_API_KEY비밀API key for HTTP authentication. Required to bind the HTTP transport or the dashboard beyond loopback (0.0.0.0, a LAN address, a container): without it the server refuses to start
IRIS_OPENAI_API_KEY비밀Enables evaluate_with_llm_judge and verify_citations with an OpenAI model. Optional: the deterministic rules never need a key
IRIS_WEBHOOK_SECRET비밀The signing key of that webhook (any string, or whsec_ + base64); the iris format refuses to run without one
IRIS_ALLOWED_ORIGINSComma-separated Origin allowlist for the HTTP transport and dashboard CORS (default http://localhost:*)
IRIS_ALLOW_UNAUTHENTICATEDSet to 1 to run a non-loopback bind with NO API key on purpose (lifts the startup refusal; the network is then your boundary)
IRIS_API_KEY_FILEPath to a file whose trimmed contents are the API key (the secret-file pattern Docker and Kubernetes mount). Set this or IRIS_API_KEY, not both; further keys and rotation are security.apiKeys in config.json
IRIS_CITATION_ALLOW_FETCHLet verify_citations fetch URLs (off by default); a tool argument can turn it off for a call, never on
IRIS_CITATION_DOMAINSComma-separated domain allowlist for verify_citations fetches (only consulted when IRIS_CITATION_ALLOW_FETCH is on); a tool argument can narrow it, never add to it
IRIS_CITATION_MAX_COST_USD_TOTALThe most one verify_citations call may spend on judge calls, in USD (default 1); max_cost_usd_total can lower it, never raise it
IRIS_DASHBOARDStart the dashboard alongside the server (same as --dashboard)
IRIS_DASHBOARD_HOSTBind host for the dashboard (default 127.0.0.1); same as --dashboard-host
IRIS_DASHBOARD_PORTDashboard port (default 6920); IRIS_PORT is the MCP HTTP transport port (default 3000)
IRIS_DB_PATHSQLite database path
IRIS_HOMEDirectory for the database, custom rules and preferences (default ~/.iris)
IRIS_HOSTBind host for the HTTP transport (default 127.0.0.1); only used with --transport http
IRIS_LLM_JUDGE_DAILY_BUDGET_USDWhat every judge call together may spend per UTC day, per tenant, in USD (default 1): the relevance judge, evaluate_with_llm_judge and verify_citations. Past it no judge call is made
IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVALHard cap on the worst-case cost of one LLM judge call (default 0.25); max_cost_usd can lower it for a call, never raise it
IRIS_LOG_LEVELLog level: debug, info, warn, error
IRIS_MAX_INPUT_ELEMENTSThe most array elements and object keys one input may hold before it is validated (default 100000): a tool call's arguments, and a dashboard or OTLP JSON request body. A tool call over it gets an error result; a body, 413. OTLP protobuf has its own ceiling, one element per 5 bytes of the request size limit
IRIS_MAX_TOOL_INPUT_ELEMENTSThe name of IRIS_MAX_INPUT_ELEMENTS before 0.21.0, read when the new name is unset
IRIS_OTEL_ENDPOINTOTLP/HTTP collector URL; when set, every stored trace is also exported as OpenTelemetry spans (best-effort, asynchronous)
IRIS_OTEL_HEADERSComma-separated key=value headers sent with every OTLP export (for example an auth token for your collector)
IRIS_OTEL_SERVICE_NAMEservice.name resource attribute on exported spans (default iris-eval)
IRIS_OTEL_TIMEOUT_MSTimeout in milliseconds for each OTLP export request
IRIS_PORTMCP HTTP transport port (default 3000); only used with --transport http
IRIS_RELEVANCE_JUDGE_DAILY_BUDGET_USDThe name of IRIS_LLM_JUDGE_DAILY_BUDGET_USD before 0.21.0. Read when the new name is unset, and it then limits every judge call
IRIS_RELEVANCE_JUDGE_MAX_CALLS_PER_REQUESTRelevance judge calls one request may make (default 20)
IRIS_RELEVANCE_JUDGE_MODELA priced judge model; answers_the_ask then asks it on every evaluation with input, sending that input and output to the model's provider, and gates on its verdict (off by default)
IRIS_RELEVANCE_JUDGE_REDACTon (default) replaces the PII and credentials no_pii flags before the relevance judge sends input and output to its provider; off sends them as they are
IRIS_SEARCH_BUDGET_MSHow long one trace search may read before it answers with the matches found so far, in milliseconds (50-60000, default 1000)
IRIS_SEARCH_INDEXon (default) or off. off keeps no full-text index of the traces: writes are cheaper, and a search reads the traces within its time budget
IRIS_SQLITE_DRIVERWhich SQLite driver holds the database: native (better-sqlite3, the default) or node (Node's built-in node:sqlite, Node 22.13+). Unset: native, falling back to node when the native module cannot load
IRIS_TRANSPORTTransport: stdio (default) or http; same as --transport
IRIS_WEBHOOK_URLThe receiver of the webhook that fires on a moment (docs/webhooks.md); merged over notify.webhook in config.json
| 이름 | 설명 | 유형 |
|---|---|---|
| IRIS_ANTHROPIC_API_KEY | Enables evaluate_with_llm_judge and verify_citations with an Anthropic model. Optional: the deterministic rules never need a key | 비밀 |
| IRIS_API_KEY | API key for HTTP authentication. Required to bind the HTTP transport or the dashboard beyond loopback (0.0.0.0, a LAN address, a container): without it the server refuses to start | 비밀 |
| IRIS_OPENAI_API_KEY | Enables evaluate_with_llm_judge and verify_citations with an OpenAI model. Optional: the deterministic rules never need a key | 비밀 |
| IRIS_WEBHOOK_SECRET | The signing key of that webhook (any string, or whsec_ + base64); the iris format refuses to run without one | 비밀 |
| IRIS_ALLOWED_ORIGINS | Comma-separated Origin allowlist for the HTTP transport and dashboard CORS (default http://localhost:*) | |
| IRIS_ALLOW_UNAUTHENTICATED | Set to 1 to run a non-loopback bind with NO API key on purpose (lifts the startup refusal; the network is then your boundary) | |
| IRIS_API_KEY_FILE | Path to a file whose trimmed contents are the API key (the secret-file pattern Docker and Kubernetes mount). Set this or IRIS_API_KEY, not both; further keys and rotation are security.apiKeys in config.json | |
| IRIS_CITATION_ALLOW_FETCH | Let verify_citations fetch URLs (off by default); a tool argument can turn it off for a call, never on | |
| IRIS_CITATION_DOMAINS | Comma-separated domain allowlist for verify_citations fetches (only consulted when IRIS_CITATION_ALLOW_FETCH is on); a tool argument can narrow it, never add to it | |
| IRIS_CITATION_MAX_COST_USD_TOTAL | The most one verify_citations call may spend on judge calls, in USD (default 1); max_cost_usd_total can lower it, never raise it | |
| IRIS_DASHBOARD | Start the dashboard alongside the server (same as --dashboard) | |
| IRIS_DASHBOARD_HOST | Bind host for the dashboard (default 127.0.0.1); same as --dashboard-host | |
| IRIS_DASHBOARD_PORT | Dashboard port (default 6920); IRIS_PORT is the MCP HTTP transport port (default 3000) | |
| IRIS_DB_PATH | SQLite database path | |
| IRIS_HOME | Directory for the database, custom rules and preferences (default ~/.iris) | |
| IRIS_HOST | Bind host for the HTTP transport (default 127.0.0.1); only used with --transport http | |
| IRIS_LLM_JUDGE_DAILY_BUDGET_USD | What every judge call together may spend per UTC day, per tenant, in USD (default 1): the relevance judge, evaluate_with_llm_judge and verify_citations. Past it no judge call is made | |
| IRIS_LLM_JUDGE_MAX_COST_USD_PER_EVAL | Hard cap on the worst-case cost of one LLM judge call (default 0.25); max_cost_usd can lower it for a call, never raise it | |
| IRIS_LOG_LEVEL | Log level: debug, info, warn, error | |
| IRIS_MAX_INPUT_ELEMENTS | The most array elements and object keys one input may hold before it is validated (default 100000): a tool call's arguments, and a dashboard or OTLP JSON request body. A tool call over it gets an error result; a body, 413. OTLP protobuf has its own ceiling, one element per 5 bytes of the request size limit | |
| IRIS_MAX_TOOL_INPUT_ELEMENTS | The name of IRIS_MAX_INPUT_ELEMENTS before 0.21.0, read when the new name is unset | |
| IRIS_OTEL_ENDPOINT | OTLP/HTTP collector URL; when set, every stored trace is also exported as OpenTelemetry spans (best-effort, asynchronous) | |
| IRIS_OTEL_HEADERS | Comma-separated key=value headers sent with every OTLP export (for example an auth token for your collector) | |
| IRIS_OTEL_SERVICE_NAME | service.name resource attribute on exported spans (default iris-eval) | |
| IRIS_OTEL_TIMEOUT_MS | Timeout in milliseconds for each OTLP export request | |
| IRIS_PORT | MCP HTTP transport port (default 3000); only used with --transport http | |
| IRIS_RELEVANCE_JUDGE_DAILY_BUDGET_USD | The name of IRIS_LLM_JUDGE_DAILY_BUDGET_USD before 0.21.0. Read when the new name is unset, and it then limits every judge call | |
| IRIS_RELEVANCE_JUDGE_MAX_CALLS_PER_REQUEST | Relevance judge calls one request may make (default 20) | |
| IRIS_RELEVANCE_JUDGE_MODEL | A priced judge model; answers_the_ask then asks it on every evaluation with input, sending that input and output to the model's provider, and gates on its verdict (off by default) | |
| IRIS_RELEVANCE_JUDGE_REDACT | on (default) replaces the PII and credentials no_pii flags before the relevance judge sends input and output to its provider; off sends them as they are | |
| IRIS_SEARCH_BUDGET_MS | How long one trace search may read before it answers with the matches found so far, in milliseconds (50-60000, default 1000) | |
| IRIS_SEARCH_INDEX | on (default) or off. off keeps no full-text index of the traces: writes are cheaper, and a search reads the traces within its time budget | |
| IRIS_SQLITE_DRIVER | Which SQLite driver holds the database: native (better-sqlite3, the default) or node (Node's built-in node:sqlite, Node 22.13+). Unset: native, falling back to node when the native module cannot load | |
| IRIS_TRANSPORT | Transport: stdio (default) or http; same as --transport | |
| IRIS_WEBHOOK_URL | The receiver of the webhook that fires on a moment (docs/webhooks.md); merged over notify.webhook in config.json |
자동 검사
2026년 11월 12일에 실행됨전체 보고서 →자동 검사를 통과했습니다. 이는 보안 감사가 아닙니다. 코드를 직접 읽고 서버에 필요한 최소한의 권한만 부여하십시오. 검사 방식
- 소스 저장소 연결 및 접근 가능통과
- 오픈 소스 라이선스 (OSI)통과
- 최근 1년 내 커밋 또는 릴리스통과
- 패키지 이름이 인기 패키지와 유사하지 않음통과
- 원격 엔드포인트가 HTTPS 사용검사되지 않음
- 파괴적인 도구에 라벨 표시됨검사되지 않음
- 모델에 대한 숨겨진 지침 없음통과
- 패키지 및 해당 종속성에 중요한 보안 권고 없음통과
README
GitHub에서 보기Iris — stop shipping agents on vibes
Send Iris an agent run and ask for a verdict on its quality, safety, and cost — on your machine, with no SDK and no account. Most agent projects check quality by running a few remembered prompts and eyeballing the output. Iris replaces that with numbers you can audit: your agent's runs land in a SQLite database on your disk, 28 built-in rules score them deterministically — PII, prompt injection, hallucination markers, cost thresholds, and the agent's own tool calls — free, with no LLM calls, and an optional LLM judge with a hard per-eval cost cap handles the semantic questions. Every rule is inspectable and editable, because a judge you can't audit is just vibes with a number on it. MIT licensed, no telemetry. Nothing leaves your machine unless you turn one of these on: an OpenTelemetry endpoint (IRIS_OTEL_ENDPOINT), which exports traces to the collector you name; the LLM judge with your own key, which sends the text it judges to that provider, and whose citation check fetches the pages an output cites; or a webhook, which posts ids, the verdict and rule names, never the text, to the address you set.
Requires Node.js 22.13 or later. Check with node --version.

The demo database, recorded by scripts/demo-media.mts; the source is demo.mp4. A still: dashboard-overview.png.
A failure on screen in 60 seconds
No agent wiring, no config — one command:
npx @iris-eval/mcp-server --demo
This seeds a demo database — five small agents, two weeks of runs, every verdict the engine's own — and serves the dashboard against it at http://localhost:6920 (your browser opens automatically on first run). The dashboard lands on Failures: what failed, worst and newest first, each card naming the rule and its evidence. Worth clicking into — a PII leak caught by the safety rules, a hidden directive in a forum post that the summarizer complied with, a number the source document never said, two runs on the same twelve questions compared with an interval (Runs), a deployed custom rule and a paused one with their audit rows, and a failed LLM-judge score with its rationale.
Demo data lives in its own database (demo.db in your Iris home directory — ~/.iris on macOS/Linux, %USERPROFILE%\.iris on Windows) and never mixes with your real traces. Remove all of it with one command:
npx @iris-eval/mcp-server --demo-clear
Hook up your own agent
First, prove the install works on this machine — it runs offline and opens nothing of yours:
npx @iris-eval/mcp-server --self-test # exit 0 = healthy
Then add Iris to your MCP client. One command writes the client's own config file, keeps every other server in it, and pins the version you ran:
npx -y @iris-eval/mcp-server install claude-code
The clients: claude-code, claude-desktop, cursor, windsurf, continue, vscode, cline, zed, codex, gemini. install --list shows the ones found on this machine, the Iris each one runs and the file it reads; install <client> --uninstall takes Iris out again. Every client shares one database, so after an upgrade move them all at once with install --upgrade (Updating). Restart the client to load it.
Claude Desktop: one click. Each release from 0.20.0 on attaches iris-eval.mcpb, an MCP Bundle: download the latest, open it, and Claude Desktop shows an install dialog. Nothing on it is required — an Anthropic or OpenAI key for the LLM judge is optional, and the dashboard is a switch that starts off. The bundle holds the npm package and its dependencies, so nothing else needs installing: Claude Desktop runs it under the Node it ships when that Node is 22.13 or newer (Claude Desktop 1.1.6679 ships 24.13), and Iris stores traces with Node's built-in SQLite, in the same ~/.iris every other install uses. The release notes show how to verify its signature and build attestation.
It runs in any MCP client, and every client it names has a row saying how it is checked. Verified on every CI run: Claude Code, Gemini CLI — the real client starts Iris from the config the installer wrote and reports it connected (claude mcp list, gemini mcp list), on Linux, macOS and Windows; Claude Code's capture plugin hooks are driven through the real scripts too. Configured from each client's own MCP documentation — the installer writes the file each client documents, at its documented path and in its own format (an mcpServers block, VS Code's servers, Zed's context_servers, Codex's TOML), keeping comments and every other server, and unit tests hold each path and every format writer to that documentation: Claude Desktop, Cursor, Devin Desktop (Windsurf), Continue, VS Code, Cline, Zed, OpenAI Codex CLI. Every row with its source and the date it was read: https://iris-eval.com/clients. By hand instead, one block, dashboard included:
{
"mcpServers": {
"iris-eval": {
"command": "npx",
"args": ["-y", "@iris-eval/mcp-server", "--dashboard"]
}
}
}
Your client lists Iris's twelve tools on connect, and the dashboard serves at http://localhost:6920. Now paste this to your agent:
Log that last task to Iris and evaluate the output.
The trace lands on the dashboard with its scores. Prefer the MCP server headless? Drop --dashboard from the args — you can open the same dashboard any time with npx @iris-eval/mcp-server --dashboard.
One thing worth knowing up front: MCP tools are called when the model decides to call them. Iris doesn't intercept your agent, so traces are logged when your agent asks it to log them — either because you told it to, or because your code calls the tools directly. Ask your agent to "log this to Iris and evaluate it" and it will. If you want capture that doesn't depend on the model choosing, POST /api/v1/traces does exactly that — your code sends the trace over plain HTTP, no model in the loop (see docs/http-ingest.md). The CLI and host hooks on the roadmap will be thin clients over the same endpoint.
Capture over HTTP (no model in the loop)
The ingest endpoint lives on the dashboard port — 6920 by default, not the MCP transport port — and it exists only while the dashboard is running. Pass --dashboard (or set IRIS_DASHBOARD=true); --transport http on its own does not start it, and a request to the transport port returns 404. With the dashboard up, anything that can send an HTTP request can log a trace — and optionally run the deterministic evals in the same request. GET /api/v1/capabilities on the same port says what this server can judge, what each rule needs, the judge state with the steps that enable it, and the limits — the same object the MCP resource iris://capabilities serves — so an HTTP caller has the frame an MCP client gets at initialize:
curl -s -X POST "http://127.0.0.1:6920/api/v1/traces" \
-H "Content-Type: application/json" \
-d '{
"agent_name": "support-bot",
"input": "What is the refund policy?",
"output": "Refunds are available within 30 days of purchase.",
"evaluate": true,
"eval_type": "safety"
}'
Returns 201 with the stored trace_id and the evaluation result (in --demo mode the endpoint refuses writes with 403, so demo data never mixes with yours). The endpoint accepts the same body as the log_trace tool and sits behind the same middleware stack as the rest of the dashboard: loopback bind and the DNS-rebinding guard by default, plus Bearer auth when you set one. Two plain facts about it: it accepts unauthenticated writes unless Iris was started with --api-key (or IRIS_API_KEY) — the loopback bind is what keeps it to your machine by default, so set a key before binding beyond loopback; and what it stores is verbatim — input and output land in iris.db exactly as sent, including any text no_pii goes on to flag. Full contract, field reference, and error semantics: docs/http-ingest.md.
Capture every Claude Code turn (optional)
/plugin marketplace add iris-eval/mcp-server
/plugin install iris-eval-capture@iris-eval
A second, separately installed plugin: its hooks record each turn's prompt, tool calls (the failed ones included, with the error the agent received) and final answer, keyed on the prompt, so an interrupted turn's calls and a background sub-agent's are sent too, and hand them to iris-eval ingest, detached, with critical spans redacted in the stored evaluation text — capture that does not depend on the model deciding to call a tool. A turn that ended in an API error is stored without being judged; when the model logged the turn too, both traces are kept and the plugin's names the model's. It never prints, never blocks, never sends anything anywhere. Installing iris-eval alone changes nothing about your turn loop. Limits and removal: claude-plugin-capture/README.md.
Python
pip install iris-eval
from iris_eval import IrisClient
iris = IrisClient() # IRIS_URL, or the running dashboard's runtime.json
iris.evaluate_output("…", input="…", agent_name="support-bot")["verdict"] # {"state": "pass", "basis": "clean", "by": []}
A thin client over the HTTP API of server 0.16.0 and later, versioned on its own — iris_eval.__version__ and the PyPI page carry its number, which is not the server's: log_trace(), evaluate_output(), get_traces(), get_trace(), health(), capabilities(), sync and async, typed answers, the server's own sentence on a refusal — and a pytest plugin: an iris fixture and assert_iris(output, expect="pass") that asserts on the verdict's state. packages/python/README.md.
Record every OpenAI and Anthropic call
from iris_eval import wrap_openai
client = wrap_openai(OpenAI(), agent_name="support-bot") # every call: a GenAI span to POST /v1/traces, scored
import { wrapOpenAI } from '@iris-eval/sdk';
const openai = wrapOpenAI(new OpenAI(), { agentName: 'support-bot' });
Wrap the provider client once and each model call becomes one OpenTelemetry GenAI span sent to the OTLP door, stored with its input, output, token usage and tool calls, and scored: capture that does not depend on the model calling a tool. wrap_openai / wrap_anthropic in the Python client; wrapOpenAI, wrapAnthropic and irisMiddleware for the Vercel AI SDK in @iris-eval/sdk. Both are not yet published (the next iris-eval release on PyPI; @iris-eval/sdk is built from source until its first npm release). Streams, the SDKs' stream helpers and tool calls are covered, the original client is not changed, and Iris being down never breaks a call — packages/sdk/README.md, packages/python/README.md.
Score every LangChain and LangGraph run
from iris_eval.langchain import IrisCallbackHandler
graph.invoke(inputs, config={"callbacks": [IrisCallbackHandler(agent_name="support-bot")]})
Each top-level run becomes one trace (the run, its model calls, its tool calls and its graph nodes as GenAI spans) with its input, output, tool calls, token usage and a verdict. Python in the client (pip install iris-eval, 0.2.0 and later), JavaScript as @iris-eval/langchain (in this repository, not yet published to npm). Both are proven in CI against a real LangGraph app with a scripted model; LangSmith's own OpenTelemetry export is proven the same way — docs/otel-recipes.md.
A CI gate, no server needed
npx -y @iris-eval/mcp-server ingest --file traces.ndjson --evaluate --fail-on any
any fails the job on every verdict that is not a pass: a failure, and a verdict that was not checked. Name one basis instead (--fail-on detector_veto) to fail only on that one.
Or the GitHub Action (0.16.0), which fails the job on the verdicts you name, writes the receipt to the job summary and posts it as one pull-request comment updated in place: uses: iris-eval/mcp-server/.github/actions/gate@v0.21.0 with traces: traces.ndjson — docs/ci-gate.md.
A fourth door (0.15.0): POST /v1/traces on the dashboard port takes the OTLP/HTTP JSON or protobuf your OpenTelemetry instrumentation already emits (the Python SDK's exporter speaks protobuf only, so this is the Python door too), and each OTLP trace becomes an Iris trace with its spans — docs/otel-integration.md; one recipe per framework (Pydantic AI, Google ADK, LangGraph via LangSmith, CrewAI, the OpenAI Agents SDK in Python and JavaScript, LlamaIndex, AutoGen, Microsoft Agent Framework, Semantic Kernel, the Vercel AI SDK and Mastra), each proved by a fixture, in docs/otel-recipes.md. ingest reads one JSON trace (or NDJSON, one per line) from stdin or a file, stores it, evaluates it under exactly the rules evaluate_output runs, prints one JSON line per trace with the verdict and its basis, and exits 1 when a verdict matches --fail-on. --dataset <id|label> restricts that gate to the case keys in a dataset (POST /api/v1/datasets promotes a run's case keys into one), so a job fails only on the cases you chose. The full recipe, the exit codes and the eight bases are in docs/ci-gate.md.
Write a rule as code
eval.plugins in config.json loads rules you wrote — an ES module whose default export is { name, kind, mechanism, version, needs, evaluate(ctx) } — pinned by the sha256 of the file, so a file that changed since you pinned it refuses startup rather than running. A loaded plugin fires like a built-in and shows on list_rules under plugins. The contract, the hash recipe and what a plugin may return: docs/plugins.md.
Use the engine in your own process
The evaluation engine is importable — no server, no database, no model:
import { EvalEngine, defaultConfig } from '@iris-eval/mcp-server/engine';
const engine = new EvalEngine(defaultConfig.eval.defaultThreshold, defaultConfig.eval.ruleThresholds, defaultConfig.eval);
const result = await engine.evaluateAll({ output: answer, input: prompt, toolCalls, costUsd });
result.verdict.state; // 'pass' | 'fail' | 'unknown', with result.verdict.basis and result.interpretations
The same engine, the same rules and the same composer the server runs; builtInRules(), createCustomRule(), compose() and the published-accuracy readers are exported beside it.
A typed client for the HTTP route
import { createClient } from '@iris-eval/mcp-server/client';
const iris = createClient({ baseUrl: 'http://127.0.0.1:6920', apiKey: process.env.IRIS_API_KEY });
const { trace_id, evaluation } = await iris.logTrace({ agent_name: 'support-bot', input, output, tool_calls, evaluate: true });
evaluation?.verdict?.state; // the same object evaluate_output returns
One body on every door: it is what log_trace and iris-eval ingest accept. A refusal throws IrisClientError with the server's own sentence and status. Both subpaths are checked from a packed tarball on every build.
Verify your install
npx @iris-eval/mcp-server --self-test # offline diagnostic; exit 0 = healthy, 1 = a check failed
npx @iris-eval/mcp-server --version # prints the bare version, e.g. 1.2.3
--self-test first creates your Iris home if it is missing and checks that it is writable (exit 1, naming the path, if it is not), reports where your database's search index is (whole, how many traces a background build has indexed so far, or no FTS5 on this SQLite), reads your database's schema (exit 1, with the fix, when this version or an MCP client pinned to an older release cannot open it), then runs its checks — storage round-trip, a planted SSN and a planted injection caught by the safety rules, dashboard boot, the DNS-rebinding guard — inside an isolated temp home. Your real database is only read, never changed. Everything Iris writes lives under one directory, your Iris home: ~/.iris by default (%USERPROFILE%\.iris on Windows), or wherever IRIS_HOME points. That is where iris.db, config.json, custom-rules.json, audit.log, preferences.json and the demo files live; point IRIS_HOME at a scratch directory to try Iris without touching your real data.
| Client | Status | What that means | Read |
|---|---|---|---|
| Claude Code | verified | a test drives the real client on every CI run | 2026-09-25 |
| Claude Desktop | claimed | configured from the client's own MCP documentation; the installer's path and format for it are unit-tested | 2026-09-25 |
| Cursor | claimed | configured from the client's own MCP documentation; the installer's path and format for it are unit-tested | 2026-09-25 |
| Devin Desktop (Windsurf) | claimed | configured from the client's own MCP documentation; the installer's path and format for it are unit-tested | 2026-09-25 |
| Continue | claimed | configured from the client's own MCP documentation; the installer's path and format for it are unit-tested | 2026-09-25 |
| VS Code | claimed | configured from the client's own MCP documentation; the installer's path and format for it are unit-tested | 2026-09-25 |
| Cline | claimed | configured from the client's own MCP documentation; the installer's path and format for it are unit-tested | 2026-09-25 |
| Zed | claimed | configured from the client's own MCP documentation; the installer's path and format for it are unit-tested | 2026-09-25 |
| OpenAI Codex CLI | claimed | configured from the client's own MCP documentation; the installer's path and format for it are unit-tested | 2026-09-25 |
| Gemini CLI | verified | a test drives the real client on every CI run | 2026-09-25 |
Every row with what was checked: iris-eval.com/clients. No client is called supported without a row.
npx -y @iris-eval/mcp-server install <client> writes each of these for you. By hand, per client:
Claude Desktop
Edit your MCP config file:
- macOS:
~/Library/Application Support/Claude/claude_desktop_config.json - Windows:
%APPDATA%\Claude\claude_desktop_config.json
Add the JSON config above, then restart Claude Desktop.
Claude Code
claude mcp add --transport stdio iris-eval -- npx -y @iris-eval/mcp-server
Then restart the session (/clear or relaunch) for tools to load.
Windows note: Do not use
cmd /cwrapper — it causes path parsing issues. Thenpxcommand works directly.
Cursor
Add the JSON config above to ~/.cursor/mcp.json (every project) or .cursor/mcp.json in a workspace, with "type": "stdio" in the iris-eval entry — Cursor's docs mark it required.
Devin Desktop (Windsurf)
Add the JSON config above to mcp_config.json: ~/.config/devin/mcp_config.json on macOS and Linux, %APPDATA%\devin\mcp_config.json on Windows.
Continue
Save the JSON config above as its own file in Continue's mcpServers folder: ~/.continue/mcpServers/iris-eval.json (every workspace) or .continue/mcpServers/iris-eval.json in one.
VS Code (native MCP)
Add to .vscode/mcp.json in your workspace (note: VS Code uses servers, not mcpServers):
{
"servers": {
"iris-eval": {
"command": "npx",
"args": ["-y", "@iris-eval/mcp-server"]
}
}
}
Cline
Open Cline's MCP Servers panel → Configure MCP Servers, and add the mcpServers JSON config above to cline_mcp_settings.json (~/.cline/data/settings/cline_mcp_settings.json, shared by Cline in VS Code, JetBrains and the CLI).
Zed
Add to Zed settings.json:
{
"context_servers": {
"iris-eval": {
"command": "npx",
"args": ["-y", "@iris-eval/mcp-server"],
"env": {}
}
}
}
OpenAI Codex CLI
Add to ~/.codex/config.toml:
[mcp_servers.iris-eval]
command = "npx"
args = ["-y", "@iris-eval/mcp-server"]
Gemini CLI
Add the mcpServers JSON config above to ~/.gemini/settings.json. Gemini CLI connects to MCP servers only in folders it trusts: if gemini mcp list shows iris-eval as Disabled, run /permissions in that folder.
Anything else that speaks MCP
Iris is a standard stdio MCP server — one npx @iris-eval/mcp-server command, no SDK, no code changes. If your client supports MCP, it supports Iris. Client config formats change; when in doubt, check your client's MCP docs and point it at that command.
Other Install Methods
# Global install (recommended for persistent data and faster startup)
npm install -g @iris-eval/mcp-server
iris-eval --dashboard
# Docker — two servers, two ports: 3000 = MCP HTTP transport,
# 6920 = dashboard (which also serves the POST /api/v1/traces ingest endpoint).
Iris MCP 질문
Iris MCP를 Claude Code에 추가하려면 어떻게 하나요?
터미널에서 claude mcp add --env 'IRIS_ANTHROPIC_API_KEY=<YOUR_IRIS_ANTHROPIC_API_KEY>' --env 'IRIS_API_KEY=<YOUR_IRIS_API_KEY>' --env 'IRIS_OPENAI_API_KEY=<YOUR_IRIS_OPENAI_API_KEY>' --env 'IRIS_WEBHOOK_SECRET=<YOUR_IRIS_WEBHOOK_SECRET>' mcp-server -- npx -y @iris-eval/mcp-server를 실행합니다. 모든 프로젝트에서 사용할 수 있게 하려면 --scope user를 추가합니다.
Iris MCP를 Cursor에 추가하려면 어떻게 하나요?
~/.cursor/mcp.json를 열고 mcpServers 아래에 서버를 추가합니다. 위 설치 섹션에 표시된 Cursor 구성을 사용합니다.
Iris MCP는 어떤 도구를 제공하나요?
17개 도구를 제공합니다. 예: log_trace, evaluate_output, get_traces, list_rules, deploy_rule.
Iris를 사용하려면 계정이나 LLM이 필요한가요?
아니요, 계정이나 SDK 없이 로컬 환경에서 Iris를 실행할 수 있습니다. LLM 호출이 필요 없는 28가지 내장 결정론적 규칙을 사용합니다.
데이터는 어디에 저장되나요?
에이전트 실행 기록은 로컬 디스크의 SQLite 데이터베이스에 저장됩니다.
모델이 도구를 호출하지 않는 경우에도 에이전트 트레이스를 캡처할 수 있나요?
일반 HTTP를 통해 트레이스를 전송할 수 있는 HTTP ingest 엔드포인트를 사용하거나, 전용 Claude Code 캡처 플러그인을 사용할 수 있습니다.
데이터가 클라우드로 전송되나요?
OpenTelemetry 엔드포인트, LLM judge 또는 webhook을 명시적으로 활성화하지 않는 한, 기본적으로 데이터는 로컬 환경을 벗어나지 않습니다.