Monitors LLM applications for behavioral drift by creating and running evaluation specs directly within an MCP-compatible client.
Also known as bch1212/modelwatch, modelwatch-mcp
- Stars
- 0
- Tools
- 9
- Downloads, 30 days
- —
- Latest version
- 0.1.3
- Last commit
- Nov 9, 2026
- License
- —
What is ModelWatch MCP?
ModelWatch monitors LLM applications for behavioral drift by creating and running evaluation specs directly within an MCP-compatible client. It exposes 9 tools, such as list_endpoints, create_endpoint, list_specs.
Summary generated from the project's README
About ModelWatch MCP server
Modelwatch provides a way for developers to monitor behavioral drift in LLM applications directly through an MCP-compatible client like Claude Desktop or Claude Code. By integrating this server, users can manage evaluation specifications and monitor model performance without switching between different applications. The server requires a ModelWatch API key provided via an environment variable to connect to the service.
Users can interact with several specific tools to manage their monitoring workflows. The server includes functions to list endpoints, create new specs with specific prompts and schedules, and run evaluations on demand. It also allows for resetting baselines after model swaps and retrieving drift events or health status for individual specifications. The free tier includes five specs and 500 runs per month.
What you can do with it
- 1Create and run evaluation specsAsk the AI client to set up a new behavioral test for a specific model endpoint and execute it immediately.
- 2Monitor model drift automaticallySet up a recurring schedule for a prompt to detect when model behavior changes over time.
Written from the project's README. Every use case is checked against a quote from it.
How to install ModelWatch MCP
Formats verified 10 Nov 2026Add to CursorAdd to GooseInstall ModelWatch MCP in Claude Code
$ claude mcp add --env 'MODELWATCH_API_KEY=<YOUR_MODELWATCH_API_KEY>' modelwatch -- npx -y modelwatch-mcpInstall ModelWatch MCP in Cursor
~/.cursor/mcp.json
{
"mcpServers": {
"modelwatch": {
"command": "npx",
"args": [
"-y",
"modelwatch-mcp"
],
"env": {
"MODELWATCH_API_KEY": "<YOUR_MODELWATCH_API_KEY>"
}
}
}
}Install ModelWatch MCP in VS Code
.vscode/mcp.json
{
"servers": {
"modelwatch": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"modelwatch-mcp"
],
"env": {
"MODELWATCH_API_KEY": "<YOUR_MODELWATCH_API_KEY>"
}
}
}
}Install ModelWatch MCP in VS Code CLI
$ code --add-mcp '{"name":"modelwatch","type":"stdio","command":"npx","args":["-y","modelwatch-mcp"],"env":{"MODELWATCH_API_KEY":"<YOUR_MODELWATCH_API_KEY>"}}'Install ModelWatch MCP in Claude Desktop
claude_desktop_config.json
{
"mcpServers": {
"modelwatch": {
"command": "npx",
"args": [
"-y",
"modelwatch-mcp"
],
"env": {
"MODELWATCH_API_KEY": "<YOUR_MODELWATCH_API_KEY>"
}
}
}
}Install ModelWatch MCP in Codex
$ codex mcp add modelwatch --env 'MODELWATCH_API_KEY=<YOUR_MODELWATCH_API_KEY>' -- npx -y modelwatch-mcpInstall ModelWatch MCP in Codex config.toml
~/.codex/config.toml
[mcp_servers.modelwatch] command = "npx" args = ["-y", "modelwatch-mcp"] [mcp_servers.modelwatch.env] MODELWATCH_API_KEY = "<YOUR_MODELWATCH_API_KEY>"
Install ModelWatch MCP in Gemini CLI
$ gemini mcp add -s user -e 'MODELWATCH_API_KEY=<YOUR_MODELWATCH_API_KEY>' modelwatch npx -- -y modelwatch-mcpInstall ModelWatch MCP in Gemini settings.json
~/.gemini/settings.json
{
"mcpServers": {
"modelwatch": {
"command": "npx",
"args": [
"-y",
"modelwatch-mcp"
],
"env": {
"MODELWATCH_API_KEY": "<YOUR_MODELWATCH_API_KEY>"
}
}
}
}ModelWatch MCP tools 9
Detected from the READMElist_endpointscreate_endpointlist_specscreate_specrun_specreset_baselineget_drift_eventsget_spec_historyget_health
| list_endpoints | — | |
| create_endpoint | — | |
| list_specs | — | |
| create_spec | — | |
| run_spec | — | |
| reset_baseline | — | |
| get_drift_events | — | |
| get_spec_history | — | |
| get_health | — |
Configuration
MODELWATCH_API_KEYRequiredSecretAPI key from https://modelwatch.app (free tier: 5 specs, 500 runs/mo).
MODELWATCH_API_BASEOverride the API base URL for self-hosted deployments.
| Name | Description | |
|---|---|---|
| MODELWATCH_API_KEY | API key from https://modelwatch.app (free tier: 5 specs, 500 runs/mo). | RequiredSecret |
| MODELWATCH_API_BASE | Override the API base URL for self-hosted deployments. |
Automated checks
Ran Nov 14, 2026Full report →Passed our automated checks. That is not a security audit: read the code and give the server the narrowest access it needs. How checks work · What to fix for more points
- Source repository linked and reachablebch1212/modelwatch is public and reachablegithub.com/bch1212/modelwatch ↗Pass
- Package names this repository as its sourcemodelwatch-mcp names bch1212/modelwatch as sourcenpmjs.com/package/modelwatch-mcp ↗Pass
- Build provenance (Sigstore attestation)Sigstore attestation for modelwatch-mcpnpmjs.com/package/modelwatch-mcp/v/0.1.3#provenance ↗Pass
- Open source license (OSI)No licenseFinding
- Commit or release in the last yearLast commit or release 2026-11-09github.com/bch1212/modelwatch ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNo popular package one edit away from modelwatch-mcpPass
- No critical advisory in the package or its dependencies99 resolved dependencies checked; no critical advisoryosv.dev ↗Pass
- No hidden instructions to the model9 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNo tool named like delete, drop or execPass
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubmodelwatch-mcp
MCP server for ModelWatch — continuous behavioral drift monitoring for LLM-powered applications.
Lets Claude Desktop / Claude Code (or any MCP-compatible client) create drift specs, run them, and surface drift events without leaving the chat.
Install
npm install -g modelwatch-mcp
Configure (Claude Desktop)
Add to ~/Library/Application Support/Claude/claude_desktop_config.json:
{
"mcpServers": {
"modelwatch": {
"command": "npx",
"args": ["-y", "modelwatch-mcp"],
"env": {
"MODELWATCH_API_KEY": "mw_PASTE_FROM_EMAIL"
}
}
}
}
Get a free key at https://modelwatch.app/#signup (5 specs, 500 runs/mo, no card). Then restart Claude Desktop.
The example configuration stores a plaintext key in a local config file. Restrict that file to your user account, do not commit it, and rotate the key if it is exposed. The key never belongs in a command-line argument.
Tools
| Tool | What it does |
|---|---|
list_endpoints |
Show monitored LLM endpoints |
create_endpoint |
Add an endpoint (provider + model) |
list_specs |
Show behavioral specs |
create_spec |
Add a spec (prompt + schedule + optional semantic similarity threshold) |
run_spec |
Run a spec on demand; baseline if first run |
reset_baseline |
Re-baseline after an intentional model swap |
get_drift_events |
Recent drift events across the workspace |
get_spec_history |
Run history for one spec |
get_health |
Per-spec health status, last drift score and last run time |
Example session
You: Create a refusal canary against the OpenAI endpoint, asking about chest pain. Run it now and tell me the baseline.
Claude calls list_endpoints → create_spec → run_spec and reports the
score. The same spec then runs daily forever and emails you when drift is
detected.
Environment variables
| Var | Required | Default |
|---|---|---|
MODELWATCH_API_KEY |
Yes | — |
MODELWATCH_API_BASE |
No | https://api.modelwatch.app |
Self-hosted? Point MODELWATCH_API_BASE at an HTTPS origin. HTTP is accepted
only for loopback development; paths, query strings, URL credentials and
redirects are rejected before a key is sent.
Source
https://github.com/bch1212/modelwatch — MIT licensed.
ModelWatch MCP questions
How do I add ModelWatch MCP to Claude Code?
Run claude mcp add --env 'MODELWATCH_API_KEY=<YOUR_MODELWATCH_API_KEY>' modelwatch -- npx -y modelwatch-mcp in your terminal. Add --scope user to make it available in every project.
How do I add ModelWatch MCP to Cursor?
Open ~/.cursor/mcp.json and add the server under mcpServers, using the Cursor config shown in the install section above.
What tools does ModelWatch MCP provide?
It exposes 9 tools, including list_endpoints, create_endpoint, list_specs, create_spec, run_spec.
How do I install ModelWatch MCP?
You can install it globally using npm by running 'npm install -g modelwatch-mcp'.
Is there a free tier available?
Yes, you can get a free key that includes 5 specs and 500 runs per month without a credit card.