Agent Reliability MCP server
Commit Sep 7, 2026Provides a curated knowledge graph for testing, benchmarking, and auditing autonomous AI agents with cited claims and confidence scores.
Also known as citarium/agentreliability-mcp, agentreliability-mcp
- Stars
- 0
- Tools
- 8
- Downloads, 30 days
- —
- Latest version
- 0.6.14
- Last commit
- Sep 7, 2026
- License
- CC-BY-4.0
What is Agent Reliability MCP?
Agent Reliability provides a curated knowledge graph for testing, benchmarking, and auditing autonomous AI agents with cited claims and confidence scores. It exposes 8 tools, such as get_overview, search, answer.
Summary generated from the project's README
About Agent Reliability MCP server
Agent Reliability provides a remote MCP server built around a curated knowledge graph designed for testing, benchmarking, and auditing autonomous AI agents. It serves as a specialized resource for developers working with agent evaluation, LLM-as-judge failure modes, and benchmark contamination. The server is hosted as a streamable-HTTP endpoint, meaning it requires no local installation, API keys, or user accounts to access its read-only data.
Users can connect via Claude Code or any client supporting the mcpServers configuration. The server offers eight specific tools, including search, answer, and get_entity, to navigate its corpus of knowledge objects. Unlike standard LLM responses, every claim returned by this server includes a confidence score, an evidence tier, and a direct citation to a registered source. This allows an agent to verify information or decline to use weak claims rather than asserting unverified facts.
What you can do with it
- 1Verify agent evaluation methodsAsk the server to explain the difference between a real evaluation and a contaminated benchmark.
- 2Audit claim reliabilityRetrieve specific knowledge objects to inspect the confidence scores and sources behind every assertion.
- 3Explore related technical topicsUse the graph capabilities to find connected concepts within the reliability domain.
Written from the project's README. Every use case is checked against a quote from it.
How to install Agent Reliability MCP
Formats verified 10 Nov 2026Add to CursorAdd to GooseInstall Agent Reliability MCP in Claude Code
$ claude mcp add --transport http agentreliability-mcp 'https://agentreliability.dev/mcp?via=manifest'Install Agent Reliability MCP in Cursor
~/.cursor/mcp.json
{
"mcpServers": {
"agentreliability-mcp": {
"url": "https://agentreliability.dev/mcp?via=manifest"
}
}
}Install Agent Reliability MCP in VS Code
.vscode/mcp.json
{
"servers": {
"agentreliability-mcp": {
"type": "http",
"url": "https://agentreliability.dev/mcp?via=manifest"
}
}
}Install Agent Reliability MCP in VS Code CLI
$ code --add-mcp '{"name":"agentreliability-mcp","type":"http","url":"https://agentreliability.dev/mcp?via=manifest"}'Install Agent Reliability MCP in Codex
$ codex mcp add agentreliability-mcp --url 'https://agentreliability.dev/mcp?via=manifest'Install Agent Reliability MCP in Codex config.toml
~/.codex/config.toml
[mcp_servers.agentreliability-mcp] url = "https://agentreliability.dev/mcp?via=manifest"
Install Agent Reliability MCP in Gemini CLI
$ gemini mcp add -s user --transport http agentreliability-mcp 'https://agentreliability.dev/mcp?via=manifest'Install Agent Reliability MCP in Gemini settings.json
~/.gemini/settings.json
{
"mcpServers": {
"agentreliability-mcp": {
"url": "https://agentreliability.dev/mcp?via=manifest",
"type": "http"
}
}
}Agent Reliability MCP tools 8
Detected from the READMEget_overviewsearchanswerget_entityget_topicget_relatedget_sourcesget_latest
Automated checks
Ran Nov 15, 2026Full report →Not verified yet. A server needs a score of 80, no serious finding, and enough data for us to check. How checks work · What to fix for more points
- Source repository linked and reachablecitarium/agentreliability-mcp is public and reachablegithub.com/citarium/agentreliability-mcp ↗Pass
- Package names this repository as its sourceNot checked
- Build provenance (Sigstore attestation)Not checked
- Open source license (OSI)CC-BY-4.0 is not OSI-approvedspdx.org/licenses/CC-BY-4.0.html ↗Finding
- Commit or release in the last yearLast commit or release 2026-09-07github.com/citarium/agentreliability-mcp ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNot checked
- No critical advisory in the package or its dependenciesNot checked
- No hidden instructions to the model8 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNo tool named like delete, drop or execPass
- Remote endpoint answers the MCP handshakeinitialize answered with protocol 2025-06-18 in 471 msPass
README
View on GitHubAgent Reliability — MCP server
Testing, benchmarking and auditing autonomous AI agents — methods, harnesses, evidence
A remote MCP server over a curated knowledge graph. Every claim it returns is bound to a registered source: the tools hand back claims with their citations and a confidence value, so an agent can show its work instead of asserting.
Nothing to install. It is a hosted streamable-HTTP endpoint:
https://agentreliability.dev/mcp
Also listed on Smithery.
Add it to a client
Claude Code
claude mcp add --transport http agent-reliability https://agentreliability.dev/mcp
Claude Desktop / any client reading mcpServers
{
"mcpServers": {
"agent-reliability": {
"type": "streamable-http",
"url": "https://agentreliability.dev/mcp"
}
}
}
No API key, no account, no auth. Read-only.
Check it answers, without any client at all:
curl -s https://agentreliability.dev/mcp \
-H 'Content-Type: application/json' \
-H 'Accept: application/json, text/event-stream' \
-H 'mcp-protocol-version: 2025-06-18' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
Tools
Eight, each with an outputSchema, each returning structuredContent.
| tool | arguments | what it does |
|---|---|---|
get_overview |
— | Corpus overview: what this instance knows, counts by type, published tags, freshness. Start here when you land and do not yet know whether this corpus can answer your question. |
search |
query, limit? |
Full-text search over the knowledge graph. Accent- and apostrophe-insensitive, so query in the user's own words; every hit carries its relevance score and the fields it matched. |
answer |
question |
Answer a question from the corpus. Returns the matched object's claims with sources and confidence — never an unsourced answer. |
get_entity |
id |
Fetch one knowledge object by id, with its claims and the sources each claim cites. |
get_topic |
tag |
List the knowledge objects carrying a tag (topics are content-backed tags). |
get_related |
id |
Graph neighbours of an object: outgoing and incoming relations, each with its relation type. |
get_sources |
object_id? |
The whole source registry, or just the sources cited by one object. Use it to judge the corpus before trusting it. |
get_latest |
limit? |
Most recently verified knowledge objects — a freshness signal. |
The intended path is get_overview → search or answer → get_entity
→ get_related. get_overview exists because an agent that has just
arrived needs to know whether this corpus can help before it spends a
call guessing.
What is in the corpus
| knowledge objects | 38 |
| registered sources | 31 |
| published topics | 93 |
| type | objects |
|---|---|
| entity | 25 |
| guide | 9 |
| comparison | 2 |
| faq | 1 |
| glossary | 1 |
Subject matter: evals and benchmarks (GAIA, AgentBench, Inspect), LLM-as-judge and its failure modes, Goodhart and benchmark contamination, fault injection and chaos testing, approval gates and autonomy levels, grounding and faithfulness.
Questions it is built to answer
- How do I tell a real eval from a benchmark my agent has memorised?
- What does calibration mean for an LLM judge, and how is it measured?
- Which failure modes does fault injection actually catch?
What an answer actually looks like
A real call against the live endpoint — answer with
"how do I tell a real eval from benchmark contamination" — returns this structuredContent, trimmed:
{
"answered": true,
"entity": {
"id": "agent-reliability-glossary",
"name": "Agent reliability glossary",
"evidence_tier": "secondary",
"confidence": 0.85,
"last_verified": "2026-08-08",
"canonical_url": "https://agentreliability.dev/k/agent-reliability-glossary"
},
"claims": [
{
"text": "An eval is a structured, repeatable test that measures an LLM or LLM-based system against a defined dimension; frameworks package evals as registries of reusable templates.",
"sources": [{ "title": "openai/evals — framework for evaluating LLMs and LLM systems" }]
}
]
}
Note what travels with the answer: the evidence tier, a confidence, the date it was last verified, and the source behind the claim — not as prose an agent has to parse, but as fields it can act on. An agent can decline to use a weak claim, or cite the primary source directly.
When the corpus cannot answer, answered is false. It does not
improvise, and the miss is recorded so the gap can be filled.
Machine-readable surfaces
The MCP endpoint is one of several. The same corpus is served as plain files an agent can read directly:
| surface | what it is |
|---|---|
/llms.txt |
the index, as text/plain |
/llms-full.txt |
the whole corpus in one file |
/ai-index.json |
every surface this instance publishes, with its content type |
/api/index.json |
one JSON document per knowledge object |
/api/sources.json |
the source registry, in full |
/.well-known/mcp/server.json |
this server's manifest |
Each knowledge object has a human page and a machine twin at the same id, with a canonical URL that agrees across all of them.
Behaviour worth knowing before you integrate
POSTonly. Every other method answers405with anAllow: POST, OPTIONSheader.- Rate limit: 120 requests per minute per client, counted in a shared
store, published on every response as
RateLimit-Limit,RateLimit-RemainingandRateLimit-Reset(all three exposed via CORS). It fails open: if the store is unreachable the request is served. - Malformed input gets a spec-correct JSON-RPC error —
-32700for unparseable bodies,-32602for an unknown tool — never an HTML error page. - Request bodies are capped and validated before transport.
Privacy
No accounts, no cookies, no ads. Usage is measured in aggregate with daily-rotating hashed identifiers and a 200-day retention; raw IPs are never stored. Full policy: PRIVACY.md.
Provenance and licence
Knowledge content is CC-BY-4.0: use it, cite it. The source registry is
public precisely so a claim can be checked rather than trusted —
get_sources returns what any given claim rests on.
Claims carry an evidence tier and a last_verified date. Where the
evidence is weaker, the object says so rather than rounding up.
How it is built
Compiled and served by Citarium, a source-available framework for turning a knowledge graph into a website, an API, an MCP server and agent-readable files from a single source, under external evaluation.
The framework's code is licensed under the Business Source License 1.1 and
its repository is not public. What is public — and what actually matters for
trusting an answer — is this server, the corpus it serves, and the registered
source behind every claim: get_sources returns what any given claim rests
on, so it can be checked rather than trusted.
This repository is the server's public face: its manifest and its documentation. The corpus itself lives at agentreliability.dev.
Running it yourself
The endpoint above is the server. If you want to run a copy locally — or
if a directory needs to start one in order to inspect it — the recipe is
in CONTAINER.md: fetch-corpus.mjs downloads this
instance's published corpus and @citarium/mcp serves it. Both files are
GENERATED by citarium mirror; edit the instance, not them.
The container serves a snapshot taken when its image was built. This endpoint is the thing itself.
Agent Reliability MCP questions
How do I add Agent Reliability MCP to Claude Code?
Run claude mcp add --transport http agentreliability-mcp 'https://agentreliability.dev/mcp?via=manifest' in your terminal. Add --scope user to make it available in every project.
How do I add Agent Reliability MCP to Cursor?
Open ~/.cursor/mcp.json and add the server under mcpServers, using the Cursor config shown in the install section above.
What tools does Agent Reliability MCP provide?
It exposes 8 tools, including get_overview, search, answer, get_entity, get_topic.
Does this server require an API key?
No, the server requires no API key, no account, and no authentication for read-only access.
How can I verify the information provided by the server?
Every claim returned is bound to a registered source, providing citations and a confidence value so agents can show their work.
What happens if the corpus cannot answer a question?
The server does not improvise; it returns a response where the answered field is false and records the gap.
What are the rate limits for the endpoint?
The rate limit is 120 requests per minute per client.