context-report MCP server
Commit Nov 2, 2026Provides an open, signed report format to verify the efficacy, reachability, and cost of AI agent plugins, hooks, skills, and AGENTS.md files.
Also known as open-coder-ai/context-report
- Stars
- 1
- Tools
- —
- Downloads, 30 days
- —
- Latest version
- —
- Last commit
- Nov 2, 2026
- License
- Apache-2.0
What is context-report MCP?
context-report provides an open, signed report format to verify the efficacy, reachability, and cost of AI agent plugins, hooks, skills, and AGENTS.md files.
Summary generated from the project's README
About context-report MCP server
Verifying the reliability and cost of AI agent components is what context-report enables for developers building plugins, hooks, skills, or AGENTS.md files. It provides an open, signed report format that documents whether an artifact actually works by measuring specific facts rather than providing a simple pass or fail grade. This allows consumers to set their own thresholds based on evidence regarding reachability, efficacy, and resource usage.
Users interact with the tool via a command line interface using Python 3.10 or newer. The process involves using context-report produce to run a subject's own hook commands against recorded payloads, followed by context-report verify to check that re-derivable rows match the subject. The system is designed for security and privacy, ensuring the production phase requires no LLM or tokens, as it avoids importing networking or model modules. It serves teams integrating guardrails into CI/CD pipelines or developers creating plugins for clients like Claude Code, Cursor, and Copilot.
What you can do with it
- 1Verify plugin reachability and costAn AI developer can generate a signed report to prove a plugin's token usage and whether it is reachable from different working directories.
- 2Audit agentic guardrailsSecurity teams can use the tool to ensure that hooks designed to block unsafe code or credential leaks are actually functioning as intended.
Written from the project's README. Every use case is checked against a quote from it.
Automated checks
Ran Nov 15, 2026Full report →Not verified yet. A server needs a score of 80, no serious finding, and enough data for us to check. How checks work · What to fix for more points
- Source repository linked and reachableopen-coder-ai/context-report is public and reachablegithub.com/open-coder-ai/context-report ↗Pass
- Package names this repository as its sourceNot checked
- Build provenance (Sigstore attestation)Not checked
- Open source license (OSI)Apache-2.0spdx.org/licenses/Apache-2.0.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-11-02github.com/open-coder-ai/context-report ↗Pass
- OpenSSF ScorecardOpenSSF Scorecard 6.9 / 10; weakest: Maintained 0, Code-Review 0, CII-Best-Practices 0scorecard.dev/viewer/?uri=github.com/open-coder-ai/context-r ↗Partial 10/15
- Package name not close to a popular packageNot checked
- No critical advisory in the package or its dependenciesNot checked
- No hidden instructions to the model0 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNot checked
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubTeach your AI agent what not to do.
Open-source guardrails for AI coding agents: rules the agent reads, checks that run as it writes, and gates at commit and in CI. context-report is the evidence arm: an open, signed report of whether an agent plugin, hook, skill or AGENTS.md actually works.
chock · chock-catalog · chock.sh (launching soon)
context-report is an open, signed report format for one question: does this agent context artifact actually work? A plugin, hook, skill, AGENTS.md or MCP server gets one row per measured fact, each row declaring whether it is recomputable by anyone or only claimed by the author, and never a single pass or fail for the artifact as a whole: the consumer sets its own thresholds. Free and open source (Apache-2.0).
Application security for the code your agents write
A guard is only a guard if it starts. Chock's checks refuse known classes of flaw as the agent writes, at commit and in CI; many of them ship as agent plugins and hooks, and nobody records whether such an artifact reaches the agent at all, how it fails when it cannot run, or what it costs. context-report is the evidence for that: a predicate an author's CI produces and a catalog verifies at submission. A rule an agent reads is advice; a hook that exits non-zero is a control; context-report measures which one you actually have. The findings are under What it stops, below.
| Area | What gets refused | Policy | Tier |
|---|---|---|---|
| Java & Kotlin | injection, XXE, SSRF, unsafe deserialization, weak crypto, dependencies below a known fix | java-security |
commit |
| Unsafe code, IAM | eval, shell=True, os.system, pickle; IAM Action: * |
block-unsafe-code-execution, block-wildcard-iam |
commit |
| Supply chain | dependencies off an allowlist, Actions on a mutable tag, MCP servers and images at @latest |
verify-dependency-exists, pin-github-actions, block-unpinned-agent-components |
commit |
| Agent code | host execution, approvals switched off, credential leaks | agentic-code-security |
commit |
| OWASP Agentic Top 10 | a policy for each of ASI01 to ASI10; 7 have a slice refused at commit; 0 are fully covered | owasp-asi01…10 |
advisory |
| Accessibility | a stripped alt, aria-label, label or lang |
no-a11y-regression |
commit |
| Prompt injection, memory | bidi and tag characters; secrets written into agent memory | block-invisible-unicode, guard-memory-writes |
commit |
| Test integrity | deleted tests, lost assertions, new skips | protect-test-integrity |
commit |
| Also included | secrets, destructive commands, agent self-protection | scan-secrets, block-destructive-commands, protect-agent-config |
commit, in-agent |
No LLM, no tokens in the producer. context-report produce (reachability, cost and fault rows) runs the subject's own hook command on recorded payloads; nothing under src/context_report/produce imports a networking or model module (grep -rln "urllib\|anthropic\|http.client\|socket\|requests" src/context_report/produce, empty at 5fbef7b). Only the efficacy path, context-report run, calls the subject and judge models you configure. Chock adds no new place your code goes, and a report is a set of hashes and measurements.
Install
chock is on PyPI, but the release there (0.15.2, 30 Sep 2026) is older than the engine this page describes. Install the frozen engine from its commit (Python 3.11 or newer):
pip install "chock @ git+https://github.com/open-coder-ai/chock@992711af4cf8d4fd9c4c861f10ef6e53374d75d7"
Two ways to adopt it
- In your repository, for teams. Run
chock init ., thenchock add <id> --ref <catalog commit> --verify-sha <sha256> --skip-compilefor each policy, thenchock sync --repo . --ci. Commit the result. Every clone runschock sync --repo .once, because git never clones hooks. The commit gates are enforced at commit and in CI. - In your coding agent, as plugins. Best-effort, and they fail open. One repo per client: Claude Code, Cursor, Copilot, Codex, Devin. Each README has the install line for its client.
- One Claude Code plugin from a selection. The chock.sh builder (launching soon) gives a
chock install --selection '…' --applycommand.
Install context-report
pip install context-report (Python 3.10 or newer; the quickstart below runs end to end).
30-second quickstart
pip install context-report
mkdir -p my-plugin/hooks
cat > my-plugin/hooks/guard.py <<'PY'
#!/usr/bin/env python3
import json, sys
event = json.load(sys.stdin)
command = event.get("tool_input", {}).get("command", "")
if "destructive-pattern" in command:
print(json.dumps({"decision": "deny", "reason": "blocked destructive command"}))
sys.exit(0)
PY
cat > my-plugin/hooks/hooks.json <<'JSON'
{
"hooks": {
"PreToolUse": [
{"hooks": [{"command": "python3", "args": ["${CLAUDE_PLUGIN_ROOT}/hooks/guard.py"]}]}
]
}
}
JSON
context-report produce --subject ./my-plugin --kind plugin --target claude_code --n 20 --out report.json
context-report verify report.json --subject ./my-plugin
verify reprints each row's basis and result, then ends with well-formed and bound: True —
never a "pass" for the artifact as a whole. report.json is one row per fact; each row's basis
is re-derivable (anyone can recompute it from the subject) or claimed (the author asserts
it). Three real rows from the report.json this exact block just produced:
[
{
"attribute": "reachability",
"basis": "re-derivable",
"result": "FAILED",
"inputHash": "sha256:fc126ed825ddfeb440a0d1d5bf133780e4b275b8c4c18d1556ea5ab251405bc9",
"conditions": {"cwdTested": ["root", "nested", "parent", "outside"]},
"values": {"reachable_from": ["parent"], "unreachable_from": ["root", "nested", "outside"]}
},
{
"attribute": "fault.malformedOutput",
"basis": "re-derivable",
"result": "PASSED",
"inputHash": "sha256:85fab0cf63d9db25ab5aba6ada7d396feceb0474f3018f83c24ce542441c1a88",
"conditions": {"cases": ["on_malformed_json", "on_empty_stdin", "on_null_tool_input", "control_benign"]},
"reasoning": "exit 0 on malformed input is how a Claude Code hook fails open; whether that is acceptable is the consumer's threshold."
},
{
"attribute": "cost.context_tokens",
"basis": "re-derivable",
"result": "PASSED",
"inputHash": "sha256:8d2af315178e658732a9e48147395296f8b33277bfb7413d55ad9722053b9026",
"conditions": {"tokenizer": "approx-regex-v1", "files": 1},
"measurement": {"unit": "tokens", "n": 1, "min": 50, "max": 50, "mean": 50}
}
]
reachability FAILED here is not a bug in the example: ${CLAUDE_PLUGIN_ROOT} resolves to the
relative path ./my-plugin you passed, so the hook only starts from the one cwd where that path
still points at the plugin — the exact failure mode the measurement below calls out.
The full statement has 11 rows, not 3 — every row's basis, from this exact quickstart block run fresh:
Beyond one statement at a time, context-report run drives a whole manifest — subjects × models ×
tasks — and context-report compare puts every run of that manifest side by side; see
docs/cli.md for the manifest schema, --dry-run/--n/--resume, and which
model providers a manifest can reach.
How it works
| Layer | What it is |
|---|---|
| Evidence | context-report: a signed report of whether a plugin, hook or skill works. chock-threat-intel: a weekly ledger of threats, scored against the catalog. |
| Your agents | Claude Code, Cursor, Copilot, Codex, Gemini, Windsurf, Devin and the rest of the matrix. |
| Plugins | One generated repo per client: Claude Code, Cursor, Copilot, Codex, Devin. |
| chock-catalog | 71 policies, each labelled by what it enforces. Installed with chock add <id>. |
| chock | Governance as code: one policy becomes a git hook, a CI gate, native agent hooks and an AGENTS.md rule. |
| agentseam | The primitives: one handler API over each agent's hooks, instruction files, plugin packaging and config. |
| Templates | chock-quickstart is what chock init leaves behind. chock-example is a working adoption. |
A producer runs the artifact's own hook command against recorded per-target payloads and writes one row per fact.
A verifier recomputes each re-derivable row from the subject and checks the statement is well-formed and
bound to the artifact by digest. Trimmed from a committed statement over a real public plugin
(paper/measurements/catalog-sample/official/ai-plugins.json):
{
"subjectKind": "plugin",
"target": {"name": "claude_code"},
"attributes": [
{
"attribute": "cost.context_tokens",
"basis": "re-derivable",
"result": "PASSED",
"inputHash": "sha256:4e85a09a2014600e...",
"conditions": {"tokenizer": "approx-regex-v1", "files": 2},
"measurement": {"unit": "tokens", "n": 1, "mean": 2289}
}
]
}
v0.1 rows: conformance · reachability · decision · fault.scriptMissing ·
fault.interpreterMissing · fault.timeout · fault.malformedOutput · cost.latency_ms ·
cost.context_tokens · interference · efficacy (extensions use an x- prefix). A row that
could not be measured says NotAvailable, Error or NotApplicable and why — never a silent
pass. v0.1 draft: schema at
spec/attestation/v0.1/schema.json, worked example at
spec/attestation/v0.1/examples/plugin-copilot.json,
predicate type https://open-coder-ai.github.io/context-report/attestation/v0.1, hosted at
https://open-coder-ai.github.io/context-report/attestation/v0.1/.
Two models, not one
Efficacy needs two roles, never one: the subject model runs a task with the rule prepended and without it; the judge model never performs the task, only reads the transcript and decides whether that arm met the rule's criterion, held fixed across every subject model so a comparison across models is fair. A machine-checkable criterion is graded by code instead, never guessed at.
Subject models come from the manifest, never from code — one manifest lines up every model you can reach, the API-shaped ones answering without tools or a checkout:
| Provider | What it reaches |
|---|---|
anthropic |
the Anthropic API |
claude-cli |
the local claude CLI — compares opus, sonnet and fable under one account login |
openai-compatible |
any server speaking the chat-completions shape, given a baseUrl: hosted (OpenAI, Gemini, Mistral, Groq) or local (Ollama, vLLM, LM Studio) |
| anything else | an honest NotAvailable efficacy row, never a guess |
See docs/cli.md for the full picture of what a manifest
can reach.
Use as a library
Beyond the CLI, context_report exposes a small stable API for a catalog or CI job to import
directly: validate, verify, produce_statement, load_manifest, run, resolve_run_dir,
history_markdown, rule_history_markdown, render_table, render_history (see __all__ in
context_report/__init__.py).
from context_report import validate, verify
errors = validate(stmt) # schema errors, [] means well-formed
result = verify(stmt, subject_path="clone/") # bound + schema check, never a verdict
See docs/library.md for a full catalog-verification and CI-production example.
What it stops
context-report stops nothing by itself; it makes a guard's failure visible before you rely on it. Each row below is a measured finding from the measurement paper, over chock's bundles, a sample of 18 public Claude Code plugins and seven third-party instruction files and skills.
| Finding | Reading for a guard |
|---|---|
3 of 18 public Claude Code plugins (carta-cap-table, carta-crm, carta-investors) share a dispatch script with no execute bit: reachability FAILED, 0 of 4, exit 126 |
the hook is registered and never runs |
| Every hook that runs, across both samples, allows on malformed input | hooks fail open on malformed input |
npx hooks cost 916.8 to 941.5 ms p50; local scripts 7.3 to 53.9 ms; context weight about 1,500 to 147,000 tokens |
cost varies by two orders of magnitude, per tool call and per context window |
No efficacy row reaches PASSED; a prompt-injection rule moved nothing on any model |
measure a rule before relying on it |
Source for each row: paper §5.2 (catalog plugins) and §5.3 (instruction files, three models). Timings: produced 2026-09-06, n = 20 samples per hook, on a four-CPU Linux machine (paper line 226).
Three measurements
Reachable is not the same as executable. Of 18 public plugins, three (carta-cap-table,
carta-crm, carta-investors) share a dispatch script with no execute bit — reachability
FAILED, 0 of 4, exit 126. Every hook that runs, across both samples, allows on malformed input.
See §5.2.
Cost spans two orders of magnitude. Hooks that shell out to npx cost 916.8–941.5 ms p50; a
local script costs 7.3–53.9 ms. Context weight varies about a hundredfold across the sample,
roughly 1,500 to 147,000 tokens. See §5.2.
No efficacy row reaches PASSED. Three instruction files, ablated on opus, sonnet and
fable (168 recorded transcripts, one judge model held fixed): with four observations per arm the
95% interval is about ±0.49 wide, and the row reports the interval instead of rounding it to a
verdict. A naming-convention rule was the one consistent positive (+0.25 to +0.50 on every model);
a prompt-injection rule moved nothing on any model. See
§5.3.
Who it's for
| Who | What they want |
|---|---|
| An artifact author | a report their own CI can produce before anyone else asks for one |
| A catalog maintainer | a submission format their existing verifier can check without adopting anyone else's test suite, and a re-derivable/claimed split to build a policy on |
| A researcher or reviewer | a re-derivable record of what was actually measured, not a vendor's prose description of it |
Supported agents
produce drives the subject's own hook command through recorded per-target payloads — no live
agent required. reachability, cost.latency_ms and fault.malformedOutput are re-derivable for
every target below; the three fault rows that need a live client (fault.scriptMissing,
fault.interpreterMissing, fault.timeout) are always NotAvailable in v0.1 and cite a vendor-docs
oracle where one is on file (none yet for codex_cli — see
Good first contributions).
| Agent | What is measured | Config file |
|---|---|---|
claude_code |
reachability · cost · fault (malformedOutput measured; scriptMissing/interpreterMissing/timeout NotAvailable, oracle on file) |
hooks/hooks.json (+ .claude-plugin/plugin.json) |
codex_cli |
reachability · cost · fault (malformedOutput measured; scriptMissing/interpreterMissing/timeout NotAvailable, no oracle on file) |
hooks/hooks.json |
copilot |
reachability · cost · fault (malformedOutput measured; scriptMissing/interpreterMissing/timeout NotAvailable, oracle on file) |
com.github.copilot/hooks/hooks.json |
cursor |
reachability · cost · fault (malformedOutput measured; scriptMissing/interpreterMissing/timeout NotAvailable, oracle on file) |
hooks/hooks.json |
Guardrails, not guarantees
Tiers: commit is a git hook or CI gate that exits non-zero. in-agent is the agent's pre-tool hook: best-effort, and it fails open. advisory is rule text the agent reads. No agent reaches enforced today. OWASP mappings are partial and the engine is frozen at the commit above. Chock does not stop every attack, and context-report does not stop any: it closes common, known entry points before they ship and makes a guard's failure visible.
FAQ for people and agents
Does context-report use an LLM? The producer for reachability, cost and fault rows does not: it makes no model call. Efficacy rows use the subject and judge models you configure in a manifest; without one the row says NotAvailable.
Does my code leave my machine? context-report adds no new place your code goes. produce and verify run locally; run sends prompts to the model providers you configure.
Which agents does it work with? claude_code, codex_cli, copilot and cursor (see Supported agents).
How do I install it, through the repo or through plugins? pip install context-report for the tool. To adopt the guardrails it measures, see Install above.
What does it cost? Free and open source (Apache-2.0). A produce run costs no tokens.
Does it replace SAST or code review? No. It reports on agent artifacts (plugins, hooks, skills); it does not scan your code.
Which OWASP and CWE items does it cover? None; it maps no standards. The catalog's OWASP mappings are partial and labelled so, in chock-catalog docs/coverage.md.
For tools and agents
llms.txt: a short machine-readable summary of this repo.spec/attestation/v0.1/schema.json: the predicate schema; predicate typehttps://open-coder-ai.github.io/context-report/attestation/v0.1.spec/attestation/v0.1/attributes.md: every row's definition.spec/attestation/v0.1/examples/plugin-copilot.json: a worked example.registry.yamlanddocs/coverage.md: the catalog's policies and coverage..claude-plugin/marketplace.json: the Claude plugin marketplace.
Part of open-coder-ai
The 13 public repositories:
| Repository | What it is |
|---|---|
| agentseam | Core: One handler API over every coding agent. |
| chock | Core: Author a policy once, enforce it on every agent. |
| chock-catalog | Policies: The policies, each labelled by what it enforces, with replayed evals. |
| context-report | Evidence: A signed report of whether an agent artifact works. |
| chock-threat-intel | Evidence: A weekly threat ledger, each entry scored against the catalog. |
| chock-claude-plugins | Plugins: The catalog as Claude Code plugins (generated). |
| chock-copilot-plugins | Plugins: The catalog as Copilot CLI and VS Code plugins (generated). |
| chock-cursor-plugins | Plugins: The catalog as Cursor plugins (generated). |
| chock-codex-plugins | Plugins: The catalog as Codex plugins (generated). |
| chock-devin-plugins | Plugins: The catalog as Devin plugins (generated). |
| chock-quickstart | Template: What chock init leaves behind. |
| chock-example | Template: A working adoption, one policy per layer. |
| .github | Community: Org profile and community health files. |
context-report MCP questions
What does context-report actually measure?
It measures whether an agent context artifact like a plugin, hook, or skill actually works by recording facts such as reachability, cost, and fault tolerance.
Does generating a report require an LLM or tokens?
No, the production process does not use LLMs or tokens; it runs the subject's own hook command on recorded payloads.
How are the results in the report categorized?
Each row in the report declares whether the fact is recomputable by anyone (re-derivable) or only claimed by the author (claimed).