pmstack plugin
Error discovery for AI products: read real traces, name failure modes and success modes by stage, and turn the ones that matter into code checks and AI judges you can trust. Includes Eval Studio and a zero-dependency command-line tool.
How to install the pmstack plugin
Run both commands inside Claude Code. The first adds the marketplace once; the second installs the plugin.
/plugin marketplace add RyanAlberts/pmstack
/plugin install pmstack@pmstackWhat's inside pmstack
24 parts read from the plugin's own files.
Commands 12
Run as /pmstack:name- /pmstack:build-judge[project.json] [failure mode]
Build an AI judge for one failure mode
- /pmstack:check-sources[folder or project.json]
Check an answer bot that searches documents: did it find the right sources, and does the answer stick to them?
- /pmstack:custom-view[folder]
Show your traces the way your customer saw them with a custom view in Eval Studio
- /pmstack:evals-checkup[folder, repository, or eval files]
Give the evals you already have a checkup, and list what to fix first
- /pmstack:find-failures[folder or trace file]
Find how your AI product fails: review real traces in Eval Studio and group your notes into failure modes
- /pmstack:make-traces[what your product does]
Make realistic test traces when you have no real ones yet
- /pmstack:regression-checks[project.json]
Run your checks on every code change and keep watching production
- /pmstack:start[what you are working on]
Pick where to start: real traces, no traces yet, an agent that uses tools, or evals you inherited
- /pmstack:test-judge[project.json] [check id]
Test whether an AI judge agrees with you, then estimate how often the product really fails
- /pmstack:tool-grounding[project.json]
Check that an agent's replies match what its tools returned
- /pmstack:tool-policy[trace file or project.json]
Write your company's rules for an agent's tool calls, test them on past traces, and run them on every change
- /pmstack:tool-relevance[project.json]
Check that an agent calls the right tools, with the right details, for what the customer asked
Skills 12
Claude loads it when the task matches- pmstack-build-judge
Builds and runs AI judges to evaluate specific failure modes in AI traces using Pass/Fail rules and example-based prompting.
- pmstack-check-sources
Evaluates RAG-based answer bots by separately measuring retrieval accuracy and answer faithfulness using Recall@k and mean reciprocal rank.
- pmstack-custom-view
Creates custom trace visualizations for Eval Studio using panel layouts or custom MJS renderer files.
- pmstack-evals-checkup
Audits existing AI evaluation setups by inspecting artifacts to identify quality risks and providing concrete fixes ordered by product impact.
- pmstack-find-failures
Analyzes AI product traces to identify failure modes, suggest groupings, and manage error analysis through the Eval Studio interface.
- pmstack-make-traces
Generates realistic synthetic test traces for AI products by creating diverse user request combinations and running them through the system.
- pmstack-regression-checks
Automates regression testing and quality monitoring for AI products by replaying traces and running automated checks.
- pmstack-start
Routes users to specific pmstack skills to identify AI product failures, group them into failure modes, and build automated checks or AI judges.
- pmstack-test-judge
Validates an AI judge by comparing its verdicts against human labels and estimating the true failure rate with a 95% confidence range.
- pmstack-tool-grounding
Verifies that AI agent replies accurately reflect tool outputs by implementing code-based value checks and AI-driven grounding judges.
- pmstack-tool-policy
Writes and tests tool policies for AI agents to enforce company security, compliance, and permission rules during tool calls.
- pmstack-tool-relevance
Evaluates whether an AI agent's tool calls are relevant to the user's request by using intent maps or AI judge templates.
Using pmstack in Claude Code
Type a command in a Claude Code session:
/pmstack:build-judge [project.json] [failure mode]
/pmstack:check-sources [folder or project.json]
/pmstack:custom-view [folder]Skills need no command. Claude reads them when your request matches what they describe.
Update, turn off or remove
- Update the plugin (shell)
- claude plugin update pmstack@pmstack
- Refresh the marketplace
- /plugin marketplace update pmstack
- Turn it off for now
- /plugin disable pmstack@pmstack
- Remove it
- /plugin uninstall pmstack@pmstack
Skills in pmstack
- pmstack-find-failuresRyanAlberts/pmstack
Analyzes AI product traces to identify failure modes, suggest groupings, and manage error analysis through the Eval Studio interface.
2 files9stars - pmstack-build-judgeRyanAlberts/pmstack
Builds and runs AI judges to evaluate specific failure modes in AI traces using Pass/Fail rules and example-based prompting.
9stars - pmstack-check-sourcesRyanAlberts/pmstack
Evaluates RAG-based answer bots by separately measuring retrieval accuracy and answer faithfulness using Recall@k and mean reciprocal rank.
9stars - pmstack-custom-viewRyanAlberts/pmstack
Creates custom trace visualizations for Eval Studio using panel layouts or custom MJS renderer files.
9stars - pmstack-evals-checkupRyanAlberts/pmstack
Audits existing AI evaluation setups by inspecting artifacts to identify quality risks and providing concrete fixes ordered by product impact.
9stars - pmstack-make-tracesRyanAlberts/pmstack
Generates realistic synthetic test traces for AI products by creating diverse user request combinations and running them through the system.
9stars - pmstack-regression-checksRyanAlberts/pmstack
Automates regression testing and quality monitoring for AI products by replaying traces and running automated checks.
9stars - pmstack-startRyanAlberts/pmstack
Routes users to specific pmstack skills to identify AI product failures, group them into failure modes, and build automated checks or AI judges.
9stars - pmstack-test-judgeRyanAlberts/pmstack
Validates an AI judge by comparing its verdicts against human labels and estimating the true failure rate with a 95% confidence range.
9stars - pmstack-tool-groundingRyanAlberts/pmstack
Verifies that AI agent replies accurately reflect tool outputs by implementing code-based value checks and AI-driven grounding judges.
9stars - pmstack-tool-policyRyanAlberts/pmstack
Writes and tests tool policies for AI agents to enforce company security, compliance, and permission rules during tool calls.
9stars - pmstack-tool-relevanceRyanAlberts/pmstack
Evaluates whether an AI agent's tool calls are relevant to the user's request by using intent maps or AI judge templates.
9stars
pmstack plugin questions
How do I install the pmstack plugin?
In Claude Code run /plugin marketplace add RyanAlberts/pmstack, then /plugin install pmstack@pmstack.
What does the pmstack plugin include?
pmstack bundles 12 commands, 12 skills.
What commands does the pmstack plugin add?
It adds /pmstack:build-judge, /pmstack:check-sources, /pmstack:custom-view, /pmstack:evals-checkup, /pmstack:find-failures, /pmstack:make-traces +6. Run them in Claude Code with the plugin prefix, for example /pmstack:build-judge.
How do I update or remove the pmstack plugin?
Update it with claude plugin update pmstack@pmstack in your shell, or from the Installed tab of /plugin. Remove it with /plugin uninstall pmstack@pmstack.
What license is the pmstack plugin under?
MIT, as stated in the plugin's manifest.