Plugin pmstack

Build an AI judge for one failure mode

Check an answer bot that searches documents: did it find the right sources, and does the answer stick to them?

Show your traces the way your customer saw them with a custom view in Eval Studio

Give the evals you already have a checkup, and list what to fix first

Find how your AI product fails: review real traces in Eval Studio and group your notes into failure modes

Make realistic test traces when you have no real ones yet

Run your checks on every code change and keep watching production

Pick where to start: real traces, no traces yet, an agent that uses tools, or evals you inherited

Test whether an AI judge agrees with you, then estimate how often the product really fails

Check that an agent's replies match what its tools returned

Write your company's rules for an agent's tool calls, test them on past traces, and run them on every change

Check that an agent calls the right tools, with the right details, for what the customer asked

Builds and runs AI judges to evaluate specific failure modes in AI traces using Pass/Fail rules and example-based prompting.

Evaluates RAG-based answer bots by separately measuring retrieval accuracy and answer faithfulness using Recall@k and mean reciprocal rank.

Creates custom trace visualizations for Eval Studio using panel layouts or custom MJS renderer files.

Audits existing AI evaluation setups by inspecting artifacts to identify quality risks and providing concrete fixes ordered by product impact.

Analyzes AI product traces to identify failure modes, suggest groupings, and manage error analysis through the Eval Studio interface.

Generates realistic synthetic test traces for AI products by creating diverse user request combinations and running them through the system.

Automates regression testing and quality monitoring for AI products by replaying traces and running automated checks.

Routes users to specific pmstack skills to identify AI product failures, group them into failure modes, and build automated checks or AI judges.

Validates an AI judge by comparing its verdicts against human labels and estimating the true failure rate with a 95% confidence range.

Verifies that AI agent replies accurately reflect tool outputs by implementing code-based value checks and AI-driven grounding judges.

Writes and tests tool policies for AI agents to enforce company security, compliance, and permission rules during tool calls.

Evaluates whether an AI agent's tool calls are relevant to the user's request by using intent maps or AI judge templates.