flowproof MCP server

flowproof

End-to-end tests an AI agent records for you

flowproof is open-source E2E testing for the age of agents. Write the test the way you would explain it to a colleague. An AI agent performs it once against your real application, and while it works, flowproof records what it did: the element it found for every step, the action it took, and what it checked. That recording is the test. From then on a deterministic engine replays it on every commit with zero LLM calls: no API key, nothing billed, nothing left to sampling.

# examples/tricentis-insurance-natural.flow.yaml
name: Tricentis Insurance (natural)
app: web
url: https://sampleapp.tricentis.com/101
steps:
  - Open the truck insurance quote
  - Fill out all the vehicle data and click next
  - Fill out all the insurant data and click next
  - Fill out all the product data and click next
  - Pick one of the price options
  - Fill in what sending the quote needs and send it
  - assert: page shows Sending e-mail success! within 20s

flowproof record examples/tricentis-insurance-natural.flow.yaml   # the agent drives the app, once
flowproof run examples/tricentis-insurance-natural.flow.yaml      # replays the recording, no model

No selectors, no page objects. "Fill out all the vehicle data and click next" is one step: the agent fills every field it finds on the live page and clicks Next, and the committed trace holds that as fourteen actions, each replayed on its own. (The file also pins the viewport and the clock; the trace sits beside it.)

What the agent does while it records:

• It works from what is on screen. Each step is matched against the live app's real elements (the DOM, the UI Automation tree, SAP scripting ids, OCR on a Citrix screen), so it cannot invent a selector that is not there.
• It records a ladder, not one selector. Native id, then structure, then a text anchor, so a renamed id degrades a step instead of failing it (trace format).
• It asks rather than guesses. A step too vague to perform ("make the required field changes") comes back as the stuck step plus the live screen's field inventory, so you, or the coding agent driving flowproof, can say what was meant (self-help).

What happens after: every run writes report.html, junit.xml, result.json and screenshot checkpoints. When the app changes, replay falls back rung by rung and flags the degraded match before the last rung dies; flowproof heal sends the agent back to re-author the broken steps against the live app and proposes a trace diff with before/after frames. Nothing changes until you pass --apply (resilience and healing).

One spec format, every surface: web (Chromium over CDP), Windows desktop (UI Automation), SAP GUI, Citrix and RDP (pixels and OCR), HTTP and SQL, and AI agents themselves.

Your bot behaved today. Prove it'll behave tomorrow.

The same record-once, replay-forever mechanism tests an agent of your own.

Software robots already do real work in production. They raise orders in SAP, update records, close tickets: jobs a person used to do by hand, and jobs agents are taking over more of every month. flowproof is how you test that one of them only ever does the job it was given, and never deletes, sends or approves the thing it must not.

Record the bot doing its job once. After that, every commit runs that recording again and checks it: no model, no API key, nothing billed. Name the lines it must never cross, and the build breaks the instant one is crossed. Not a suite you hope still passes. Evidence, every push.

Everywhere your bots work: web, desktop, SAP, Citrix.

One spec, one record against a real model, then run forever with none. The frames are a capture of scripts/demo/ actually running, not a mock-up. The same three commands carry the guard path: add assert_no_tool_call and the call an agent must never make is proven on every commit (agent-boundary testing).

The same promise, precisely

A rule the bot has to obey (never refund without approval, never delete a customer) is what flowproof calls a control, and its job is to keep proving that control still holds.

Record your agent's real run once: every model request and every tool-call decision it made. From then on, replay it with zero LLM calls and assert what matters: which tools were called, with which arguments, and which were never called. flowproof audit --since exits non-zero the moment a control starts failing or quietly stops being checked, so a guarantee you made last month is re-proved on every commit instead of being remembered.

The same spec format drives web, Windows, SAP GUI, Citrix and HTTP. A bot can go wrong in two places (what it decided to do, and what it actually did to your system), and one recording covers both. An agent that files an SAP order is one job to get right, not two.

Three verbs, three mechanisms, and they are not equally strong. Worth knowing which one is holding a given line, because the answer decides what a green build actually proves:

The claim	Backed by	What it is
never approves	assert_no_tool_call	the agent did not ask for the call, on any platform
never deletes	assert_no_tool_call	same, when the delete goes through a tool, which is how an agent deletes a customer
never sends	assert_no_egress	a real seccomp filter refused it, on Linux; the step fails outright anywhere it cannot be enforced, rather than passing vacuously

What is not asserted: a file the agent's own process destroys directly. On Linux that is now reported (the run prints what it unlinked, renamed or truncated), but a report is not a control, and no build breaks on one. See filesystem observation.

Flows are plain YAML - short enough for an agent to write, readable enough for a human to review in a diff.

Adding it to an existing agent? docs/adopting.md is written to be handed to a coding agent.

Product page: automators.ai/flowproof

Choose your path

Goal	Start here
Record and replay your first UI or agent flow	Record and replay
Add Flowproof to an existing project	Adopting Flowproof
Run multiple flows with shared data or dependencies	Run a suite
Look up exact action and assertion syntax	Authoring grammar
Test model calls, tool calls, or agent containment	Agent-boundary testing
Choose between web, desktop, SAP, Citrix, API, and agent flows	Live application tests

How it works

Agents author, a deterministic engine executes. An agent performs a flow once from a natural-language YAML spec and records a trace: the resolved selectors, actions, and assertions. The trace replays deterministically in CI with zero LLM calls. When the app changes and a step breaks, healing proposes a reviewable diff, never a silent mutation.

flowproof is built agent-native: the primary caller is a program (usually an AI agent), with humans in an oversight role. Every operation is a library call returning structured results, including a structured "here is what I'd need to know" payload when a step is too ambiguous to author. The CLI and MCP server are thin renderings over the same code paths.

Open-source automation has moved in steps: Selenium made browsers scriptable, Robot Framework widened automation beyond the browser to acceptance testing and RPA, Playwright made web automation reliable. flowproof is built for the next step: the era in which AI agents write and maintain the automation, and what matters is that their output is deterministic to execute and cheap to review.

Two kinds of claim, and the difference matters

flowproof makes two sorts of guarantee, and it is worth being blunt about which is which.

Containment claims: "it cannot." Declare an agent's allowed network with allow_egress and certify it with assert_no_egress, and on Linux the agent runs under an unprivileged default-deny seccomp filter: undeclared destinations are refused by the kernel. Declare a tool under mcp: with a result: and flowproof answers it as the server: the real tool never executes, in either phase. These hold regardless of what the model decides, because a mechanism enforces them rather than an instruction. On macOS and Windows containment is not enforced, and the run reports "not contained" rather than passing vacuously.

Behaviour claims: "it did not, and it is re-checked on every change." assert_no_tool_call proves the agent did not request a forbidden tool given the recorded model response, and the cassette pins every argument byte-exactly, so drift you never thought to assert still fails. This is regression evidence, not proof of impossibility: a model that behaved on the day you recorded is not a model that always will.

A guard flow is strongest when it is paired with enforcement, and when the recording contains a model that genuinely tried. If the only thing between a user and a destructive call is a sentence in a system prompt, that is not a control. flowproof's job is to make that visible, not to paper over it (guard-flow enforcement).

Quick start

npx flowproof --version      # no install, no Python

npm install --save-dev flowproof
# or: pip install flowproof

A spec is natural-language steps and no selectors. This one tests an agent, a real one, built on the official OpenAI SDK, shipped in examples/agent-demo/:

# examples/agent-demo/weather-node.flow.yaml
name: Weather assistant answers with the forecast (Node)
app: agent
agent:
  command: node examples/agent-demo/weather_agent.mjs
tools:
  - name: get_weather
    result: { city: Nairobi, sky: sunny, temp_c: 26 }
steps:
  - prompt: What is the weather in Nairobi right now? Use your tools.
  - assert_tool_call: get_weather where city contains Nairobi
  - assert: reply contains sunny

Record once. The only step that calls a real model, so the only one that needs a key:

npm install openai
npx flowproof config ai             # stores the model API key with a masked prompt
npx flowproof record examples/agent-demo/weather-node.flow.yaml

Replay for ever. No key, no model, no network to the provider:

npx flowproof run examples/agent-demo/weather-node.flow.yaml

The agent ran for real both times: same client, same tool loop. At record flowproof captured the exchange at the model boundary; at replay it served that recording back, so the trajectory is fixed and nothing was billed. assert_tool_call is the part that fails when the agent regresses: wrong tool, wrong argument, or a tool called out of order.

Want a green run before you have a key? The GIF's cassette is committed, so pip install openai && flowproof run scripts/demo/order-status.flow.yaml passes on a fresh clone: no key, no provider network.

Any OS. For a UI instead of an agent, point app: web at a page, or app: api at a flow with no UI at all (examples/); the Windows Calculator walkthrough lives in Record and replay, which is the complete version of this section. Add --json for the structured report on stdout.

Upgrading

pip install --upgrade flowproof
# or: npm install --save-dev flowproof@latest

Every command checks once a day (cached, 1.5s network timeout, never blocks or fails a command) and prints a notice on stderr when a newer release exists. Set FLOWPROOF_NO_UPDATE_CHECK=1 to disable it, e.g. in CI or an air-gapped environment.

Rust crates, the Python wheel and the npm package all move together on one version, so upgrading any one of them gets you the same release everywhere. The trace format carries its own version number (docs/trace-format.md), separate from the flowproof release version, and a format change is called out there and in CHANGELOG.md, which is the source of truth for version-specific behaviour changes, not just a list of new features. Check it before upgrading in CI.

Python API

from flowproof import Flow

flow = Flow("calc.flow.yaml")
flow.record()                    # RecordResult(..., routing=({"route": "llm", ...},))

result = flow.run()              # RunResult, truthy iff the flow passed
result.steps[4].status           # "passed"
result.report_path               # result.json artifact for this run

trace = flow.get_trace()         # inspect the recorded trace programmatically

Agents can drive the same four operations (record, run, get_trace, heal) over MCP: pip install flowproof[mcp], run flowproof-mcp.

What works today

Author: plain scalar steps are human intent: a model grounds them against the live app's real elements via neutral target tokens, and the result replays with zero model calls. It cannot invent selectors. Use rules: <text> for one deterministic step or --author rules for an entire flow; the complete grammar is in the authoring grammar, with every documented form enforced by a test. Anthropic and OpenAI-compatible endpoints (including vLLM) are supported.

• When a step is too ambiguous to author ("make required field changes"), recording returns a structured clarification payload (the stuck step plus the live screen's field inventory) so the driving agent can resolve the ambiguity and re-record (docs/self-help.md).

Execute: deterministic replay with a provenance-tagged selector ladder (native id → structural → text anchor) that falls back rung by rung and flags degraded matches for healing; auto-waiting assertions; --retries for infra flakes. Point run at a directory to execute a whole suite: one shared browser with an isolated context per flow, a suite.yaml manifest for shared env, seed/cleanup hooks, and data minted by an external CLI (env_from → ${VAR}).

Review: every run writes a bundle: result.json, JUnit XML, an HTML report, and screenshot checkpoints. GIF/video assembly is off by default for faster execution; pass --video when a trace-synced recording.gif is useful. Use --recording-detail low or --recording-detail off to reduce capture work further. Add --highlight-cursor to render an unmistakable cursor and click halo into the evidence frames. flowproof heal re-authors a broken flow against the live app and proposes a reviewable trace diff with before/after frames; applied only with explicit --apply.

Reach: adapters behind one spec format:

• app: web: Chromium via DevTools protocol, cross-platform; record shows the browser by default, run (replay) stays headless; override either with --headed/--headless or FLOWPROOF_HEADED; --keep-open leaves a single flow visible for inspection until its window is closed
• Windows desktop via UI Automation (calc, notepad)
• app: sap: SAP GUI Scripting over COM: native scripting ids, transaction-code navigation (Go to /nVA01), SAP virtual keys, SAP Logon launch/connection selection, and optional environment-backed login; an in-memory fake engine keeps the pipeline tested on every platform
• app: vision: pixels-only driving for Citrix/RDP: OCR perception (Windows OCR and pure-Rust ocrs, merged), the accessibility tree when the window has one, operator glyphs recognised by shape, spatial text anchors, real input injection
• app: api: no UI at all: flows made of HTTP and SQL assertions
• app: agent: test an AI agent at the model boundary. Record its trajectory once against a real model, replay it deterministically with zero model calls, and assert the tool calls it makes (agent-boundary testing). On Linux, an agent.command flow can be run under real, unprivileged egress containment (seccomp): declare its network with allow_egress and certify it with assert_no_egress

Verify beyond the UI: out-of-band truth in any flow: assert_sql (postgres) and assert_api (status, body matching, JSON request body, auth headers); and for an app: agent flow, assert_tool_call / assert_no_tool_call over the agent's trajectory. Secrets travel as ${VAR} references: resolved from the environment when the step fires, never stored in a trace; password fields are masked in captured frames.

Prove security controls hold: a control is just a property that must hold, asserted deterministically over a recorded flow. Name one with a stable control: id; assert an access-control denial as a composed pattern (become a low-privilege identity from the suite's identities:, prove it is alive, then assert the denial); catch a leaked secret in agent output with assert_no_secret_leak: ${VAR} (agent flows in v1); and fold the control-bearing flows into a pass/fail/capability-error coverage map with flowproof audit (security controls).

Set up once, diagnose before you spend a key: flowproof config sap / fiori / ai store SAP GUI, Fiori, and model-authoring credentials in one per-machine file, seeded into the environment as a fallback so an explicit shell export or CI secret always wins; flowproof config skill installs an Agent Skill so a coding agent can walk a user through setup instead of them reading docs. flowproof doctor --sap / --fiori / --ai / --agent then report what's actually reachable: connectivity and (for Fiori and AI) a real check, before a flow is written or a key is spent (secrets and configuration). flowproof author-from-doc is an experimental third path into a flow: draft a .flow.yaml from a requirement/test-case PDF, then resolve it with one live record pass (the authoring grammar). flowproof author-from-code drafts one from an outcome and the app's source, each step citing the line of code it came from. flowproof author-from-video drafts one from a screen recording, read in the browser web flows already use.

Debug what a tool sends: flowproof capture is a byte-fidelity HTTP capture endpoint: point a tool-under-test at it and every request is printed and saved verbatim (method, path, all headers, raw body as text and hexdump, plus any SAP /BA1/-style namespace field names), answered 200 so the send side completes. It ships inside the flowproof binary, so it is one command to run (docs/capture.md).

All of it ships as one wheel (PyO3/maturin): Rust engine, Python API, CLI, MCP server. Proven in CI on every push: a Notepad flow records and replays on GitHub's Windows runners, the web adapter drives real Chromium on Linux, the SAP pipeline runs against the fake engine, and vision runs real OCR over synthetic screens.

Architecture

spec (natural language, YAML)
   │  flowproof record          ← model grounds plain human intent; explicit
   ▼                              `rules:` stays deterministic
trace (JSON-lines, versioned)  ← selectors + actions + assertions, ${VAR} refs
   │  flowproof run             ← deterministic, zero LLM calls
   ▼
run bundle                     ← result.json · junit.xml · report.html ·
                                 screenshot checkpoints (--video adds
                                 recording.gif) · healing diffs on drift

Path	What it is
crates/flowproof-driver	Screen/input/UIA driver + out-of-band probes
crates/flowproof-trace	Trace format, selector ladder, secret indirection, JSON Schema
crates/flowproof-replay	Deterministic executor + run reports
crates/flowproof-agent	Spec parsing, rules + model authoring, recorder, healing
crates/flowproof-adapters	Web (CDP), SAP GUI COM, vision adapters
crates/flowproof-cli	flowproof CLI (thin wrapper over the library)
sdk/python	The flowproof Python package (bundles the engine + MCP server)

Status

Early, in active development; interfaces may still change between minor versions. The badges above carry the current release: the Rust crates, the Python wheel and the npm package move together, so they are the same number.

Real and tested in CI: the record→replay spine, all six adapters (web, Windows desktop, sap, vision, api, agent), model-grounded authoring, healing with reviewable diffs, suites, the security-control surface (flowproof audit), the MCP server, and, at the agent boundary, the OpenAI-compatible proxy with assert_tool_call, the Anthropic Messages dialect, streaming replay in both dialects, plus the MCP tool boundary over stdio.

Built, but with thinner coverage: agent.url services and the MCP boundary over streamable HTTP. agent-testing status names each gap in a per-capability table rather than leaving "built" to imply "tested".

Two limits to know before you start: an agent flow is one turn, not a conversation, and egress containment is Linux-only: elsewhere a command: agent is reported "not contained" rather than silently trusted.

Choosing between flowproof and an existing browser-automation suite? docs/comparison.md is an honest read on when flowproof fits and when to keep what you have.

Contributing

See CONTRIBUTING.md, and internal/design.md for the design notes behind the engine. Licensed under Apache-2.0.