AgentController MCPサーバー

AgentController

Native app QA automation for AI agents, exposed over MCP (Model Context Protocol).

AgentController lets Claude Code—or any MCP client—drive and test native applications on macOS, Windows, Linux, and iOS (simulators and physical iPhones). It exposes a portable automation vocabulary for clicking, typing, screenshots, menus, accessibility inspection, assertions, and replayable flows, backed by native platform APIs rather than a browser-only abstraction.

Its defining feature is background-first automation. On macOS, every tool is background-safe by default. On Windows and Linux, accessibility actions run without moving the pointer, while the five raw-input gestures require explicit foreground:true authorization and restore the prior focus and cursor afterward.

Platform support

Platform	Native stack	Transport	Distribution
macOS 14+	Swift, AXUIElement, CGEvent, ScreenCaptureKit	Local HTTP through a resilient stdio bridge	Signed .app and .dmg
Windows 10/11	C#/.NET, UI Automation, Win32 SendInput	MCP stdio directly	Self-contained .exe
Linux	Python 3.11+, AT-SPI, ImageMagick import / grim	MCP stdio directly	python3 -m agentcontroller_linux
iOS (simulator + iPhone)	TypeScript/Node, idb, native AX scanner, WebDriverAgent	MCP stdio directly (runs on the Mac)	node dist/cli.js

The three desktop backends register the same 52-tool contract. Windows and Linux currently have 49 native implementations and return explicit unsupported errors for reset_app_state, start_recording, and stop_recording; see the Windows guide and Linux guide for platform-specific behavior. The iOS backend registers its own 36-tool surface shaped for phones (gestures, hardware buttons, device lifecycle) rather than force-fitting the desktop vocabulary — see the iOS guide. Snapshot/click/type auto-route to CDP, bpy, idb, or native accessibility from the target identity; you do not pick Playwright vs AX.

---

Table of contents

• Highlights
• Platform support
• How it works
• Requirements
• Installation
• Setup
• Migrating from v1
• Quick start
• Command line
• Fully-background QA
• Tool catalog
• Security
• Development
• Troubleshooting
• Contributing
• Credits and prior art
• License

Highlights

• 52 desktop MCP tools covering app control, AX inspection, input, assertions, screenshots, video recording, menus, clipboard, windows, replayable flows, and a dynamic backend router (inspect_capabilities, run_app_code) — see the full Tool Reference — plus a 36-tool iOS backend for simulators and physical iPhones (iOS guide).
• One surface, auto-routed backends — pass a bundle ID, PID, URL, or simulator UDID. The server handshakes and picks CDP (web), bpy (Blender), idb/WDA (iOS sim), or native AX. Agents do not choose Playwright vs AX.
• Background by default — apps launch without activating, input is delivered per-process (CGEvent.postToPid) or via pure AX actions, menus are resolved by reading the AX tree, and screenshots read the window's own backing store (works even fully covered or hidden).
• Real assertions — assert_visible / assert_not_visible / assert_value poll until satisfied and return MCP isError on failure, so an agent's control loop gets an unambiguous PASS/FAIL instead of parsing prose.
• Stable element handles — snapshot returns a compact [{id, role, label, enabled, frame}] list; interaction tools accept elementId for O(1) reuse without re-searching.
• Token-lean by design — JPEG screenshots with a size cap by default, compact snapshots instead of full trees, text extraction instead of OCR.
• Replayable flows — record a sequence once (save_flow), re-run it as a regression test (run_saved_flow).
• Hardened transport — loopback-only listener, per-launch 256-bit bearer token, DNS-rebinding defenses. See Security.

How it works

Claude Code ──stdio──▶ agentcontroller-mcp-bridge.sh ──HTTP POST──▶ AgentController.app
 (MCP client)           (~/.agentcontroller/)           localhost     (menu-bar app)
                                                                 │
                                                  Accessibility API + CGEvent
                                                  ScreenCaptureKit
                                                                 │
                                                                 ▼
                                                          App under test

AgentController runs as a menu-bar app (LSUIElement, no Dock icon) hosting a bespoke HTTP + JSON-RPC 2.0 server on a random loopback port. A resilient stdio bridge script translates MCP's stdio transport to HTTP, re-resolving the port and auth token automatically — so rebuilding or restarting AgentController never kills an active Claude Code session.

Module	Responsibility
Sources/App	Menu-bar app, permissions UX, server lifecycle
Sources/MCPServer	HTTP listener, JSON-RPC 2.0, auth
Sources/MCPTools	The 52 tool definitions, handlers, and backend router
Sources/AccessibilityEngine	AX tree walking/search, input synthesis, window/app management
Sources/ScreenCapture	ScreenCaptureKit screenshots, video recording, content caching
Linux/	AT-SPI MCP stdio server (python3 -m agentcontroller_linux)

Requirements

macOS: macOS 14 Sonoma or later on Apple Silicon, plus Xcode 16 / Swift 5.10+ and a stable code-signing identity when building from source. Recording requires the macOS 15 SDK to compile and is runtime-gated to macOS 15+.

Windows: Windows 10 or 11 and the .NET 9 SDK for source builds. The default publish command produces a self-contained x64 executable, so target machines do not need a separate .NET installation.

Linux: Python 3.11 or newer. Runtime is the standard library only. Optional system packages: python3-gi and gir1.2-atspi-2.0 for AT-SPI, ImageMagick (import/convert) on X11, grim on Wayland.

Installation

Clone once for any platform:

git clone https://github.com/Kasempiternal/agentcontroller.git
cd agentcontroller

macOS

# Full release: Developer ID sign + notarize + staple + DMG + install
./build.sh

# Signed but not notarized (day-to-day iteration; Gatekeeper warns once)
./build.sh --skip-notarize

# Ad-hoc dev mode (fastest, but TCC grants do NOT persist across rebuilds)
./build.sh --dev

build.sh builds, signs, installs to /Applications/AgentController.app, installs the CLI, the iOS MCP server and the bridges under ~/.agentcontroller/, sets up Claude Code (see below; skip with AGENTCONTROLLER_SKIP_CLAUDE=1), and relaunches the app in the background. A DMG install gets the same pieces: the app carries them and installs them itself at launch.

Windows

.\Windows\build.ps1
.\Windows\smoke.ps1 -Server .\Windows\publish\win-x64\agentcontroller-windows.exe

Register Windows\publish\win-x64\agentcontroller-windows.exe directly as an MCP stdio server. Full configuration and the elevated interactive integration test are documented in Windows/README.md.

Linux

pip install -e Linux/
./Linux/smoke.sh

Register with an MCP client using command plus args (Cursor CLI / .mcp.json):

{
  "mcpServers": {
    "agentcontroller-linux": {
      "command": "python3",
      "args": ["-m", "agentcontroller_linux"]
    }
  }
}

After the editable install, the agentcontroller-linux console script is equivalent. Without pip, set PYTHONPATH=Linux/src. AT-SPI, capture helpers, and compositor limits are documented in Linux/README.md.

The default signing identity is set at the top of build.sh; override per-invocation with:

SIGN_ID='Apple Development: Your Name (TEAMID1234)' ./build.sh --skip-notarize

For notarized release builds, store notarization credentials once (create an app-specific password at appleid.apple.com first):

xcrun notarytool store-credentials agentcontroller-notary \
  --apple-id <your-apple-id> --team-id <your-team-id> --password <app-specific-password>

Why signing matters

macOS TCC keys Accessibility and Screen Recording grants to the signing identity's Team ID. Signing with any stable certificate (Developer ID or Apple Development, same team) means you grant the permissions once and they survive every rebuild. Ad-hoc signing (--dev) keys grants to the binary hash, which changes on every compile.

Setup

1. Grant permissions (one time). On first launch AgentController prompts for the two permissions it needs — grant them to AgentController.app (not your terminal):

• System Settings → Privacy & Security → Accessibility → AgentController
• System Settings → Privacy & Security → Screen Recording → AgentController

2. Set up Claude Code — one command.

~/.agentcontroller/bin/agentcontroller install-claude          # install or refresh
~/.agentcontroller/bin/agentcontroller install-claude --check  # verify, change nothing

It registers both MCP servers (agentcontroller on the compiled bridge, agentcontroller-ios on the bundled iOS server) at user scope, and installs the autonomy pack:

• a managed block in ~/.claude/CLAUDE.md that makes using AgentController a MUST while it is connected: Claude gets keys, fills forms, finishes CLI/OAuth logins and checks dashboards and devices itself, in the background with your saved logins, and asks you only for 2FA, captchas, payments, missing secrets or irreversible actions;
• the agentcontroller skill (~/.claude/skills/agentcontroller) with the playbooks;
• a PreToolUse hook that refuses focus-stealing shell commands (open without -g, osascript activate/keystroke, open -a Simulator) and points at the background tool instead — append # agentcontroller:allow-focus when you asked to see something in front.

Every file it changes is backed up to ~/.agentcontroller/backups/; --uninstall removes the rule, hook and skill. The servers' own MCP instructions carry the same mandate, so any MCP client is told the same thing.

To register by hand instead, user-scoped:

claude mcp add --scope user agentcontroller -- ~/.agentcontroller/bin/agentcontroller mcp
claude mcp add --scope user agentcontroller-ios -- node ~/.agentcontroller/ios-server/dist/cli.js

…or per-project, by adding to the project's .mcp.json:

{
  "mcpServers": {
    "agentcontroller": {
      "command": "/Users/<you>/.agentcontroller/bin/agentcontroller",
      "args": ["mcp"]
    }
  }
}

3. Verify. Restart Claude Code; the tools appear as mcp__agentcontroller__*. Ask Claude to run check_permissions — it should report "allGranted": true. (Auto-start on login: System Settings → General → Login Items → add AgentController.)

Migrating from v1

Version 2.0 changes the product name, executable names, bundle identifier, MCP server names, and runtime directories. Existing installations are intentionally not deleted automatically.

• On macOS, remove the old MCP registration, install AgentController.app, grant Accessibility and Screen Recording again for the new izotz.agentcontroller bundle identifier, then register agentcontroller using ~/.agentcontroller/agentcontroller-mcp-bridge.sh.
• On Windows, update the MCP command to agentcontroller-windows.exe; saved flow files now live under %LOCALAPPDATA%\AgentController\flows.
• On Linux, register python3 -m agentcontroller_linux (or the agentcontroller-linux console script). Saved flows live under $XDG_DATA_HOME/agentcontroller/flows.
• After confirming v2 works, the previous app and its legacy runtime directory can be removed manually. They are not read or overwritten by AgentController.

Quick start

A typical QA exchange with Claude Code:

"Launch MyApp and verify the onboarding flow works, while I keep working."

Under the hood the agent composes tools like this:

// 1. Start the app WITHOUT stealing focus
{ "tool": "launch_app", "args": { "bundleId": "com.example.MyApp" } }

// 2. (optional) Make it completely invisible for the whole run
{ "tool": "hide_app", "args": { "app": "com.example.MyApp" } }

// 3. See what's on screen — compact list with stable ids e1, e2, …
{ "tool": "snapshot", "args": { "app": "com.example.MyApp" } }

// 4. Interact by handle or selector (AX press / PID-targeted events)
{ "tool": "click",     "args": { "app": "com.example.MyApp", "elementId": "e7" } }
{ "tool": "type_text", "args": { "app": "com.example.MyApp", "role": "AXTextField", "text": "hello" } }

// 5. Assert — isError on failure makes PASS/FAIL unambiguous
{ "tool": "assert_visible", "args": { "app": "com.example.MyApp", "labelContains": "Welcome" } }

// 6. Visual evidence (works while hidden/covered)
{ "tool": "screenshot_window", "args": { "app": "com.example.MyApp" } }

// 7. Save the sequence as a regression test
{ "tool": "save_flow", "args": { "name": "onboarding-smoke", "steps": [ /* … */ ] } }

Selector matchers (role, title, titleContains, identifier, value, description, descriptionContains, labelContains, index) are accepted consistently across interaction, assertion, and inspection tools. Tip: use labelContains when you can see text on screen but don't know which AX attribute carries it — SwiftUI varies.

Command line

Everything above assumes an MCP client. There is also a plain CLI, so a shell script, a Makefile, or a person poking at an app can drive the same tools with nothing else installed. build.sh puts it at ~/.agentcontroller/bin/agentcontroller:

export PATH="$HOME/.agentcontroller/bin:$PATH"

agentcontroller status                      # app running? permissions granted?
agentcontroller tools menu                  # the tool list, filtered
agentcontroller describe navigate_menu      # one tool's parameters

agentcontroller snapshot app=com.apple.TextEdit
agentcontroller click app=com.apple.TextEdit role=AXButton title=Save
agentcontroller navigate_menu app=com.apple.TextEdit menuPath:='["Format","Make Plain Text"]'
agentcontroller screenshot_window app=com.apple.Safari -o shot.jpg

key=value is typed from the tool's own schema, so app=1234 stays the string a PID needs instead of becoming a number that resolves to nothing. key:=<json> passes literal JSON for arrays and objects.

It is a client of the running app, not a second implementation: same loopback endpoint, same single Accessibility and Screen Recording grant, same Focus Guard. A CLI that drove the accessibility APIs itself would need its own permission grants and would sit outside Focus Guard, which is the one thing this project promises cannot happen.

Exit codes make it scriptable without parsing prose — 0 the tool did what was asked, 1 it ran and reported failure, 2 a usage or transport problem:

agentcontroller assert_visible app=com.example.App labelContains=Welcome \
  && echo PASS || echo FAIL

Fully-background QA

Every tool is background-safe by default — a full test run happens while the user keeps working in another app, with their focus, cursor, and key window untouched:

• launch_app / open_url start apps without activating them (foreground: true to opt out)
• Input goes through CGEvent.postToPid (per-process queue) or pure AX actions — never the global HID stream, never a cursor warp
• navigate_menu resolves the menu tree by reading it; no menu ever opens on screen
• Screenshots and video read the window's own backing store: the tested window can be fully covered by the user's windows — capture stays correct
• hide_app makes the tested app completely invisible; interactions and even screenshots keep working (ScreenCaptureKit renders hidden windows fresh — verified)

Verified limitations, also stated in the tool descriptions so agents self-correct:

Limitation	Workaround
macOS hides windows on an inactive Space from the accessibility layer entirely — with one app fullscreen, every other app reports zero AX windows	list_windows recovers them from the focused-window attribute and the window server and labels each entry's source; they can be screenshot by windowTitle, but only source: "accessibility" entries can be moved, resized or addressed by windowIndex. Bring the Space forward for full access.
An app that has never been frontmost exposes no focused window either, so snapshot can only reach its menu bar	snapshot reports root: "application" with a warning rather than passing menu items off as screen content
Some apps (Safari most reliably) release their window's backing surface while it is off-screen, so it cannot be captured	screenshot_window names the cause instead of surfacing the raw ScreenCaptureKit error; use screenshot_screen, unhide_app, or read the UI with snapshot / read_all_text
Minimized windows cannot be captured	restore_window first (makes the window visible again)
Clipboard / responder-chain commands (Cmd+C/V, Copy/Paste menu items) no-op without an active app	navigate_menu now detects this — such items read as disabled and return an error rather than a false success. For the send_shortcut equivalents, verify with read_text / assert_value, or activate_app briefly for paste flows
PID-targeted drags can desync (apps that poll the real pointer)	Set foreground: true for that gesture

The single intentionally focus-changing tool is activate_app; everything else only escalates behind an explicit foreground: true. And since v1.4.0, Focus Guard (default on, toggle in the menu bar) turns that convention into a hard guarantee: while enabled, the dispatcher refuses activate_app and every foreground:true call with an error that points the agent back to the background-safe path — a misbehaving agent cannot steal your focus.

Tool catalog

52 tools — full parameter documentation in docs/TOOLS.md (generated from the live server's tools/list).

Category	Tools
App control	list_apps · launch_app · quit_app · activate_app · hide_app · unhide_app · get_frontmost_app · open_url · reset_app_state
Snapshot & handles	snapshot / describe_screen
Inspection	get_element_tree · find_elements · get_element_attributes · get_focused_element · wait_for_element
Assertions	assert_visible · assert_not_visible · assert_value
Text	read_text · read_all_text
Input	click · double_click · right_click · type_text · send_shortcut · scroll · scroll_until_visible · swipe · drag_drop
Windows	list_windows · get_window_bounds · set_window_bounds · minimize_window · restore_window
Screenshots & video	screenshot_window · screenshot_element · screenshot_screen · start_recording · stop_recording
Menus	navigate_menu · get_menu_structure
Clipboard	get_clipboard · set_clipboard
Flows	run_steps · save_flow · list_flows · run_saved_flow
System	check_permissions
Routing	inspect_capabilities · run_app_code

Security

The HTTP server is pinned to IPv4 loopback (never 0.0.0.0), so it is unreachable from the network. Every request must carry Authorization: Bearer <token>; the server generates a fresh 256-bit token per launch, writes it to ~/.agentcontroller/mcp-token (mode 0600, alongside mcp-port), and the bridge replays it on each call with automatic re-resolution after restarts. Bearer comparison is constant-time. Requests carrying an Origin header or a non-loopback Host are rejected (403) to defeat DNS rebinding from a browser. Request bodies are size-capped (16 MiB) and reads are deadline-bounded (slowloris protection).

Destructive operations are opt-in: reset_app_state only deletes an app's sandbox container behind an explicit wipeData: true flag, and the clipboard tools warn that they touch system-wide state.

The Windows and Linux backends speak MCP over stdio and open no listening socket, so the loopback and token considerations above do not apply to them. The iOS backend is stdio too; its only listening socket is a loopback-pinned live-screen viewer, and WebDriverAgent's own unauthenticated port on the phone is called out honestly in the iOS guide. Automating an app means reading its contents, and screenshots capture whatever is on screen — see SECURITY.md for the full threat model, what is explicitly out of scope, and how to report a vulnerability privately.