gui-agent MCP server

The Philosophy: Why gui-agent?

Autonomous AI agents interacting with modern graphical user interfaces are frequently burdened by fragmented architectures, high latency, and fragile computer vision loops. Traditional automation setups force models to perceive operating systems exclusively through repetitive raster screenshots, incurring prohibitive round-trip times (RTT) of 2 to 5 seconds per motor action. This reductionist approach causes severe spatial misalignments under fractional scaling, loses transient UI components like disappearing toasts or click-away menus, and quickly exhausts context windows with redundant image payloads.

gui-agent redefines desktop interaction by aligning agent decisions with the actual ontology of modern operating systems: structured processes in RAM, accessibility trees (AT-SPI2 / D-Bus), window compositors (X11 / Wayland), and kernel event subsystems (uinput, evdev). Built on two foundational pillars—Progressive Escalation (L3/L2/L1) and the CodeAct Local REPL Engine—the server enables models to actuate targets in RAM within 50 ms via native Rust mediation (gui-agent-atspi), fallback to local RapidOCR or calibrated Cartesian grids when needed, and execute entire multi-step action sequences locally in host memory with sub-5ms latency.

Operating through a single, resilient standard input/output (stdio) FastMCP connection, gui-agent functions with a lean baseline memory footprint below 50 MB RAM, fully preserving dual-core host responsiveness without external cloud vision dependencies or persistent background daemons. Platform implementations reside in dedicated, sealed root directories (linux/, windows/, macos/) with dynamic XDG Base Directory path resolution, zero hardcoded user paths, and full continuous screen video recording preservation (gui_start_video_recording, gui_stop_video_recording) for deterministic auditing.

---

Core Capabilities

The table below outlines the Target Architecture v1.0 (15 unified primitives) specifying the target contract across the modular roadmap, alongside the active MCP tools currently exposed by the Linux runtime under their operational gui_* and AT-SPI2 namespaces.

Tool Name	Domain	Description	Status
execute_action_batch		Executes local Python/Bash scripts with preloaded mcp_core (Open Interpreter model), running multi-action batches without RTT (Phase 2).
process_run		Spawns interactive shell commands via PTY to inject credentials and bypass Polkit/sudo security dialogs (Phase 3).
process_list		Inspects the /proc filesystem in read-only mode to probe active processes without altering desktop state (Phase 3).
activate_window		Switches focus at the display compositor level using unique Window IDs (active runtime alias: gui_window_focus).
get_app_state		Inspects the AT-SPI2 accessibility tree via native Rust mediation (gui-agent-atspi) directly in RAM.
perform_action		Triggers semantic actions directly in RAM via D-Bus IPC in sub-50ms without physical pointer movements.
set_value		Mutates text or numerical values directly into component memory variables without physical keystrokes.
find_text		Discovers on-screen text coordinates via local OCR without sending image payloads (active runtime alias: gui_find_text).
screen_capture		Captures raw framebuffer with calibrated Cartesian grid overlays for opaque surfaces (active runtime alias: gui_take_screenshot).
mouse_click_at		Injects physical mouse click events (left, right, middle, double) at target (x, y) coordinates (active runtime alias: gui_mouse_click).
mouse_drag_smooth		Dispatches continuous kinematic mouse trajectory interpolation to overcome UI drag thresholds (active runtime alias: gui_mouse_drag).
mouse_scroll		Executes hardware wheel scrolls to materialize virtualized DOM elements (active runtime alias: gui_mouse_scroll).
key_tap		Injects standard keystrokes, navigation hotkeys, and chord combinations (active runtime aliases: gui_keyboard_press, gui_keyboard_type).
gui_start_video_recording		Starts background low-overhead screen video recording via FFmpeg (x11grab / H.264 ultrafast).
gui_stop_video_recording		Cleanly halts the active FFmpeg recording, flushes the MP4 container, and prevents descriptor leaks.

---

How It Works

gui-agent operates as a closed-loop Computer Use bridge between frontier LLM reasoning engines and the host operating system, combining semantic RAM actuation, local script execution, and visual-motor feedback.

Technical Execution Pipeline & Foundational Pillars

1. Pillar I : Progressive Escalation & Layered Actuation: Rather than enforcing a single interaction mode, the architecture prioritizes cognitive and execution efficiency across layered stages: Level L3 accesses the OS accessibility tree (AT-SPI2 / D-Bus via the compiled Rust mediator gui-agent-atspi) directly in RAM for deterministic sub-50ms actuation with zero image tokens; Level L2 runs decoupled local OCR (RapidOCR/Tesseract) on typography without model inference overhead; Level L1 operates as the ultimate hardware safety net using calibrated Cartesian grid screenshots with native input dispatchers (with direct kernel uinput/evdev drivers scheduled on the roadmap); and an interactive PTY shell layer provides seamless handling of privileged commands.
2. Pillar II : High-Efficiency Execution Architecture: Paving the way to eliminate multi-turn network round-trip time (RTT) latency, the project architecture designs an isolated local execution environment (execute_action_batch via core/repl.py). Models will project multi-step inspection and action logic directly as Python code executed in host memory via the unified mcp_core SDK. Complex condition checking, kinematic drag calculations, and dynamic polling resolve in a single cognitive round-trip with sub-5ms execution speed and less than 15 MB RAM consumption (slated for Phase 2 roadmap).
3. Sub-second Screen Ingestion & Cartesian Grid Overlay: When an agent requests visual state via gui_take_screenshot, the server captures the raw framebuffer through MSS, with automatic fallback to KDE Spectacle or Scrot on XWayland surfaces. The engine overlays a millimeter Cartesian coordinate grid with adaptive contrast-buffered labels at configurable intervals (e.g., 100px), allowing models to infer target coordinates with mathematical certainty.
4. Dual Coordinate Normalization Engine: The server accepts coordinates in either absolute physical pixels (x, y) or normalized ratios [0, 1000] across any display geometry or multi-monitor setup. An automatic converter handles boundary clamping, DPI scaling, and coordinate translation transparently.
5. Native OS Input & Window Dispatcher: Keystrokes, hotkeys, mouse clicks, and drag operations are routed through low-latency native drivers (xdotool and python-xlib under Linux, Win32 API under Windows). Humanized delays and micro-jitter emulate natural user interaction. Window management commands (wmctrl / xprop) inspect and manipulate window states without window manager locks.
6. Local Vision, OCR & Playwright Automation: Template matching (cv2.matchTemplate) enables robust icon detection even under theme variations. Text discovery combines Tesseract OCR with RapidOCR ONNX fallback. Web automation leverages Playwright to inspect ARIA trees and manipulate DOM nodes directly without visual ambiguity.

Multi-Platform Root Architecture & Dynamic Path Resolution

The codebase organizes platform implementations into dedicated root directories with zero hardcoded filesystem paths:

• linux/ : Complete Linux implementation featuring the core server (server.py), native Rust AT-SPI2 / D-Bus mediator (linux/crates/atspi_mediator compiled to gui-agent-atspi), automated install/uninstall scripts (install.sh, uninstall.sh), dedicated tests and examples, and dynamic XDG Base Directory path resolution (paths.py).
• windows/ : Dedicated Windows directory (install.ps1, uninstall.ps1, native UI Automation backend in active development).
• macos/ : Dedicated macOS directory reserved for upcoming NSAccessibility and Quartz Event Taps implementations. All runtime paths—including screenshots ($XDG_CACHE_HOME/gui-agent/screenshots or GUI_AGENT_SCREENSHOTS_DIR), persistent continuous video captures ($XDG_CACHE_HOME/gui-agent/videos or GUI_AGENT_VIDEOS_DIR), and data storage ($XDG_DATA_HOME/gui-agent)—are resolved dynamically at runtime.

---

Installation

For detailed OS-specific instructions, troubleshooting matrices, and offline setups, see the Detailed Installation Guide (INSTALL.md).

1. Automated Installation (Recommended)

Linux (Bash)

Run the automated installer to check dependencies, install Astral uv, build the native Rust mediator, and register the MCP server:

# Download and execute the automated installer via curl
curl -fsSL https://raw.githubusercontent.com/leandre755/gui_agent/7a49514/linux/install.sh | bash

# Or execute locally from a cloned repository
./linux/install.sh

Microsoft Windows (PowerShell)

Launch PowerShell (standard user or administrator) and execute the automated setup script:

# Download and execute the installation script
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/leandre755/gui_agent/7a49514/windows/install.ps1" -OutFile "install.ps1"
powershell -ExecutionPolicy Bypass -File .\install.ps1

# Or execute locally from a cloned repository
.\windows\install.ps1 -Local

2. Isolated Deployment via uv tool

Install gui-agent directly into an isolated environment with global CLI entrypoints:

# Install from PyPI
uv tool install gui-agent

# Or install from GitHub repository
uv tool install "git+https://github.com/leandre755/gui_agent.git"

# Upgrade to latest release
uv tool upgrade gui-agent

3. Linux System Prerequisites

Under Linux, install the native window management, OCR, multimedia, AT-SPI accessibility, and Rust build libraries:

# Debian / Ubuntu / Linux Mint
sudo apt-get update && sudo apt-get install -y \
  xdotool wmctrl spectacle ffmpeg xclip tesseract-ocr libgl1 libatspi-dev cargo rustc

# Fedora / RHEL
sudo dnf install -y \
  xdotool wmctrl spectacle ffmpeg xclip tesseract libglvnd-glx at-spi2-core-devel cargo rust

# Arch Linux / Manjaro
sudo pacman -S --needed \
  xdotool wmctrl spectacle ffmpeg xclip tesseract at-spi2-core cargo rust

---

MCP Client Configuration

1. Claude Code CLI

Register the server with Claude Code CLI in a single command:

# If installed via uv tool
claude mcp add gui-agent -- gui-agent

# Direct on-the-fly execution via uvx (zero pre-installation)
claude mcp add gui-agent -- uvx --from gui-agent gui-agent

2. Antigravity CLI

Add the server definition to your Antigravity global MCP configuration:

• Linux / macOS: ~/.gemini/config/mcp_config.json
• Windows: %USERPROFILE%\.gemini\config\mcp_config.json

{
  "mcpServers": {
    "gui-agent": {
      "command": "gui-agent",
      "args": [],
      "env": {
        "DISPLAY": ":0"
      }
    }
  }
}

(Note: The alias binary mcp-gui-server can also be used as the command target).

3. Cursor & VSCode

Add the following entry to your Cursor mcp.json (~/.cursor/mcp.json or .vscode/mcp.json):

{
  "mcpServers": {
    "gui-agent": {
      "command": "uvx",
      "args": ["--from", "gui-agent", "gui-agent"]
    }
  }
}

---

Toolset & CLI Reference

execute_action_batch

Executes multi-step Python or Bash action blocks directly in host memory with preloaded mcp_core SDK (Open Interpreter paradigm), eliminating network RTT.

• Parameters:
  • language (str, default "python"): Execution runtime environment ("python" or "bash").
  • code (str): Multi-step script containing conditional logic, loops, and rapid polling routines.
  • timeout (float, default 30.0): Execution deadline in seconds before terminating the runner process.
• Returns: dict containing execution status, captured stdout, stderr, and execution elapsed_seconds.

process_run

Executes shell commands in a pseudo-terminal (PTY) session, allowing credential injection to bypass security modals.

• Parameters:
  • command (str | list[str]): Command line string or argument list to execute.
  • background (bool, default False): Spawns detached in background (True) or waits synchronously (False).
  • sudo_password (str | None, default None): Password injected into PTY stdin for Polkit/sudo escalation.
• Returns: dict containing execution status, exit code returncode, stdout, and stderr.

process_list

Inspects the /proc filesystem in read-only mode to probe active desktop processes and hierarchy without mutation.

• Parameters: None.
• Returns: list[dict] containing active system process entries with pid, name, and status metadata.

activate_window

Switches desktop focus directly at the display compositor level using unique Window IDs, avoiding PID collision (active alias: gui_window_focus).

• Parameters:
  • window_id (str | int): Target compositor Window ID to raise and focus.
• Returns: dict containing operation status and confirmed active window identifier.

get_app_state

Inspects the accessibility tree via native Rust mediation (gui-agent-atspi) directly in RAM with zero image tokens.

• Parameters:
  • include_screenshot (bool, default False): Attaches an optional visual framebuffer capture.
• Returns: dict containing structured tree nodes, numeric element_index, bounds, states, and snapshot_id.

perform_action

Invokes semantic actions directly in target application memory via D-Bus IPC in sub-50ms without pointer motion.

• Parameters:
  • element_id (str | int): Node identifier or cache index from the active snapshot_id.
  • action (str, default "activate"): Semantic action name ("activate", "click", "press").
• Returns: dict containing execution status, target identifier, and action verification response.

set_value

Mutates text or numerical values directly into component memory variables without emitting physical keystrokes.

• Parameters:
  • element_id (str | int): Target editable field or widget identifier.
  • value (str): Text or numerical value to assign directly into component memory.
• Returns: dict containing mutation status, target identifier, and assigned value confirmation.

find_text

Discovers on-screen text coordinates via decoupled local OCR engines (RapidOCR / Tesseract; active alias: gui_find_text).

• Parameters:
  • text (str): Target text string to identify across the desktop screen.
  • confidence (float, default 0.85): Minimum detection confidence score (0.0 to 1.0).
• Returns: dict containing detected text centroid {"x": int, "y": int}, bounding box, and match confidence.

screen_capture

Captures the raw display framebuffer with an optional calibrated Cartesian coordinate grid overlay (active alias: gui_take_screenshot).

• Parameters:
  • show_grid (bool, default True): Overlays a Cartesian coordinate grid with adaptive contrast labels.
  • grid_step (int, default 100): Pixel distance between coordinate grid lines (minimum 20px).
  • output_path (str | None, default None): Output destination path with atomic reservation protection.
• Returns: dict containing resolved screenshot_path, image dimensions, format, and grid status.

mouse_click_at

Injects physical mouse click events directly via low-level input subsystems at exact target coordinates (active alias: gui_mouse_click).

• Parameters:
  • x (int | float): Absolute X pixel coordinate.
  • y (int | float): Absolute Y pixel coordinate.
  • button (str, default "left"): Mouse button identifier ("left", "right", "middle").
  • double (bool, default False): Dispatches a consecutive double-click sequence when enabled.
• Returns: dict confirming click execution status, target coordinates, and dispatched button.