computer-use-linux MCPサーバー

Running open models for your company? Tiyuvta helps with model choice, deployment and optimization, and fine-tuning on your hardware or cloud account.

computer-use-linux gives AI agents real, structured control of a local Linux desktop: semantic AT-SPI accessibility trees, bounded screenshots, compositor-aware window targeting, Wayland/X11 input synthesis, and single-round-trip post-action verification across GNOME, KDE Plasma/KWin, Hyprland, niri, COSMIC, i3, Sway, labwc and other wlroots compositors, and X11/EWMH, plus a click-through on-screen activity indicator on GNOME and on Wayland compositors with wlr-layer-shell.

Quick start

Pick the setup that matches your agent or shell workflow:

Host / workflow	Install command
Claude Code (plugin + skill)	claude plugin marketplace add agent-sh/computer-use-linux && claude plugin install computer-use-linux@computer-use-linux
Codex CLI (plugin + skill)	codex plugin marketplace add agent-sh/computer-use-linux && codex plugin add computer-use-linux@computer-use-linux
Pi Coding Agent (native lazy-loaded tools)	pi install npm:@agent-sh/computer-use-linux
Hermes Agent (CLI + skill + MCP server)	npm install -g @agent-sh/computer-use-linux && hermes skills tap add agent-sh/computer-use-linux && hermes skills install agent-sh/computer-use-linux/computer-use-linux && hermes mcp add computer-use-linux --command computer-use-linux --args mcp
CLI + any stdio MCP client (npm)	npm install -g @agent-sh/computer-use-linux
CLI + system setup from source	git clone https://github.com/agent-sh/computer-use-linux && cd computer-use-linux && ./install.sh
Rust crate (crates.io)	cargo install computer-use-linux

Verify desktop readiness from the shell (after installing the CLI on PATH via npm install -g @agent-sh/computer-use-linux, ./install.sh, or cargo install computer-use-linux):

computer-use-linux doctor | jq .readiness
computer-use-linux setup                      # enable AT-SPI accessibility bridge if needed
computer-use-linux setup-window-targeting     # install GNOME Wayland window-control extension if needed

Plugin-only or Pi-only installs: The Claude Code / Codex plugin launcher and pi install manage private cached binaries and do not add computer-use-linux to PATH. Either run the MCP doctor / setup_accessibility / setup_window_targeting tools (or computer_use_linux_doctor in Pi) directly from your agent, or also install the CLI above when you want shell subcommands on PATH.

The Rust crate is published as computer-use-linux and the npm package as @agent-sh/computer-use-linux. Prebuilt x86_64 and aarch64 Linux binaries ship with the latest release. They need glibc 2.28 or newer: RHEL 8 and its rebuilds, Debian 10, Ubuntu 20.04, and anything later. On an older system, build from source with cargo install computer-use-linux.

Need an isolated desktop where the agent never touches your real mouse, keyboard, or focused window? Use the sibling project agent-workspace-linux alongside this one.

What this is

computer-use-linux is a Rust MCP server and CLI for Linux desktop control. It ships three binaries: the main computer-use-linux server/CLI, the computer-use-linux-cosmic Wayland toplevel helper for COSMIC, and the computer-use-linux-indicator click-through activity overlay. Any MCP host (Claude Code, Codex CLI, Codex Desktop's Linux build, Pi Coding Agent, Hermes Agent, Claude Desktop, or your own client) can spawn it to inspect accessibility trees, list and focus windows, take screenshots, click, drag, scroll, type, and invoke semantic accessibility actions.

Most computer-use MCP servers are macOS-only (they lean on AppKit, AXUIElement, CGEvent). The few that target Linux either drive xdotool against an X11 root window or shell out to OCR over screenshots. It differs in five ways:

• Wayland actually works. On Wayland, pointer actions use /dev/uinput or org.freedesktop.portal.RemoteDesktop. ydotool sends only untargeted wheel events there, and a coordinate action with neither route is refused. Literal text prefers KDE clipboard integration on Plasma, wtype on compatible Wayland compositors, and portal keysyms on other Wayland sessions, including GNOME with ydotool installed. Portal text startup or conversion errors fail the call without replaying text through ydotool. Raw ydotool text accepts printable ASCII, tab and newline only, and its physical US key positions can produce different text under the active layout. Screenshots use the GNOME Shell DBus screenshot method when present, org.freedesktop.portal.Screenshot otherwise, then on a native X11 session a root-window GetImage (device pixels, no toolkit scaling), and finally spawn gnome-screenshot for background/systemd contexts where the DBus paths are denied.
• Window targeting is compositor-aware. The window registry tries GNOME Shell extension, GNOME Shell Introspect, COSMIC Wayland helper, KWin DBus scripting, Hyprland hyprctl, the niri IPC socket, i3 IPC, labwc's labwc.sock, wlr-foreign-toplevel-management, and generic X11/EWMH in order, then reports exactly which backend won or why each backend failed. Terminal windows are enriched with their controlling TTY and foreground process so agents can target "the terminal running pytest" directly. Every window_id fits JavaScript's exact integer range (at most 2^53-1), so JSON clients can pass it back unchanged; a larger value is rejected with a prompt to re-run list_windows.
• Semantic selectors first, pixel coordinates when needed. Tools like click, perform_action, and set_value accept role / name / text / states selectors or element_index values backed by AT-SPI, and every mutating tool can return a post-action screenshot and updated accessibility tree in the same round trip. Pixel coordinates remain available for rendering-only surfaces (canvas, games, X clients without ATK).
• Visible to humans, invisible to screenshots. While an agent acts, computer-use-linux-indicator renders a click-through overlay with a per-agent cursor glide, click ripple, keycaps/typing banner, screen-edge glow, and status pill, and hides automatically under a shared capture lock before any screenshot is read.
• One JSON readiness report. computer-use-linux doctor returns a structured document covering platform, portals, AT-SPI, windowing, input, and a readiness summary with explicit blockers and a recommended next step. MCP hosts can render or surface that to the user without parsing prose.

The crate was extracted from codex-desktop-linux (the Linux distribution of Codex Desktop), which bundles this backend for its native Computer Use feature. This standalone repo is the upstream.

MCP tools & features

Category	Tool	What it does
Diagnostics & setup	doctor	Single-shot JSON readiness report (platform, portals, accessibility, windowing, input, readiness summary, and capability map).
	setup_accessibility	Enables and verifies GNOME's org.gnome.desktop.interface toolkit-accessibility setting so toolkit apps expose AT-SPI trees.
	setup_window_targeting	Installs and enables the bundled GNOME Shell extension. GNOME Wayland needs it for exact window activation and move/resize; every GNOME session needs it for the on-screen indicator.
Discovery & observation	get_app_state	Combined screenshot + compacted AT-SPI accessibility tree with stable element_index handles. Scope with app_name_or_bundle_identifier or a window target (window_id, pid, app_id, wm_class, title).
	get_tree_page	Page through the latest cached tree by node offset, for trees too large to return inline when the host cannot read the spill file. Pass snapshot_id to refuse a page from a tree a later snapshot replaced.
	screenshot	Bounded PNG or JPEG capture of the full desktop (full_screen: true or no target) or a raised and cropped target window (window_id, pid, app_id, wm_class, title, raise_window).
	list_windows	Compositor windows with window_id, title, app_id, wm_class, PID, focus state, client type (wayland/x11), bounds, and terminal TTY/process metadata.
	focused_window	The window currently holding keyboard focus.
	list_apps	Running desktop applications visible to the AT-SPI registry.
Input	click	Click by element_index, semantic selector (role/name/text/states), or desktop/window-relative coordinates (left/right/middle, click_count).
	drag	Desktop coordinate drag from (start_x, start_y) to (end_x, end_y).
	scroll	Page-based vertical/horizontal scroll on an element, window, or pixel coordinate.
	press_key	Single keys or chords (Ctrl+L, Alt+F4, Return), optionally focusing a target window or terminal first.
	type_text	Literal text entry, optionally targeted at a window or terminal (tty, terminal_pid, terminal_command, terminal_cwd).
Semantic actions	perform_action	Invoke any AT-SPI action exposed by an element (Press, Activate, Toggle, Jump, …); defaults to the primary action.
	set_value	Write directly to a settable accessibility element (text fields, sliders, spinners).
Window management	activate_window	Focus a window by window_id, pid, app_id, wm_class, title, or terminal selectors.
	move_window / resize_window	Reposition or resize a window in desktop coordinates (GNOME Shell extension and X11/EWMH backends).
Opt-in host tools	complete_interaction	Parameter-free desktop completion notification (notify-send); registered only when COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1.
	run_shell	Bounded non-login /bin/sh -c execution; registered only when COMPUTER_USE_LINUX_ENABLE_SHELL=1 (absent by default).

Every tree result carries a snapshot_id: get_app_state, an include_tree observation, and get_tree_page. The server caches one tree, and each new tree replaces it. Pass the snapshot_id with an element_index (or a selector, except on scroll) to click, scroll, perform_action or set_value, and to get_tree_page: if a later tree replaced that snapshot, the call is refused before any window is raised or input is sent, since the index could name another element by then. A full capture-space x/y point does not come from the tree, so it is not checked. A coordinate_space: "atspi" click maps through its element, so it is checked. Without snapshot_id, the element resolves in the latest tree. An element_identifier names the AT-SPI object itself and is not checked against a snapshot. Ids differ between server processes, so an id from before a restart is refused.

click, scroll, perform_action, and set_value take an optional app target (pid, window_id, app_id, wm_class, window_title). With one, an element_index (or a selector, where the tool takes one) from another app's get_app_state snapshot is refused; call get_app_state for the target app first. When the window backend reports no pid for a target window (i3 without xprop, COSMIC, other wlroots compositors), the snapshot must have been taken for that window, or the call is refused. perform_action and set_value do not focus the target window. An element_identifier is refused when its owner on the accessibility bus is a different pid than the target; it is not checked when either pid is unknown.

Post-action observation in one round trip

All mutating tools (click, drag, scroll, press_key, type_text, perform_action, set_value, activate_window, move_window, and resize_window) accept optional post-action observation parameters so an action and its verification complete in a single MCP tool call:

• screenshot_after: true | "screen" | "window" | false: capture after the action is dispatched and the indicator overlay is hidden (true or "screen" for full desktop, "window" cropped to the action's target window or the focused window when untargeted)
• format, quality, scale, max_width, max_height, max_bytes: same screenshot payload controls as screenshot and get_app_state
• screenshot_file_path: save the post-action screenshot to this new file instead of returning it as image content (see Screenshot files)
• include_tree: true: include a fresh compacted AT-SPI accessibility tree and its snapshot_id; it replaces the cached tree, so later element_index values come from it

When post-action screenshot capture succeeds, the image is returned as the first MCP image content block alongside the JSON action summary and screenshot metadata. If the action succeeds (ok: true) and only the post-action screenshot or tree capture fails, ok remains true and the error is reported in screenshot_error / accessibility_error and appended to message.

Screenshot sizing & coordinate scaling

Screenshot payloads are size-bounded by default before they are returned to the MCP host: max 1920 px width/height and 2 MiB image bytes, with hard caps even when callers request more. Agents that need more detail can pass max_width, max_height, max_bytes, scale, format: "jpeg", or quality, preferably with a window target or crop. PNG remains the default; JPEG lets callers trade lossless pixels for a smaller payload before the byte cap forces further resizing. Returned screenshot metadata includes coordinate_width, coordinate_height, scale, format, and quality so callers can convert from a downscaled preview to desktop coordinate pixels.

Screenshot files

screenshot, get_app_state and the screenshot_after capture of mutating tools take screenshot_file_path. With it, the image is saved to that file and no MCP image content is returned, so a capture costs no image tokens until the agent reads the file. The metadata stays the same (width, height, coordinate_width, coordinate_height, scale, format, quality, bytes) plus file_path: in the screenshot caption, and in screenshot.file_path for the other tools.

• The path must be absolute, and its parent directory must exist. Directories are not created.
• The extension must match format: .png for png (the default), .jpg or .jpeg for jpeg.
• The path must end with the file name, so a trailing / or /. is refused.
• File output creates a new file and never replaces one. Any existing entry at the path is refused: a file, a directory or a symlink.
• The image is written to a temp file in the same directory, then moved into place with a step that fails if anything exists at the path (renameat2 with RENAME_NOREPLACE, or link where that is unsupported). A reader never sees a partial file, and an entry created meanwhile is neither followed nor replaced. The file is created with mode 0600.
• A bad path is refused before anything is captured or any input is sent.
• A failed write is never answered with the image inline. screenshot returns an error, get_app_state sets screenshot_error, and a mutating tool sets screenshot_error and keeps the action's ok.
• Over stdio the path is on the machine that runs the server. Over HTTP (mcp --listen) the parameter is refused, so a remote client gets no way to write files on the host.