XuLan ScreenPilot MCP server
Commit Nov 15, 2026Converts Windows screens into structured, numbered element tables that allow AI agents to interact with UI components via IDs instead of coordinates.
Also known as dahuam/screenpilot, screenpilot
- Stars
- 0
- Tools
- —
- Downloads, 30 days
- —
- Latest version
- 1.0.0
- Last commit
- Nov 15, 2026
- License
- Apache-2.0
What is XuLan ScreenPilot MCP?
XuLan ScreenPilot converts Windows screens into structured, numbered element tables that allow AI agents to interact with UI components via IDs instead of coordinates.
Summary generated from the project's README
About XuLan ScreenPilot MCP server
Automating Windows desktop interfaces by providing AI agents with structured element tables instead of raw screenshots is the primary function of XuLan ScreenPilot. It converts visual interfaces into numbered lists of interactive components, allowing an agent to interact via specific IDs using tools like click_element and type_text. This approach prevents the common issue of agents missing targets due to coordinate guessing errors.
The server runs locally on Windows 10 or 11 using only the CPU, requiring no cloud models, GPUs, or API keys. It utilizes three parallel engines—UIA for native apps, RapidOCR for text in games or custom UIs, and OpenCV for visual shapes—to fuse a single, accurate element table. Users can configure the system via a stdio MCP setup for clients like Claude Desktop.
Key features include a compact scan mode that filters non-interactive elements to save prompt tokens and a privacy-first mode that prevents sensitive windows from being saved to a persistent element book. It also includes advanced capabilities like business-level waits with click_safe, which ensures an action is successful only when specific visual evidence appears on the screen.
What you can do with it
- 1Interact with Windows UI via IDsAn AI agent can drive a desktop application by referencing specific element numbers rather than guessing pixel coordinates.
- 2Perform verified UI actionsA developer can instruct an agent to click a button and wait until a specific piece of text or a new window actually appears.
- 3Maintain privacy during automationYou can ensure that sensitive applications like chat apps or login dialogs are not recorded into the persistent element book.
Written from the project's README. Every use case is checked against a quote from it.
Automated checks
Ran Nov 15, 2026Full report →Not verified yet. A server needs a score of 80, no serious finding, and enough data for us to check. How checks work · What to fix for more points
- Source repository linked and reachabledahuam/screenpilot is public and reachablegithub.com/dahuam/screenpilot ↗Pass
- Package names this repository as its sourceNot checked
- Build provenance (Sigstore attestation)Not checked
- Open source license (OSI)Apache-2.0spdx.org/licenses/Apache-2.0.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-11-08github.com/dahuam/screenpilot ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNot checked
- No critical advisory in the package or its dependenciesNot checked
- No hidden instructions to the model0 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNot checked
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubXuLan ScreenPilot
Turn the real Windows screen into a structured element table your AI agent can operate.
XuLan ScreenPilot solves the two classic pain points of GUI automation for AI agents — not seeing the elements and missing the click — by giving the agent a numbered element table instead of a screenshot to guess coordinates from. The agent scans once, reads the table, and drives the screen by element id (click / type), in a closed loop.
Pure local CPU. No cloud model, no GPU, no API key.
Highlights in v1.0.0
- Compact scan mode, on by default — empty non-interactive shells and duplicate labels are filtered before numbering: a typical scan drops ~23% of elements and prompt tokens while keeping every interactive control (
compact=falserestores the raw table;meta.compact_removedreports what was filtered). - Privacy-first element book, on by default — scans of sensitive windows (Notepad, chat apps, login dialogs) are recognized as usual but never written into the persistent element book (
meta.privacy_skip); an explicitbook=trueoverrides. - Hang-proof UIA calls — a target window that stops responding surfaces as a structured error in ~2 s (message-pump probe) instead of blocking 30 s+; UIA calls carry configurable timeouts, and COM failures are classified and counted in
meta.errors.uia_stats. - Unambiguous
expectsemantics — a timed-out business wait no longer collapses intook=false:action_executeddistinguishes "the click was sent but no evidence appeared" from "the action itself failed", so risky buttons are not blindly retried. - Fresh crops on demand —
crop_screen/crop_xyacceptforce_refreshto bypass the 60 s shot cache when verifying a screen right after an action.
How it works
Three engines run in parallel and fuse into one element table (priority UIA > OCR > visual):
| Engine | Source | Covers |
|---|---|---|
uia |
Windows UIA accessibility tree | Native apps, browsers (incl. web a11y tree) — precise and fast |
ocr |
RapidOCR (onnxruntime, CPU) | Games and custom-drawn UI with no control tree; full-screen text fallback |
visual |
OpenCV connected components + shape detectors | Icon candidates; dedicated channels for checkboxes / radio dots / sliders / close buttons |
Fusion dedupes by center-containment or IoU ≥ 0.3, numbers elements top-to-bottom / left-to-right, and produces:
elements.json— the element table cache (queried by id for click/type)annotated_scan.png— annotated screenshot (red = UIA, blue = OCR, green = visual, id printed on each box)
On top of the table:
- Frame-to-frame id stability — rescan keeps the same element on the same id; new elements get new ids (
meta.trackerreports kept/new/lost). - Background capture — grab a covered window's real pixels via
PrintWindow(no need to raise it). - Drifted clicks — physical clicks land with Gaussian jitter and move trails, so automation doesn't look robotic;
precise=Truerestores pixel accuracy. - Stacking awareness — every coordinate click reports which top-level window it actually hit (
hit_window), and can refuse clicks on occluded targets (top_must). - Two-level vision + text channel — crop any element at native resolution to read fine text/icons; a SigLIP vector book lets you find UI elements by name or by image without rescanning.
- Time dimension — burst watch for toasts/transient feedback, show-verify for typing, ESC probe for hangs, and static/dynamic separation (UI base layer vs. moving entities) across frames.
- Business-level waits —
expect {find, mode: appear|disappear}makes a click "successful" only when the business evidence shows up on screen. - Session cleanup — a per-session ledger tracks windows the agent launched;
cleanup_windowscloses them at task end (WM_CLOSE only, six-gate safety chain, no taskkill); a crashed session leaves a residual report that the next session auto-surfaces.
Install
Requirements: Windows 10/11, Python 3.11+, CPU only.
git clone <repo-url> screenpilot && cd screenpilot
pip install uiautomation rapidocr-onnxruntime onnxruntime opencv-python numpy pillow pyperclip pyautogui mcp
python screenpilot.py scan :: full-screen scan, prints the element table
Tested on Python 3.13. The optional element-book vector channel and the benchmark matcher additionally need tokenizers and the SigLIP int8 ONNX weights under SCREENPILOT_MODELS_DIR.
Optional: the semantic matcher and the element-book vector channel use SigLIP int8 ONNX models. Set SCREENPILOT_MODELS_DIR to a local folder holding the model directories (see README.zh-CN.md §两级视觉与文本通道). Datasets for the public benchmark go under SCREENPILOT_DATASETS_DIR.
Quickstart: the agent closed loop
# MCP tools (stdio server) or the CLI expose the same verbs:
scan_screen(window="记事本") # 1. scan -> numbered element table
click_element(element_id=7) # 2. act by id (UIA control-level when possible)
type_text(element_id=3, text="hi") # 3. type into an edit field
click_safe(element_id=12, expect={"find": "设置", "mode": "appear"}) # 4. act only when evidence shows
watch_burst(window="记事本", seconds=3) # 5. catch transient feedback frames
MCP config (e.g. Claude Desktop / ZCode):
{"mcpServers": {"screenpilot": {
"command": "python",
"args": ["<path>/screenpilot/mcp_server.py"],
"env": {"PYTHONIOENCODING": "utf-8"}
}}}
Benchmarks
Public GUI-grounding benchmarks, frozen matcher, official scoring. Pure CPU on a desktop with a GTX 1070 Ti (GPU unused):
| Benchmark | Scope | acc | oracle* | Latency |
|---|---|---|---|---|
| ScreenSpot-v2 | full 1,272 questions | 49.61% | 88.36% | 1.94 s/question |
| ScreenSpot-Pro | Windows, stratified 200 questions (4K-heavy) | 18.00% | 78.5% | 3.87 s/question |
* oracle = the GT target has a recognizable candidate in the element table — an upper bound assuming a perfect matcher. It separates the recognition engine's account from the matcher's account: on ScreenSpot-Pro the gap is almost entirely matcher-side (match_error 88% of misses), which is where the roadmap points. The acc numbers are with a frozen heuristic matcher standing in for the agent; in real use the agent reads the table itself, so live experience tracks the oracle line more than acc. We report both, no cherry-picking.
Reproduce: python bench/public_sspot.py --dataset v2 / --dataset pro (datasets not included; see the docstring for the two HF sources).
MCP server
20 tools over stdio (scan_screen, scan_image, click_element, click_safe, click_xy, type_text, press_key, scroll_screen, snap_screen, crop_screen, crop_xy, book_find, watch_burst, verify_typing, probe_hang, split_static_dynamic, window_stack, activate_window, register_window, cleanup_windows). scan_image runs the same engines offline on a static file; book_find queries the persistent element book (two-level: text match + SigLIP vector search) so recurrent UIs are found without a full rescan.
Session cleanup (register-then-clean, sp/cleanup.py): the server opens a session ledger on start; register_window logs windows the agent launched itself, and the six state-changing tools auto-log windows newly opened by an action (advisory only). At task end, cleanup_windows closes ledger windows gently — WM_CLOSE only, no taskkill anywhere — behind a six-gate safety chain: identity re-check (hwnd+pid), permanent whitelist (explorer / task manager / own process), own-session ledger only, cross-session windows report-only, dirty-content protection (unsaved work is never closed), and survivor re-check that reports instead of escalating. If a session dies, a residual report is written and the next session's first tool call surfaces it automatically.
Documentation
Full docs (in Chinese, with internals, known limitations, and the test-suite map): README.zh-CN.md.
Status & roadmap
Working system in daily use as the studio's screen channel. Known gaps are documented, not hidden: dark-theme professional apps (deep-background small text) hurt OCR recall, 4K toolbar icons stress the matcher, and the Pro benchmark score is capped by the frozen matcher — see README.zh-CN.md §已知限制与坑.
License
Open core: the core engine in this repository is released under the Apache License 2.0. Commercial value-added integrations (closed source) are not part of this repository.
Acknowledgements
Built on RapidOCR, uiautomation, OpenCV, and onnxruntime.
XuLan ScreenPilot MCP questions
Does this require a GPU or cloud API?
No, it runs purely on the local CPU with no cloud models, no GPU, and no API key required.
How does the agent interact with the screen?
The agent scans the screen to create a numbered element table and then drives the screen using element IDs for actions like clicking and typing.
What engines are used to identify screen elements?
It uses a fusion of three engines: Windows UIA for native apps, RapidOCR for games and custom UIs, and OpenCV for visual icon detection.
Is it safe to use on sensitive windows?
Yes, it features a privacy-first mode that recognizes sensitive windows like Notepad or login dialogs and prevents them from being written into the persistent element book.