Desktop Computer Use MCP server
Commit Oct 18, 2026Allows an AI client to control Windows applications in the background using accessibility patterns and window messages without stealing the user's cursor.
Also known as dev-willbird1936/Desktop-Computer-Use
- Stars
- 7
- Tools
- 21
- Downloads, 30 days
- —
- Latest version
- —
- Last commit
- Oct 18, 2026
- License
- MIT
What is Desktop Computer Use MCP?
Desktop Computer Use allows an AI client to control Windows applications in the background using accessibility patterns and window messages without stealing the user's cursor. It exposes 21 tools, such as list_apps, get_app_state, click.
Summary generated from the project's README
About Desktop Computer Use MCP server
Desktop Computer Use, or DCU, is an MCP server designed for Windows users who want to automate desktop applications without losing control of their mouse. It primarily operates in the background by using accessibility patterns and window messages, which allows a user to continue working while an AI agent performs tasks. A green virtual cursor provides a visual indicator of what the agent is doing on the screen.
Developers can connect this server to clients like Claude Code or Codex using the MCP stdio interface. While it defaults to background control, it includes a foreground mode for applications or games that require actual system cursor and keyboard input. The server provides 21 specialized tools, including desktop management tools like list_apps and get_app_state, as well as specialized game tools like game_ocr and game_bars for interacting with native surfaces.
Setup involves downloading a self-contained Windows x64 executable that does not require a separate .NET installation. Users can fine-tune behavior through a settings.json file, adjusting parameters such as input speed, humanized motion, or whether to use the real system cursor.
What you can do with it
- 1Automate background desktop tasksAn AI agent can manage windows and type text in the background while you continue using your computer.
- 2Interact with complex gamesUse specialized tools to perform mouse-look actions and read game interfaces through OCR.
- 3Monitor application statesRetrieve snapshots of the accessibility tree and labeled elements to understand what is happening in a window.
- 4Execute multi-step automation sequencesRun a batch of several desktop actions in a single server-side command to improve efficiency.
Written from the project's README. Every use case is checked against a quote from it.
Desktop Computer Use MCP tools 21
Detected from the READMElist_appsget_app_stateclicktype_textpress_keyscrolldragset_valuewait_forexecute_sequencehide_cursorhealth_check
Show all 21 toolsShow fewer tools
game_screenshotgame_clickgame_movegame_draggame_scrollgame_keygame_ocrgame_barsgame_wait
Automated checks
Ran Nov 14, 2026Full report →Not verified yet. A server needs a score of 80, no serious finding, and enough data for us to check. How checks work · What to fix for more points
- Source repository linked and reachabledev-willbird1936/desktop-computer-use is public and reachablegithub.com/dev-willbird1936/desktop-computer-use ↗Pass
- Package names this repository as its sourceNot checked
- Build provenance (Sigstore attestation)Not checked
- Open source license (OSI)MITspdx.org/licenses/MIT.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-10-18github.com/dev-willbird1936/desktop-computer-use ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNot checked
- No critical advisory in the package or its dependenciesNot checked
- No hidden instructions to the model21 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNot checked
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubDesktop Computer Use (DCU)
Desktop Computer Use, or DCU, is a Windows computer-use MCP server. Background input is the default: accessibility patterns and window messages leave your real cursor available. Explicit foreground input uses the system cursor and keyboard for games or applications that reject background messages. Some accessibility providers can still activate their application; the optional UIA text fallback can briefly take focus.
Use the short name DCU in agent instructions to distinguish it from built-in foreground computer-use tools:
Use DCU to control the Windows desktop in the background.
A cosmetic green virtual cursor shows DCU actions while you continue working normally.
Why DCU
| Capability | Result |
|---|---|
| Background control | Designed to preserve your active application and real cursor; focus-related exceptions are listed below. |
| UIA-first actions | DCU uses accessibility patterns before window-message fallbacks. |
| Background screenshots | It can capture many visible or occluded windows without activating them. |
| Observable actions | Action results report detected effects instead of dispatch success alone. |
| MCP-native interface | Claude Code, Codex, and other MCP clients can use the same tool contract. |
The benchmark suite and current results are documented in docs/benchmarks.md.
This release line contains desktop control, normal/humanized foreground input, and native game tools. JEV, the browser adapter, learned visual discovery, and alternative driver/hardware/memory input backends remain experimental and are not included. See release notes and the release checklist.
Quick Start
Download the Windows x64 ZIP from the latest release, verify it against SHA256SUMS.txt, and extract it. Keep LICENSE and NOTICE.md with dcu.exe. The executable is self-contained and does not require a separate .NET installation. Earlier releases provide only the loose executable.
DCU requires an interactive Windows desktop. The executable is unsigned, so Windows may show a reputation warning. Only install binaries from a source you trust. Connect it through MCP stdio; this release does not expose an HTTP server.
Claude Code
claude mcp add dcu -- "C:\path\to\dcu.exe"
Codex or another MCP client
[mcp_servers.dcu]
command = 'C:\path\to\dcu.exe'
Run diagnostics at any time:
.\dcu.exe --doctor
Tools
All 21 desktop/game tools are listed and callable. GameMode is an input-profile preference, not a tool-access gate. An agent can explicitly select input mode and pace without changing settings or restarting DCU.
Desktop tools
| Tool | What it does |
|---|---|
list_apps |
Lists each visible top-level window with its PID, HWND, title, and stable window_id. |
get_app_state |
Returns an accessibility-tree snapshot, labeled elements, actions, values, and an optional annotated screenshot. |
click |
Uses UIA patterns or a message click and reports an observed effect. |
type_text |
Types through control messages or UIA value patterns. |
press_key |
Posts a key or key combination. |
scroll |
Uses the UIA scroll pattern, then a wheel-message fallback. |
drag |
Performs a message-based drag with interpolated movement. |
set_value |
Sets a value or toggle state directly. |
wait_for |
Polls for text, elements, or window state on the server. |
execute_sequence |
Runs several steps in one server-side batch. |
hide_cursor |
Hides the cosmetic virtual cursor. |
health_check |
Checks the desktop session, UIA thread, and overlay. |
If more than one window matches a name, title, or PID, DCU returns an ambiguity error. Use the window_id from list_apps to select the exact window.
Snapshot-based element clicks read the current UIA bounds before acting. Background snapshot coordinates and drag points are translated when the window moves. Take a new snapshot after a resize when the application uses a responsive layout.
Game and native-surface tools
| Tool | What it does |
|---|---|
game_screenshot |
Returns a JPEG, source_frame_id, and image_scale without requiring an accessibility tree. bring_to_front=true explicitly permits activation for GPU capture. |
game_click |
Clicks in background messages or real/humanized foreground input. |
game_move |
Moves the background pointer or real cursor; relative mouse-look requires foreground input. |
game_drag |
Drags with the selected input mode. |
game_scroll |
Scrolls at a selected point without requiring a UIA tree. |
game_key |
Taps, holds, or repeats keys/chords; foreground also supports mouse-button chords. |
game_ocr |
Reads text using installed Windows OCR language support; reports unsupported configurations explicitly. |
game_bars |
Measures calibrated HUD bars; unknown fill is not reported as zero. |
game_wait |
Waits for text, frame changes/stability, or foreground without repeating the input. Returns its last screenshot on timeout. |
For image coordinates, pass the source_frame_id from the screenshot you used. Another agent's screenshot does not replace that frame's scale. Frames are specific to the server instance and selected window; resize or expired history requires a fresh screenshot. coord_space="screen" instead uses absolute screen pixels.
Game actions distinguish accepted input from observed effect: frame_changed, minor_change, no_change, or unverified. Set then_screenshot=true to get the result image in the same call. Do not blindly repeat a dispatched action because no visual change was detected. Keys are held only for a bounded call, then release is attempted on completion, cancellation, or failure. Rejected cleanup is reported; permanent down/up controller ownership is not exposed.
Multiple agents
Background actions against different windows can progress independently. Actions against the same window are coordinated across DCU processes, and execute_sequence reserves that exact window for its entire sequence. Real-cursor actions also queue for the one shared Windows cursor. There is no persistent agent/window claim to release manually.
DCU retains 64 accessibility observations per window so another observation does not immediately invalidate your IDs. IDs are opaque and specific to the server instance; do not parse them or send another server's IDs. Share window_id, then obtain your own state when handing work to a separate client. Each window has an isolated UIA worker, so a wedged provider does not block other windows' accessibility work.
Your MCP host must still register/expose DCU to each agent. DCU cannot change host tool inheritance or create multiple independent system cursors.
Settings
DCU reads settings.json next to the executable, then falls back to %APPDATA%\shadow-use\settings.json.
Security: Trust gates are disabled by default. DCU acts on the target you select, including password fields.
{
"AllowUiaTextFallback": true,
"EnableBoundsGuard": false,
"EnableDesktopCheck": false,
"ShowVirtualCursor": true,
"EnableFocusGuard": true,
"PostActionDelayMs": 150,
"UseRealCursor": false,
"GameMode": false,
"HumanizedInput": false,
"FastInput": false
}
| Key | Default | Meaning |
|---|---|---|
AllowUiaTextFallback |
true |
Allows UIA SetValue as a typing fallback. It can briefly foreground some applications. |
EnableBoundsGuard |
false |
Refuses coordinate input when the window moved after the last snapshot. |
EnableDesktopCheck |
false |
Refuses actions on the lock screen or secure desktop. |
ShowVirtualCursor |
true |
Shows the green cosmetic cursor during actions. It disappears after one minute without use. |
EnableFocusGuard |
true |
Restores the foreground window and caret after an action takes them. |
PostActionDelayMs |
150 |
Wait time before the follow-up snapshot. |
Input toggles and defaults
Default: background input with the virtual cursor visible. Real-cursor input, game mode, humanized input, and fast input are disabled. Start with this configuration for ordinary desktop automation while you continue working:
{
"ShowVirtualCursor": true,
"UseRealCursor": false,
"GameMode": false,
"HumanizedInput": false,
"FastInput": false
}
These are the input-toggle defaults, not a request to disable existing focus protection or overwrite your saved preferences. They apply when settings are omitted; an existing settings.json can override them.
| Toggle | Default | What it does | When to enable it |
|---|---|---|---|
ShowVirtualCursor |
true |
Shows the green marker for background actions. It hides after one minute without activity. | Leave on to see what DCU is doing. Turn off if the marker is distracting or not needed. |
UseRealCursor |
false |
Uses your actual cursor/keyboard and brings the selected window forward instead of sending background messages. | Enable when the target ignores background actions, needs real modifier keys, or is a raw-input game. Your cursor and keyboard are no longer independently available during those actions. |
GameMode |
false |
Uses the game input profile and retains foreground/cursor after real input, avoiding mouse-look camera resets. | Enable for gameplay or mouse-look targets. Usually combine with UseRealCursor; it does not enable real input or unlock tools by itself. |
HumanizedInput |
false |
Uses curved foreground mouse motion and varied typing/key timing. | Enable when you want natural-looking motion or less uniform pacing. It can add latency and is not a promise of bypassing application input restrictions. |
FastInput |
false |
Shortens fixed delays and pacing in either input mode. Explicit key-hold durations and observation timeouts stay unchanged. | Enable for responsive targets, repetitive tasks, or latency-sensitive gameplay. Leave off if the target misses short taps; use longer explicit holds or waits when needed. |
All five toggles can be combined. No combination is rejected or silently changes the cursor choice. Humanization and game foreground-retention apply only when real-cursor input is selected; the virtual marker applies to background actions. Fast pacing works with either mode. Automated checks cover all 32 setting combinations and their per-call overrides, not every application's compatibility.
The background “own cursor” is a cosmetic overlay, not a second hardware pointer. ShowVirtualCursor controls its visibility; UseRealCursor controls how input is delivered. Enabling game mode, humanization, or fast input never takes the real cursor by itself. Raw-input games can still require real-cursor input.
Recommended combinations
Start with the defaults above, then enable only what the task needs. Other input toggles remain at their defaults.
| Task | Enable | Reason |
|---|---|---|
| Ordinary background desktop work | Nothing extra | Keeps the real cursor available and shows background actions with the virtual marker. |
| Faster background desktop work | FastInput |
Reduces pacing without taking the real cursor. |
| An app or shortcut that rejects background input | UseRealCursor |
Delivers real system input; pause your own cursor/keyboard use while it acts. |
| Gameplay or mouse-look | UseRealCursor + GameMode |
Uses real input and avoids restoring the cursor/foreground after each action. |
| Natural-looking foreground automation | UseRealCursor + HumanizedInput |
Adds curved movement and varied timing; add GameMode only for game-style foreground retention. |
| Latency-sensitive gameplay | UseRealCursor + GameMode + FastInput |
Favors quick reactions; optionally add HumanizedInput for faster humanized motion. |
ShowVirtualCursor can stay enabled in every configuration; it does not affect real input and is used for background actions. Foreground actions still share the single system cursor and can be interrupted by user activity.
Per-call input_mode overrides the cursor default:
"background": background input, even whenUseRealCursor=true."foreground": system input, using the configured humanization preference."foreground_plain": system input without humanization."humanized": humanized system input, even when the preference is off.
Use pace="fast" or pace="normal" for a call. Choose one input mode for an entire sequence; its steps cannot change that reservation.
Additional numeric settings are HumanizedMouseSpeed=1.0 and HumanizedTypingSpeed=1.0 (0.25–4.0), and HumanizedTypoRate=0.0 (0–0.2). Typos are off by default.
Run the bundled dcu-settings.bat to edit the toggles in a browser. This optional settings page requires Python 3; the MCP executable does not. It saves to an existing executable-side settings file when present, otherwise profile settings. Reconnect each DCU process after saving persistent settings. Per-call mode and pace do not need a restart.
Build From Source
Requirements:
- Windows
- .NET 10 SDK
dotnet build src/ShadowUse/ShadowUse.csproj
dotnet run --project src/ShadowUse/ShadowUse.csproj -- --doctor
Windows shortcuts are also included:
setup.bat
run.bat --doctor
test.bat
stress-test.bat
agent-test.bat
release.bat
Run release.bat to test and package a release candidate with its license and checksums. It requires Python 3 and PowerShell 7 in addition to the .NET SDK and never pushes, tags, or publishes. See Preparing a release.
For a standalone development publish:
dotnet publish src/ShadowUse/ShadowUse.csproj -c Release -r win-x64 `
--self-contained true -p:PublishSingleFile=true `
-p:IncludeNativeLibrariesForSelfExtract=true -o publish
Architecture
Program.cs MCP stdio host; logs go to stderr
Config/Settings settings.json loader
Automation/UiaThread dedicated STA thread for UIA COM
Automation/UiaService snapshots, element IDs, and re-resolution
Automation/BackgroundInput UIA-pattern-first input with message fallbacks
Capture/ScreenshotService PrintWindow capture with bounded worker isolation
Overlay/VirtualCursorOverlay cosmetic cursor and click animation
Safety/Guard optional bounds and desktop guards
Known Limits
- Raw-input games and applications that use
GetAsyncKeyStatecannot see background messages. Select foreground input; some protected games still reject normalSendInput. No driver or anti-cheat bypass is included. - Background modifier shortcuts do not always fire because posted messages do not update system-level modifier state. Foreground input uses real key events. Chrome address-bar focus is handled through UIA as a background exception.
- A message click can rarely make an application activate itself.
PrintWindowfails on some hardware-accelerated applications. DCU then returns no screenshot by default instead of capturing an unrelated window from the visible screen.- The UIA text fallback can briefly foreground some applications.
- Minimized windows do not have a capturable backing surface.
- A non-elevated DCU process cannot control elevated targets.
- A timeout cannot abort an already-running UIA COM call. A permanently unresponsive target can require restarting that DCU process for that window; other windows have separate workers, and native discovery/game input remain available.
- Foreground input necessarily competes with the physical user's cursor/keyboard. It is explicit opt-in, not simultaneous independent hardware input. Normal mode attempts to restore the cursor/foreground; game profile retains them. If the user switches to another app, DCU does not force the old foreground back.
Desktop Computer Use MCP questions
What tools does Desktop Computer Use MCP provide?
It exposes 21 tools, including list_apps, get_app_state, click, type_text, press_key.
Does this server take control of my mouse?
By default, it uses background input via accessibility patterns, leaving your real cursor available. You can explicitly enable foreground input if an application requires it.
Can I use this for gaming?
Yes, it includes specific game and native-surface tools, and a GameMode that can be combined with real-cursor input to handle mouse-look and gameplay.
How do I install it on Windows?
Download the Windows x64 ZIP from the latest release on GitHub, verify it, and extract it. The executable is self-contained and does not require .NET.
Can it capture screenshots of windows that are not in focus?
Yes, it can capture many visible or occluded windows without activating them.