MCP server Universal Computer Control
Commit 28 de out. de 2026Permite que agentes de IA observem, cliquem, digitem e controlem computadores Windows ou Linux por meio de camadas de acessibilidade, OCR e modelos de visão.
Também conhecido como adkdev200/Universal-Computer-Control
- Stars
- 2
- Ferramentas
- —
- Downloads, 30 dias
- —
- Versão mais recente
- —
- Último commit
- 28 de out. de 2026
- Licença
- MIT
O que é o MCP Universal Computer Control?
Permite que agentes de IA observem, cliquem, digitem e controlem computadores Windows ou Linux por meio de camadas de acessibilidade, OCR e modelos de visão.
Resumo gerado a partir do README do projeto
Sobre o MCP server de Universal Computer Control
Dar olhos e mãos aos agentes de IA em qualquer computador é o objetivo principal deste MCP server. Ele permite que um cliente de IA observe, clique, digite e controle máquinas Windows ou Linux interagindo com a interface gráfica real, em vez de depender de protocolos específicos de navegador como o Playwright. Isso o torna adequado para controlar IDEs, suítes de escritório, terminais e qualquer outro aplicativo desktop.
O servidor opera localmente via stdio e fornece 29 ferramentas sob o namespace computer. Ele utiliza uma hierarquia de fallback automatizada que transita de camadas de acessibilidade, como Windows UIA ou Linux AT-SPI, para OCR, templates de imagem e modelos de visão-linguagem. Isso garante que, se um método falhar ao identificar um botão ou texto, o sistema tentará automaticamente outro para encontrar as coordenadas corretas para a entrada de dados.
Desenvolvedores podem configurar a segurança por meio de modos permissivo, padrão ou estrito, incluindo listas de permissão de comandos e etapas de confirmação para ações perigosas. Também inclui uma parada de emergência global para interromper todas as ações imediatamente. A configuração requer Python 3.11+ e pode ser estendida com endpoints de visão compatíveis com OpenAI para uma compreensão visual aprimorada.
O que você pode fazer com ele
- 1Automatizar fluxos de trabalho de aplicativos desktopUm agente de IA pode navegar por softwares que não são de navegador, como gerenciadores de arquivos ou IDEs especializadas, interagindo com elementos da UI.
- 2Realizar identificação de elementos visuaisO agente pode encontrar botões ou textos específicos na tela usando OCR ou modelos de visão quando os dados de acessibilidade não estiverem disponíveis.
Escrito a partir do README do projeto. Cada caso de uso é verificado com uma citação dele.
Configuração
UCC_ENGINE__VERIFICATION_MODEUCC_LOGGING__LEVELUCC_SECURITY__ALLOWED_COMMANDSUCC_SECURITY__MODEUCC_VISION__VLM__ENABLED
| Nome | Descrição | Tipo |
|---|---|---|
| UCC_ENGINE__VERIFICATION_MODE | — | |
| UCC_LOGGING__LEVEL | — | |
| UCC_SECURITY__ALLOWED_COMMANDS | — | |
| UCC_SECURITY__MODE | — | |
| UCC_VISION__VLM__ENABLED | — |
Verificações automatizadas
Executado em 14 de nov. de 2026Relatório completo →Ainda não verificado. Um servidor precisa de uma pontuação de 80, nenhum problema grave e dados suficientes para verificarmos. Como as verificações funcionam · O que corrigir para ganhar mais pontos
- Repositório de origem vinculado e acessíveladkdev200/universal-computer-control is public and reachablegithub.com/adkdev200/universal-computer-control ↗Passou
- O pacote indica este repositório como sua origemNão verificado
- Proveniência da build (atestação Sigstore)Não verificado
- Licença de código aberto (OSI)MITspdx.org/licenses/MIT.html ↗Passou
- Commit ou release no último anoLast commit or release 2026-10-28github.com/adkdev200/universal-computer-control ↗Passou
- OpenSSF ScorecardNão verificado
- Nome do pacote não é próximo a um pacote popularNão verificado
- Nenhum aviso crítico no pacote ou em suas dependênciasNão verificado
- Sem instruções ocultas para o modelo0 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPassou
- Ferramentas destrutivas estão rotuladasNão verificado
- O endpoint remoto responde ao handshake do MCPNão verificado
README
Ver no GitHubUniversal Computer Control MCP
Give your AI agent eyes and hands on any computer. Observe → understand → act → verify. No browser MCPs. No Playwright. Just real control.
A cross-platform computer control MCP server that lets an AI agent operate a Windows or Linux computer like a human: observe the screen → understand the UI → choose an action → perform it → observe again → verify → recover if needed.
The system deliberately has no dependency on Chrome MCP, browser MCP, Playwright MCP or any other external MCP server. Browsers, IDEs, office suites, terminals, file managers, Electron apps and canvas-heavy applications are all just GUI applications. Multiple backends live inside the server and the best available mechanism is chosen automatically per action.
AI Agent / LLM
| MCP (stdio)
v
+------------------------------+
| Universal Computer MCP Server |
+---------------+--------------+
v
+------------------------------+
| Computer Control Engine | <- usable WITHOUT MCP
+---------------+--------------+
| | |
v v v
Accessibility Vision/OCR Physical Input
Layer Layer Layer
| | | | | |
UIA AT-SPI OCR OpenCV Mouse Keyboard
(Windows)(Linux) Tesseract VLM PyAutoGUI OS APIs
Highlights
- 29 stable MCP tools under the
computer.namespace — capabilities, not implementation details. Nopyautogui_click, noocr_click, noplaywright_click. - Interaction hierarchy per action: Windows UIA → Linux AT-SPI → semantic element → OCR text → image template → vision-language model → coordinates → PyAutoGUI. Every fallback is automatic.
computer.observeis the core tool: one structured snapshot with screen geometry, active window, cursor, OCR'd text and normalized UI elements (stableelement_Nids within a cycle).
Requirements
| Mandatory | Optional (recommended) | |
|---|---|---|
| Python | 3.11+ | |
| Core | mcp, pydantic, PyYAML, Pillow |
|
| Input | PyAutoGUI ([input]) |
|
| Windows | pywinauto, pygetwindow, psutil ([windows]) |
|
| Linux | psutil ([linux]), system wmctrl, xdotool, xclip |
|
| Screenshots | mss ([screenshot]), scrot on X11 |
|
| OCR | pytesseract + system tesseract-ocr ([ocr]) |
|
| Vision | opencv-python, numpy ([vision]) |
|
| VLM | httpx ([vlm]) + an OpenAI-compatible endpoint |
Installation
Windows 10/11
git clone https://github.com/adkdev200/universal-computer-control.git && cd universal-computer-control
powershell -ExecutionPolicy Bypass -File scripts\install_windows.ps1
Manual steps:
- Install Python 3.11+ (
winget install -e --id Python.Python.3.12), tick Add to PATH. python -m venv .venv && .venv\Scripts\pip install -e ".[input,windows,ocr,vision,screenshot,vlm]"- Install Tesseract OCR:
winget install -e --id UB-Mannheim.TesseractOCR(setocr.tesseract_cmdinconfig.yamlif it is not onPATH). - Run:
.venv\Scripts\universal-computer-control(alias:.venv\Scripts\ucc)
Windows UI Automation needs no extra permissions. If you run the server elevated, non-elevated windows may refuse automation (UIA integrity levels).
Linux (X11 or XWayland)
git clone https://github.com/adkdev200/universal-computer-control.git && cd universal-computer-control
bash scripts/install_linux.sh --yes # add --no-sudo to skip system packages
./.venv/bin/ucc-doctor # verify every backend, exact fix hints
The installer is fully non-interactive with --yes: it installs system
packages (wmctrl/xdotool/xclip/scrot/tesseract/at-spi2-core), all Python
extras, and a ready ~/.universal-computer/config/config.yaml with OCR,
template matching and the VLM provider enabled. ucc-doctor then probes every
backend live and prints copy-paste fix commands for anything missing. Note:
even on machines with no wmctrl/xdotool/xclip installed, the Linux window
backend now works through a built-in python-Xlib (EWMH/ICCCM) fallback - the
system tools only make it more robust.
Manual steps:
-
Python 3.11+ and system packages:
sudo apt install python3 python3-venv python3-pip \ wmctrl xdotool xclip scrot \ tesseract-ocr at-spi2-core -
python3 -m venv .venv && ./.venv/bin/pip install -e ".[input,linux,ocr,vision,screenshot,vlm]" -
Run from a graphical session:
./.venv/bin/universal-computer-control(alias:./.venv/bin/ucc)
AT-SPI requirements (accessibility): at-spi2-core must be running (it is
on desktop distros) and for GNOME/GTK apps enable
gsettings set org.gnome.desktop.interface toolkit-accessibility true.
X11 vs Wayland: X11 is fully supported. On Wayland-with-XWayland, input
and screenshots work through XWayland but foreign window management may be
restricted by the compositor. On pure Wayland (no DISPLAY), the window
backend disables itself with a warning; AT-SPI still works for cooperating
toolkits. See docs/TROUBLESHOOTING.md.
Registering the server with an MCP client
claude_desktop_config.json (see examples/mcp-config.example*.json):
{
"mcpServers": {
"universal-computer": {
"command": "/absolute/path/to/universal-computer-control/.venv/bin/python",
"args": ["-m", "universal_computer"],
"env": {}
}
}
}
Any MCP client that speaks stdio works the same way. The engine is also a plain library — no MCP required:
from universal_computer import ComputerControlEngine, load_config
computer = ComputerControlEngine(load_config())
obs = await computer.observe("normal")
await computer.click("Continue")
await computer.type("hello")
Tools (MCP API)
| Group | Tools |
|---|---|
| Observation | computer.observe (levels: minimal/normal/full) · computer.screenshot · computer.get_ui_tree · computer.get_active_window · computer.find_text · computer.find_visual · computer.wait_for |
| Mouse | computer.click · computer.double_click · computer.right_click · computer.move_mouse · computer.drag · computer.scroll |
| Keyboard | computer.type · computer.press · computer.hotkey |
| Windows | computer.list_windows · computer.focus_window · computer.minimize_window · computer.maximize_window · computer.restore_window · computer.close_window (needs confirm=true) |
| Clipboard | computer.get_clipboard · computer.set_clipboard |
| System | computer.launch_application · computer.run_command (allowlist + confirm=true) |
| Admin | computer.backend_status · computer.emergency_stop · computer.reset_emergency_stop |
computer.observe returns a normalized snapshot:
{
"ok": true,
"id": "obs_17",
"level": "normal",
"screen": {"width": 1920, "height": 1080, "monitors": [...]},
"active_window": {"title": "Google Chrome", "application": "chrome.exe", "process_id": 1234},
"cursor": {"x": 812, "y": 530},
"elements": [
{"id": "element_1", "role": "button", "name": "Continue",
"bbox": [1050, 700, 1170, 750], "source": "uia", "confidence": 0.99}
],
"ocr": [
{"text": "Continue", "bbox": [1050, 700, 1170, 750], "confidence": 0.97}
]
}
computer.backend_status reports per-backend health:
{
"pyautogui": {"available": true, "healthy": true},
"uia": {"available": true, "healthy": true},
"atspi": {"available": false, "healthy": false,
"error": "pyatspi import failed (...); install python3-pyatspi"},
"ocr": {"available": true, "healthy": true}
}
Configuration
config.yaml (see config.example.yaml) plus UCC_ environment overrides
with __ nesting:
UCC_ENGINE__VERIFICATION_MODE=strict
UCC_SECURITY__MODE=standard
UCC_SECURITY__ALLOWED_COMMANDS='["ls", "git status", "xdotool"]'
UCC_LOGGING__LEVEL=DEBUG
UCC_VISION__VLM__ENABLED=true
Configurable: observation level & caching, verification mode
(off/basic/strict), PyAutoGUI pauses/failsafe, OCR provider & language,
vision threshold, template directory, VLM endpoint (key read from the
environment variable named in vision.vlm.api_key_env), security mode,
allowlists, rate limits, timeouts, persistence retention, log levels and
per-backend priorities (backends.priorities: {pyautogui: 5}).
Security model
| Mode | run_command |
launch_application |
|---|---|---|
permissive |
allowed (still rate-limited) | allowed |
standard (default) |
requires security.allowed_commands and confirm=true |
allowed unless a non-empty allowed_applications says otherwise |
strict |
only explicitly allowlisted; shell strings off unless allow_shell |
exact allowlist match required |
Always on: rate limiting (max_actions_per_minute), command timeouts,
credential redaction in logs and command output, no clipboard/typed-text
logging, close_window/run_command confirmation gates, and the
emergency stop — computer.emergency_stop() halts every subsequent
action (checked before each attempt) and is persisted as a stop file so it
survives restarts and can be triggered externally.
Extending
- New OCR: implement
universal_computer.vision.base.OCRProvider(detect_text(image) -> list[OCRResult]) and register it inVisionServices.from_config. - New vision model: implement
VisionLanguageModel.locate_element(async, description →VisionResultwith bbox). The OpenAI-compatible implementation shows the JSON contract. - New platform (macOS): subclass the
backends.baseABCs (InputBackend,ScreenshotBackend,AccessibilityBackend,WindowManagementBackend,ClipboardBackend,ApplicationBackend) and register them incore/bootstrap.py. Nothing else changes — the engine only knows capabilities. - Template images: put PNGs in
vision.template_dir; refer to them by name fromcomputer.find_visualor as click targets.
Development
pip install -e ".[dev]"
pytest tests # 160 unit + integration tests, no display needed
ruff check src tests
scripts/e2e_live_test.py is an end-to-end harness that boots its own
Xvfb + openbox session, creates real X11 windows, and exercises every
backend live (screenshots, OCR, window ops via both the tool paths and the
python-Xlib fallback, OpenCV template matching, the VLM pipeline against a
mock OpenAI-compatible endpoint, clipboard, input, launch, emergency stop):
python scripts/e2e_live_test.py # full stack
python scripts/e2e_live_test.py --no-tools # zero X11 tools: Xlib fallback
Unit tests run on any machine (OS-specific pieces are mocked). The
integration test boots the real server over stdio. See
docs/ARCHITECTURE.md for the module map and docs/TROUBLESHOOTING.md for
platform-specific fixes.
Contributors & Acknowledgments
- Z.ai (GLM) — co-developer: MCP server, cross-platform
backends, vision pipeline, python-Xlib window fallback and
ucc-doctordiagnostics - Claude Code (Anthropic) — co-developer: code generation, testing and plug-and-play hardening
- @adkdev200 — creator & maintainer
This project is built collaboratively with AI pair-programming: the majority of the codebase was written and reviewed by Z.ai's GLM and Anthropic's Claude Code under the direction of the maintainer.
Perguntas sobre o MCP Universal Computer Control
Isso requer um browser MCP ou Playwright?
Não, o sistema não possui dependência de Chrome MCP, browser MCP, Playwright MCP ou qualquer outro MCP server externo.
Quais sistemas operacionais são suportados?
O servidor suporta as plataformas Windows e Linux.
Como posso interromper o agente de IA se ele perder o controle?
Você pode usar a ferramenta de parada de emergência global, que é persistida como um arquivo de parada para sobreviver a reinicializações.
Ele pode funcionar sem OCR ou modelos de visão?
Sim, o servidor é totalmente funcional usando apenas PyAutoGUI caso outros componentes estejam ausentes.