Shadow Vision MCP server
Commit Oct 17, 2026Enables text-only LLMs to perform OCR, visual analysis, UI inspection, and image reconstruction using pluggable vision backends.
Also known as WardLu/shadow-vision
- Stars
- 1
- Tools
- 3
- Downloads, 30 days
- —
- Latest version
- —
- Last commit
- Oct 17, 2026
- License
- MIT
What is Shadow Vision MCP?
Shadow Vision enables text-only LLMs to perform OCR, visual analysis, UI inspection, and image reconstruction using pluggable vision backends. It exposes 3 tools, such as task, image_path, image_url.
Summary generated from the project's README
About Shadow Vision MCP server
Giving text-only LLMs the ability to see and analyze visual information is the primary purpose of shadow-vision. This MCP server allows AI agents to perform tasks like optical character recognition, UI inspection, and image reconstruction without requiring the host model itself to have native vision capabilities. It works by acting as a bridge, forwarding image data and prompts to a configured vision backend and returning the text results to the client.
Developers can use this service with various backends including Ollama for local-first privacy, or remote services like Anthropic, Gemini, and OpenAI-compatible providers. It is compatible with MCP clients such as Claude Desktop, Cursor, and Codex. The server supports multiple input methods including local file paths, base64 data, and remote HTTP(S) URLs. Specific tools provided include vision_ocr for text extraction, vision_inspect for answering questions about images, vision_annotate for identifying bounding boxes, vision_layout for analyzing UI structures, vision_reconstruct for turning screenshots into code, and vision_compare for analyzing differences between multiple images.
What you can do with it
- 1Extract text from documentsA developer can use the OCR tool to transcribe text from screenshots, invoices, or tables.
- 2Analyze user interface layoutsAn AI agent can inspect a UI screenshot to identify bugs or understand the element hierarchy.
Written from the project's README. Every use case is checked against a quote from it.
Shadow Vision MCP tools 3
Detected from the READMEtaskimage_pathimage_url
Configuration
VISION_BACKEND
Automated checks
Ran Nov 14, 2026Full report →Not verified yet. A server needs a score of 80, no serious finding, and enough data for us to check. How checks work · What to fix for more points
- Source repository linked and reachablewardlu/shadow-vision is public and reachablegithub.com/wardlu/shadow-vision ↗Pass
- Package names this repository as its sourceNot checked
- Build provenance (Sigstore attestation)Not checked
- Open source license (OSI)MITspdx.org/licenses/MIT.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-10-17github.com/wardlu/shadow-vision ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNot checked
- No critical advisory in the package or its dependenciesNot checked
- No hidden instructions to the model3 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNo tool named like delete, drop or execPass
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubShadow Vision
English | 简体中文 | Shadow Lab
Give text-only LLMs a pair of eyes. Shadow Vision is an open-source MCP vision service that enables AI Agents to see, understand, and analyze real-world information through vision_ocr, vision_inspect, vision_annotate, vision_layout, vision_reconstruct, and vision_compare — without switching host models.
Why It's Different
- MCP-Native: Compatible with Codex, Claude Desktop, Cursor, and other MCP clients
- Pluggable Backends: Ollama, OpenAI-compatible, Anthropic, Gemini
- Local-First: Keep images and inference entirely on your machine when using Ollama
- Versatile Input: Supports local file paths, base64 image data, or remote HTTP(S) URLs
How It Works
The text model calls Shadow Vision tools via MCP. Shadow Vision forwards the image and prompt to the configured vision backend and returns the text result back to the model.
Quick Start
0. One-Click Run (No Clone Needed)
No need to clone the repository. Run directly:
uvx shadow-vision # Python / uv users (recommended)
# Or for Node users
npx shadow-vision # Requires uv installed locally
MCP configuration example:
[mcp_servers.vision]
command = "uvx"
args = ["shadow-vision"]
env = { VISION_BACKEND = "ollama", VISION_MODEL = "qwen3-vl:2b-instruct" }
npx shadow-visionis a thin wrapper that invokesuvx shadow-visioninternally and requires uv installed on your machine. Both entry points behave identically.
1. Install from Source (Development / Self-Hosting)
Requires Python 3.11+ and uv:
git clone https://github.com/WardLu/shadow-vision.git
cd shadow-vision
uv sync
2. Use Local Ollama (Recommended for Beginners)
First install Ollama. If not running the Ollama desktop app, start the service manually:
ollama serve
ollama pull qwen3-vl:2b-instruct
ollama list
qwen3-vl:2b-instruct is the default vision model (non-thinking version, faster response). If you need stronger reasoning/thinking capability, switch to qwen3-vl:2b (thinking version); or change VISION_MODEL to any other vision model listed in ollama list.
3. Register as an MCP Service
Codex can run directly:
codex mcp add vision -- uv run shadow-vision
Or add to ~/.codex/config.toml:
[mcp_servers.vision]
type = "stdio"
command = "uv"
args = ["run", "shadow-vision"]
cwd = "/path/to/shadow-vision"
env = { VISION_BACKEND = "ollama", VISION_MODEL = "qwen3-vl:2b-instruct", OLLAMA_URL = "http://127.0.0.1:11434/api/chat" }
After restarting your MCP client, simply ask the model to "take a look at this image".
Switching Models and Backends
VISION_BACKEND determines how requests are routed, and VISION_MODEL specifies the vision model. Update the environment variables in your MCP configuration and restart your client.
Switch to a local model:
env = { VISION_BACKEND = "ollama", VISION_MODEL = "your-downloaded-vision-model", OLLAMA_URL = "http://127.0.0.1:11434/api/chat" }
Switch to an OpenAI-compatible service:
env = { VISION_BACKEND = "openai_compatible", VISION_MODEL = "provider-vision-model-name", OPENAI_API_BASE = "https://api.example.com/v1", OPENAI_API_KEY = "sk-...", OPENAI_MAX_TOKENS = "1024", OPENAI_MAX_TOKENS_FIELD = "max_tokens" }
OPENAI_* denotes the OpenAI Chat Completions compatible protocol, which also applies to LM Studio, vLLM, and any other service offering /v1/chat/completions.
Free vision example for Chinese platforms (Zhipu GLM-4V-Flash):
env = { VISION_BACKEND = "openai_compatible", VISION_MODEL = "glm-4v-flash", OPENAI_API_BASE = "https://open.bigmodel.cn/api/paas/v4", OPENAI_API_KEY = "your-zhipu-key" }
Other OpenAI-compatible providers only require modifying OPENAI_API_BASE and VISION_MODEL: SiliconFlow https://api.siliconflow.cn/v1, Alibaba Bailian https://dashscope.aliyuncs.com/compatible-mode/v1, StepFun https://api.stepfun.com/v1, Tencent Hunyuan https://api.hunyuan.cloud.tencent.com/v1, Moonshot https://api.moonshot.cn/v1, etc.
Privacy Notice: API backends (including third-party platforms) send image content as base64 to the respective provider's servers. For confidential or sensitive images, use the local
ollamabackend to prevent external data transfer.
Configuring Backends
General Variables
| Variable | Default | Description |
|---|---|---|
VISION_BACKEND |
ollama |
ollama / openai_compatible / anthropic / gemini |
VISION_MODEL |
qwen3-vl:2b-instruct |
Vision model name |
VISION_TIMEOUT |
180 |
Read timeout (seconds), compatibility alias for VISION_READ_TIMEOUT |
VISION_CONNECT_TIMEOUT |
10 |
Connect timeout (seconds) |
VISION_READ_TIMEOUT |
180 |
Read timeout (seconds) |
VISION_MAX_RETRIES |
2 |
Retry attempts for transient / 5xx errors (total requests = 1 + this value) |
VISION_RETRY_BASE_DELAY |
1.0 |
Exponential backoff base delay in seconds |
Advanced Configuration (Images & Security)
| Variable | Default | Description |
|---|---|---|
VISION_AUTO_COMPRESS |
true |
Whether to automatically compress large images |
VISION_MAX_LONG_EDGE |
1800 |
Compression threshold: long-edge pixels |
VISION_MAX_PIXELS |
3500000 |
Compression threshold: total pixels |
VISION_COMPRESS_QUALITY |
85 |
JPEG re-encoding quality |
VISION_AUTO_TILE |
true |
Whether to automatically tile extra-long images |
VISION_TILE_LONG_EDGE |
3600 |
Tiling threshold: long-edge pixels |
VISION_TILE_OVERLAP |
100 |
Tiling overlap pixels |
VISION_MAX_TILES |
8 |
Maximum number of tiles per image |
VISION_TASK_ROUTING |
true |
Whether to enable heuristic task routing for vision_inspect |
VISION_ALLOW_REMOTE_URL |
true |
Whether to allow remote URL image inputs |
VISION_MAX_REMOTE_SIZE |
20971520 |
Maximum remote image size in bytes (20MB) |
VISION_FETCH_TIMEOUT |
30 |
Remote fetch timeout (seconds) |
VISION_SSRF_ALLOW_PRIVATE |
false |
Whether to allow private / intranet addresses (strongly discouraged) |
VISION_MAX_BATCH_IMAGES |
5 |
Maximum number of images per vision_compare call |
Ollama
| Variable | Default | Description |
|---|---|---|
OLLAMA_URL |
http://127.0.0.1:11434/api/chat |
Ollama chat endpoint |
Run ollama pull <vision-model-name> before use to download the model.
OpenAI-compatible
| Variable | Default | Description |
|---|---|---|
OPENAI_API_BASE |
http://127.0.0.1:11434/v1 |
Compatible service base URL |
OPENAI_API_KEY |
empty | Usually left blank for local services |
OPENAI_MAX_TOKENS |
unset | Optional max output tokens; omits token limit field when unset |
OPENAI_MAX_TOKENS_FIELD |
max_tokens |
Optional: max_tokens or max_completion_tokens |
Different providers support different token limit fields: use max_tokens for legacy fields, max_completion_tokens for newer APIs, or leave OPENAI_MAX_TOKENS unset if neither is accepted. Legacy environment variables VISION_API_BASE, VISION_API_KEY, VISION_MAX_TOKENS, and VISION_MAX_TOKENS_FIELD remain supported for backward compatibility.
Anthropic / Gemini
VISION_BACKEND=anthropic ANTHROPIC_API_KEY=sk-ant-... VISION_MODEL=your-claude-vision-model uv run shadow-vision
VISION_BACKEND=gemini GEMINI_API_KEY=AIza... VISION_MODEL=your-gemini-vision-model uv run shadow-vision
Anthropic also supports ANTHROPIC_BASE_URL, ANTHROPIC_VERSION, and ANTHROPIC_MAX_TOKENS; Gemini also supports GEMINI_BASE_URL and GEMINI_MAX_TOKENS.
Tools
vision_ocr
Extract text from screenshots, invoices/receipts, documents, or tables:
vision_ocr(image_path="/tmp/receipt.png")
vision_inspect
Describe an image, or answer questions about an image:
vision_inspect(image_path="/tmp/design.png", question="List any UI bugs you see.")
Both tools also support:
task: Optional task guidance (vision_ocr:general/error/table;vision_inspect:general/ui_structure/ui_bug/chart)image_path: Server-readable local image pathimage_base64+mime_type: Base64-encoded image dataimage_url: Remote HTTP(S) image URL (with automatic SSRF protection)
All image tools accept one of image_path / image_base64 / image_url as input, with precedence: image_base64 > image_path > image_url.
vision_annotate
Identify bounding boxes, arrows, underlines, highlights, strikethroughs, handwritten notes, and other user annotations. Outputs structured JSON with annotation → target relationships, types, bounding boxes (bbox), and confidence scores:
vision_annotate(image_path="/tmp/marked.png", focus="Explain changes in order of markup")
vision_layout
Analyze image and UI layout structure, outputting structured JSON with canvas, containers, element bounding boxes (bbox), typography styles, and element hierarchy:
vision_layout(image_path="/tmp/ui.png")
vision_reconstruct
Reconstruct screenshots into code (html / react / svg), generating markup along with model self-inspection. Optionally provide JSON from vision_layout as layout reference:
vision_reconstruct(image_path="/tmp/ui.png", target_format="html", reference_layout="<layout json>")
vision_compare
Analyze multiple related images in a single call (diff / compare / sequence), with optional per-image label for easy reference:
vision_compare(images=[{"image_path": "/tmp/a.png", "label": "before"}, {"image_path": "/tmp/b.png", "label": "after"}], task="diff")
Local Model Selection and Benchmarking
The Ollama model library lists download size, context window, and vision capabilities, but model package size does not represent minimum memory requirements. It is recommended to start with qwen3-vl:2b-instruct (non-thinking version, low latency); if OCR or complex chart comprehension is insufficient, benchmark against qwen3-vl:4b, qwen3-vl:8b, or document-OCR oriented minicpm-v4.5:q4_0.
Prepare 3–5 real-world images covering OCR, UI screenshots, charts, and hard edge cases, and compare models using identical prompts:
MODEL=qwen3-vl:2b-instruct
IMAGE=/absolute/path/to/test.png
time ollama run "$MODEL" "$IMAGE" "Transcribe all text from the image accurately, outputting only the text."
time ollama run "$MODEL" "$IMAGE" "Describe the image contents and list any areas of uncertainty."
Track OCR error count, accuracy of key objects and relationships, hallucinations, full round-trip latency, and processor status in ollama ps. Testing Ollama directly first before testing through vision_ocr / vision_inspect via MCP helps distinguish model capability limits from MCP configuration issues.
Recommended resources:
- Ollama Vision Documentation
- Ollama Qwen3-VL Model Page
- Qwen3-VL Official Repository
- MiniCPM-V 4.5 Official Evaluation
- Ollama Context Length Documentation
- Ollama Modelfile Documentation
Supported Agents
All agents launch the same command: uv run shadow-vision.
| Agent | Configuration File |
|---|---|
| Codex | ~/.codex/config.toml |
| Claude Code | .mcp.json |
| Cursor | .cursor/mcp.json |
| VS Code Copilot | .vscode/mcp.json |
| Windsurf | .windsurf/mcp_config.json |
| Claude Desktop | claude_desktop_config.json |
| OpenCode | opencode.json |
Development
uv sync
uv run python -c "import vision_mcp.server; print('ok')"
uv run pytest
Contact
If you are interested in B2B products, AI product development, supply chain digitization, or Shadow series products, feel free to get in touch:
- X (Twitter): @Gollumgulu
- WeChat Official Account: Ward 的 AI 产品实战
- Xiaohongshu / Weibo / Douyin: "Ward 的 AI 产品实战" across platforms — Xiaohongshu · Weibo · Douyin
- Product Homepage: Shadow Lab
- Email: wardlu@126.com
Open for 1-on-1 consulting and advisory: Product Diagnostics · AI Implementation · Workflows / Skills · Custom Solutions
License
MIT
Shadow Vision MCP questions
What tools does Shadow Vision MCP provide?
It exposes 3 tools, including task, image_path, image_url.
How can I keep my images private?
You can use the local Ollama backend to ensure images and inference remain entirely on your machine, preventing external data transfer.
Can I use this with my existing text-only LLM?
Yes, it is designed to give text-only LLMs visual capabilities like OCR and UI inspection without needing to switch the host model.