Optics Framework MCP server
Commit Nov 13, 2026Lets an AI agent create and run self-healing CSV/YAML test automation on Android, iOS, web, smart TV and Bluetooth devices.
Also known as mozarkai/optics-framework
- Stars
- 99
- Tools
- —
- Downloads, 30 days
- —
- Latest version
- —
- Last commit
- Nov 13, 2026
- License
- Apache-2.0
What is Optics Framework MCP?
Optics Framework lets an AI agent create and run self-healing CSV/YAML test automation on Android, iOS, web, smart TV and Bluetooth devices.
Summary generated from the project's README
About Optics Framework MCP server
Automating tests across Android, iOS, web, smart TV, and Bluetooth devices is made possible through this framework which allows AI agents to drive hardware and software directly. By exposing every keyword as a typed tool and device state as resources, the server enables an AI client to interact with real-world interfaces using natural language or structured commands. It is designed for developers and QA engineers who need reliable automation that does not break when UI elements change.
The system uses a locator ladder to find elements, starting with XPath and moving through text detection, OCR, and image matching before finally using an LLM for self-healing recovery. Users can manage test data through CSV or YAML files rather than writing complex code. To use the MCP server, you can configure your client to run the command optics mcp. Depending on your target, you may need to install specific drivers like Appium, Selenium, or Playwright, along with OCR engines such as EasyOCR or Google Vision.
What you can do with it
- 1Self-healing UI automationAn AI agent can recover a test when a button's locator changes by using an LLM to read the screen.
- 2Cross-platform device testingDevelopers can run the same test keywords across mobile, web, and smart TV platforms.
- 3Non-intrusive Bluetooth testingYou can drive production devices that block debugging by using the Bluetooth driver as a HID mouse.
- 4Data-driven test managementNon-coders can write complex test scenarios using simple CSV or YAML files.
Written from the project's README. Every use case is checked against a quote from it.
Automated checks
Ran Nov 14, 2026Full report →Not verified yet. A server needs a score of 80, no serious finding, and enough data for us to check. How checks work · What to fix for more points
- Source repository linked and reachablemozarkai/optics-framework is public and reachablegithub.com/mozarkai/optics-framework ↗Pass
- Package names this repository as its sourceNot checked
- Build provenance (Sigstore attestation)Not checked
- Open source license (OSI)Apache-2.0spdx.org/licenses/Apache-2.0.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-11-12github.com/mozarkai/optics-framework ↗Pass
- OpenSSF ScorecardOpenSSF Scorecard 7.4 / 10; weakest: Token-Permissions 0, Fuzzing 0, Pinned-Dependencies 2scorecard.dev/viewer/?uri=github.com/mozarkai/optics-framewo ↗Pass
- Package name not close to a popular packageNot checked
- No critical advisory in the package or its dependenciesNot checked
- No hidden instructions to the model0 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNot checked
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubOptics Framework
Self-healing test automation for mobile, web, TV — and AI agents.
One keyword engine. Six ways to drive it: CSV/YAML files, a Python SDK, Robot Framework, a REST API, an interactive terminal, or an MCP server your AI agent talks to.

Documentation · Install · Getting Started · Keywords · Architecture
Most frameworks assume a UI element has one true locator, and a test breaks the moment that locator changes. Optics assumes an element has several plausible identities — its XPath, its visible text, what it looks like on screen — and tries all of them before giving up.
That idea runs through the whole framework: locators fall back, drivers fall back, element values fall back. Tests are data (CSV or YAML), so non-coders can write them, and the same keywords are reachable from Python, Robot Framework, HTTP, and MCP.
Install
One line puts the optics CLI on your PATH:
curl -fsSL https://optics-framework.org/install | sh # macOS, Linux
irm https://optics-framework.org/install.ps1 | iex # Windows
It installs into its own virtual environment under ~/.optics, so nothing lands in your system Python. Then optics quickstart builds a project, checks your setup, and runs your first test.
Already managing your own environment? Optics needs Python 3.12+, and the core install ships no drivers, OCR, or LLM backends — you add only what you need:
python3 -m venv venv && source venv/bin/activate
pip install "optics-framework[appium,easyocr]"
Most extra names match the config.yaml source keys, so the word you install is usually the word you enable (exceptions: google-vision enables google_vision, and the bundles and mcp aren't config keys):
| Drivers | appium · selenium · playwright · ble |
| OCR | easyocr · pytesseract · google-vision |
| AI | gemini · openai · anthropic (natural-language mode + self-heal) · mcp (MCP server) |
| Bundles | mobile · web · vision · llm · all |
optics setup installs the drivers, OCR and LLM engines above (and the bundles) by name — pinned to your installed Optics version — and bare optics setup opens a TUI picker. optics quickstart goes one step further and offers the right extras for the platform you pick:
optics setup --list
optics setup --install appium easyocr
[!NOTE] The installer always includes the
mcpserver extra. With pip, add it yourself (it isn't an engine backendoptics setupmanages):pip install "optics-framework[mcp]".
[!IMPORTANT] Some extras need system tooling beyond the Python package. A driver extra installs only the Python client — mobile testing also needs the Appium server, a device/emulator, and platform tooling (Node.js, Android SDK/
adb, JDK).pytesseractneeds the Tesseract binary;playwrightneeds its browsers (playwright install). See the Installation guide.
[!WARNING] Conda is not supported for
easyocr+optics-frameworktogether (conflicting NumPy 1.x/2.x requirements). Use a standardvenv.
Quickstart
From a fresh install to a running test with one guided command:
pip install optics-framework
optics quickstart
optics quickstart asks what you want to automate, installs the matching engine, scaffolds the project, writes a platform-correct config.yaml, and runs an optics doctor check — then prints your next steps. Every prompt has a default, so pressing Enter throughout yields a runnable project.
Prefer to drive each step yourself?
optics init my_test_project --template contact
# point my_test_project/config.yaml at your device/app, start the Appium server, then:
optics dry_run my_test_project # validate keywords, elements and module refs — no device (but the config's engines must be installed)
optics execute my_test_project
--template scaffolds a working project from a bundled sample. Each needs the matching extras installed (see Install); the headline [appium,easyocr] above covers the first three:
contact·calendar·youtube— Appium/Android, easyocr →[appium,easyocr]clock— Appium/Android, image templates + Tesseract OCR →[appium,pytesseract](plus the Tesseract binary)gmail_web— Selenium →[selenium,easyocr]playwright— Playwright →[playwright](thenplaywright installfor the browsers)
Omit --template for an empty scaffold with a commented starter config.yaml. Need only a config for an existing folder? optics configure <folder> writes one from a few questions, and optics doctor [folder] [--check] verifies engines, tooling, and project config.
Why Optics
A locator ladder, not a locator. Every element-based keyword walks a priority-ordered chain until one strategy succeeds:
| # | Strategy | How it finds the element |
|---|---|---|
| 1 | XPathStrategy |
Native XPath query through the driver's accessibility tree |
| 2 | TextElementStrategy |
Direct text / CSS / class lookup through the element source |
| 3 | TextDetectionStrategy |
Screenshot → OCR (EasyOCR, Pytesseract, Google Vision, remote OCR) |
| 4 | ImageDetectionStrategy |
Screenshot → template matching against a reference PNG |
| 5 | AI self-heal (opt-in) | All four failed → an LLM reads the screen and recovers |
Cheap strategies run first, so vision only costs you time when the tree can't help. Steps 1–4 are LocatorStrategy registrations; step 5 is a separate recovery layer, bounded to five turns and a six-keyword allowlist so it re-enters the ladder rather than tapping blind coordinates. Two more fallback axes sit alongside: multiple values per element name, and multiple enabled drivers or element sources, each tried in config order.
Beyond the ladder:
- Tests are data — elements, modules and test cases as plain CSV or YAML. No IDE, no programming.
- Targets — Android, iOS, web (Selenium/Playwright), Android TV, Samsung Tizen, LG webOS.
- Non-intrusive — the
bledriver drives production devices as a Bluetooth HID mouse/keyboard where debugging and screenshots are blocked. Coordinate-only, so pair it withcamera_screenshotand the vision strategies. - Agent-ready —
optics mcpexposes every keyword as a typed MCP tool and device state as MCP resources.
Write a test as data
my_test_project/
├── config.yaml
├── test_cases/test_cases.csv
├── modules/modules.csv
└── test_data/
├── elements.csv
├── error_definitions.csv # optional
└── input_templates/*.png # optional, for image matching
Optics discovers these files by their content (CSV headers / YAML top-level keys), not their path — the folder names above are a convention, so elements.csv works equally under test_data/ or elements/ (the folder optics live's /save writes to).
test_data/elements.csv — names mapped to locators. Doubles as a general variable store; repeating a name builds a fallback list.
Element_Name,Element_ID
Add_Contact_Button,//android.widget.Button[@content-desc="Create contact"]
First_Name_element,//android.widget.EditText[@text="First name"]
Save_Button,Save
First_Name,John
A locator can be an XPath, text=…, css=…, a plain string, an image filename from input_templates/, or TEXT_ONLY:… to force a vision-based search.
modules/modules.csv — a reusable sequence of keywords; ${name} resolves against elements.csv.
module_name,module_step,param_1,param_2
Add Contact,Press Element,${Add_Contact_Button}
Add Contact,Enter Text,${First_Name_element},${First_Name}
Add Contact,Press Element,${Save_Button}
test_cases/test_cases.csv — modules sequenced into scenarios. A test case whose name contains suite + setup (or teardown) is hoisted to run around the whole suite.
test_case,test_step
Suite Setup,Launch Contact Application
Add Contact with Contact App,Add Contact
Add Contact with Contact App,Verify Contact is Added
Six ways to run the same keywords
| Surface | Command / import | Best for |
|---|---|---|
| CLI runner | optics execute <project> |
CI suites written as CSV/YAML |
| Interactive TUI | optics live [project] |
Building a test by doing it — recording is always on, Ctrl-N toggles natural-language mode |
| Python SDK | from optics_framework import Optics |
Custom logic, embedding in existing suites |
| Robot Framework | Library optics_framework.optics.Optics |
Teams already on Robot |
| REST API | optics serve |
Remote/orchestrated execution, live workspace streaming over SSE |
| MCP server | optics mcp |
Letting an AI agent drive a real device |
Every successful keyword is buffered as you work. To persist the buffer:
/save <test_case> <module_name>
That appends the recorded keywords to modules/modules.csv as <module_name>, adds a (<test_case>, <module_name>) row to test_cases/test_cases.csv, creates a header-only elements/elements.csv stub if none exists, and copies the session's screenshots to execution_output/<module_name>/. The buffer then clears, so the next actions become the next module. If either name already exists, re-run the identical /save to confirm the append.
Other commands: /device [id], /elements, /screenshot, /help, /quit. Full reference: Live Usage.
from optics_framework import Optics
optics = Optics()
optics.setup(
driver_sources=[{"appium": {"enabled": True, "url": "http://localhost:4723"}}],
elements_sources=[{"appium_find_element": {"enabled": True}}],
)
optics.launch_app(app_identifier="com.example.app")
optics.enter_text("username_field", "testuser")
optics.press_element("submit_button")
optics.validate_element("welcome_message")
optics.quit()
{ "mcpServers": { "optics": { "command": "optics", "args": ["mcp"] } } }
Then: start_session → observe (screenshot, optics://session/{id}/source) → act (press_element, enter_text, …) → terminate_session. For networked use: optics mcp --transport http --port 8090. Sessions are not shared with optics serve — each is its own process.
Keywords
Every public method on the four API classes is automatically a keyword, on every surface above. CSV/YAML uses Title Case (Press Element → press_element).
| Category | Keywords |
|---|---|
| Actions | Press Element · Press By Percentage · Press By Coordinates · Detect And Press · Select Dropdown Option · Swipe · Swipe By Percentage · Swipe From Element · Swipe Until Element Appears · Scroll · Scroll From Element · Scroll Until Element Appears · Enter Text · Enter Text Direct · Enter Text Using Keyboard · Enter Number · Clear Element Text · Press Keycode · Get Text · Sleep · Execute Script |
| Verification | Assert Presence · Assert Visibility · Assert Equality · Validate Element · Validate Screen · Is Element · Get Interactive Elements · Get Screen Elements · Capture Screenshot · Capture Pagesource |
| App lifecycle | Launch App · Launch Other App · Start Appium Session · Get Driver Session Id · Close And Terminate App · Force Terminate App · Get App Version |
| Flow control | Run Loop · Condition · Read Data · Evaluate · Date Evaluate · Invoke API |
Run optics list for the live catalogue with signatures, or read the Keyword Usage guide for parameters and examples. Location keywords accept percentage-based Area-of-Interest bounds (aoi_x/y/width/height, 0–100) to scope a vision search to part of the screen.
[!NOTE]
Press CheckboxandPress Radio Buttonstill resolve but are deprecated aliases ofPress Element— usePress Elementdirectly.Add APIis available on theOpticsPython class only, not to the CSV/YAML runner; define APIs in anapi.yamland call them withInvoke APIinstead.
Configure once, in config.yaml
Every section is a priority-ordered list and every entry has an enabled flag. Enable a second driver and it becomes a fallback. Prefer not to hand-write it? optics configure <folder> writes one from a few questions, or optics quickstart builds the whole project including the config.
driver_sources:
- appium:
enabled: true
url: "http://localhost:4723"
capabilities:
platformName: Android
automationName: UiAutomator2
deviceName: emulator-5554
appPackage: com.google.android.contacts
appActivity: com.android.contacts.activities.PeopleActivity
elements_sources:
- appium_find_element: { enabled: true }
- appium_page_source: { enabled: true }
- appium_screenshot: { enabled: true }
text_detection:
- easyocr: { enabled: true }
image_detection:
- templatematch: { enabled: false }
log_level: INFO
| Layer | Available engines |
|---|---|
| Drivers | appium (Android, iOS, Android TV, Tizen, webOS) · selenium · playwright · ble |
| Element sources | appium_find_element · appium_page_source · appium_web · appium_screenshot · selenium_* · playwright_* · camera_screenshot |
| Text detection | easyocr · pytesseract · google_vision · remote_ocr |
| Image detection | templatematch · remote_oir |
| LLM | gemini · openai (and any OpenAI-compatible server) · anthropic |
Enabling the LLM features
The llm_models block powers both natural-language mode in optics live (Ctrl-N) and AI self-heal. Pick the engine that matches the account you already have, install its extra, and enable it:
| You have | Install | Enable |
|---|---|---|
| A Gemini API key, or Vertex AI | optics-framework[gemini] |
gemini |
| An OpenAI key | optics-framework[openai] |
openai with a model |
| An OpenRouter key (any hosted model) | optics-framework[openai] |
openai with url: https://openrouter.ai/api/v1 |
| A local model (Ollama, vLLM, LM Studio) | optics-framework[openai] |
openai with the server's url |
| A Claude key, or Claude on Bedrock / Vertex | optics-framework[anthropic] |
anthropic with a model |
llm_models:
- openai:
enabled: true
url: https://openrouter.ai/api/v1 # omit for OpenAI itself
capabilities:
model: anthropic/claude-sonnet-5-5
api_key_env: OPENROUTER_API_KEY # default OPENAI_API_KEY
ai_self_heal: true # opt into the LLM backstop; default false
Each engine reads its key from capabilities.api_key, else from the environment (GEMINI_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY). For openai and anthropic, api_key_env names the variable to read in place of that default. For direct Anthropic clients, if the named variable is unset, the SDK can fall back to ANTHROPIC_API_KEY. Anthropic on Bedrock or Vertex uses the platform credentials, not a key. Write api_key: ${MY_KEY} to keep the key out of a committed config.yaml; logs show **** in its place. If MY_KEY is not set, the run stops with E0501 before it starts, so set it first. Enable more than one entry and Optics falls back to the next when a call fails. A misconfigured LLM degrades to "no self-heal" rather than a hard failure.
Full reference: Configuration. Adding your own engine is a file drop plus an interface — see Extending the Framework.
Results
An optics execute run writes to <project>/execution_output/:
junit_output.xml— written incrementally, so CI sees progress as it happenslogs.json— structured logs whenjson_log: true- screenshots — pre/post action frames, plus strategy-annotated and AOI overlays
detected_errors_<session_id>.json— on-screen error detection
Drop an error_definitions.csv into test_data/ and Optics scans visible text for crash dialogs, Session expired, network errors, and the like — no assertions required. Matches also land in the JUnit XML as a synthetic failing testcase, so CI fails a build on "the app crashed mid-test" the same way it fails a normal assertion. See Error Detection.
CLI reference
optics init Scaffold a new project (positional NAME; --template, --path, --force, --git-init)
optics quickstart Guided walkthrough: engines + project + config + doctor check
optics configure Write/edit a project's config.yaml (guided prompts; --edit opens a TUI)
optics doctor Diagnose engines, tooling and a project's config (--check for CI)
optics setup Install engine backends (--list, --install); bare command opens a TUI
optics dry_run Validate a project without touching a device
optics execute Run a project (--runner test_runner|pytest; --include/--exclude test cases)
optics live Interactive keyword session against a live target
optics generate Emit pytest or Robot Framework code from a project
optics list Print every discoverable keyword
optics serve Start the REST API server (--host, --port, --workers)
optics mcp Start the MCP server (--transport stdio|http)
optics completion Install shell autocompletion
optics --version Print the installed version
Details in the CLI guide.
Contributing
Contributions welcome — bug reports, docs, and code are all first-class.
- Bigger items are queued under
help wanted. - Found a bug or missing a keyword? Open an issue.
The contribution guide covers development setup, step-by-step recipes (adding a keyword or engine backend), commit conventions, and the pull-request format reviewers expect. Tooling details live in the Developer Guide; behaviour expectations in our Code of Conduct.
git clone git@github.com:mozarkai/optics-framework.git
cd optics-framework
pipx install poetry
poetry install --with dev,test,docs
poetry run pytest # tests + coverage
poetry run ruff check --fix . # lint
poetry run pre-commit run --all-files
poetry run mkdocs serve # docs preview
Commits follow Conventional Commits, enforced by commitizen in the commit-msg hook.
Security issues follow SECURITY.md rather than public issues.
License & support
Apache 2.0 — see LICENSE.
Questions and bugs: GitHub Issues.
Optics Framework MCP questions
What devices can Optics Framework automate?
It targets Android, iOS, web via Selenium or Playwright, Android TV, Samsung Tizen, LG webOS, and Bluetooth BLE devices.
How does Optics self-heal broken locators?
Each element is tried through a priority chain: XPath, then text, then OCR, then image matching, and finally an opt-in LLM recovery layer.
How do I connect an AI agent to Optics over MCP?
Run the optics mcp command, which exposes every keyword as a typed MCP tool and device state as MCP resources.
Do I write tests in code?
No, tests, modules and elements are plain CSV or YAML data files, though a Python SDK and Robot Framework library also exist.
What Python version does Optics require?
Optics needs Python 3.12 or newer, and Conda is not supported for easyocr with the framework.