Point and Yap MCP server
Commit Oct 21, 2026Allows coding agents to receive voice instructions, cursor positions, and cropped screenshots via MCP to automate UI fixes.
Also known as mfmaur/pointandyap
- Stars
- 6
- Tools
- 5
- Downloads, 30 days
- —
- Latest version
- —
- Last commit
- Oct 21, 2026
- License
- MIT
What is Point and Yap MCP?
Point and Yap allows coding agents to receive voice instructions, cursor positions, and cropped screenshots via MCP to automate UI fixes. It exposes 5 tools, such as get_open_items, start_items, list_items.
Summary generated from the project's README
About Point and Yap MCP server
Automating UI fixes by showing coding agents exactly what to change is what Point and Yap enables for macOS users. By pressing Control twice, you can record voice instructions while pointing at specific screen elements. The application captures your speech, the element under your cursor, and a cropped screenshot to create a task for your AI.
This local MCP server runs over stdio and integrates with Claude Code, Codex, and Cursor. It uses Apple's on-device speech models to transcribe your words, ensuring that recordings, screenshots, and the task queue remain on your Mac. The system provides specific tools like get_open_items for agents to retrieve tasks and mark_done to signal completion.
Developers using Apple silicon and macOS 26 or later can manage these tasks through a local queue. The server handles the handoff between your visual input and the coding agent, allowing tools to identify UI components via accessibility data and resolve issues based on your verbal descriptions.
What you can do with it
- 1Fixing UI elements via voiceA developer can point at a broken button and say what is wrong to have an agent fix it.
- 2Automating visual bug reportsYou can send cropped screenshots and cursor positions directly to an AI client to describe layout issues.
Written from the project's README. Every use case is checked against a quote from it.
Point and Yap MCP tools 5
Detected from the READMEget_open_itemsstart_itemslist_itemsmark_donerelease_items
Configuration
POINTANDYAP_AGENT
Automated checks
Ran Nov 14, 2026Full report →Not verified yet. A server needs a score of 80, no serious finding, and enough data for us to check. How checks work · What to fix for more points
- Source repository linked and reachablemfmaur/pointandyap is public and reachablegithub.com/mfmaur/pointandyap ↗Pass
- Package names this repository as its sourceNot checked
- Build provenance (Sigstore attestation)Not checked
- Open source license (OSI)MITspdx.org/licenses/MIT.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-10-21github.com/mfmaur/pointandyap ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNot checked
- No critical advisory in the package or its dependenciesNot checked
- No hidden instructions to the model5 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNo tool named like delete, drop or execPass
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubPoint and Yap is a small Mac app for showing your coding agent what to change. Press Control twice, talk while you point at things on screen, press Control twice again. Each recording becomes an item in a local queue, with what you said, the element under the cursor and a cropped screenshot. Claude Code, Codex or Cursor pick it up over MCP, fix it and mark it done.
Install
Requires macOS 26 or later and a Mac with Apple silicon.
curl -fsSL https://pointandyap.com/install.sh | sh
or with Homebrew (trust the tap once, then install):
brew trust mfmaur/tap
brew install --cask mfmaur/tap/point-and-yap
or by hand: download PointAndYap.dmg from pointandyap.com or the
latest release and drag Point and Yap into Applications. Every
build is signed and notarized by Apple; the install script checks the DMG with spctl before it copies anything.
Then open it, allow the three permissions it asks for, and answer one question: it lists the coding agents it found and asks before connecting them. Connect, or Skip and do it later in Settings.
Permissions, and why
- Microphone: records what you say while you point.
- Screen Recording: takes the screenshots at each point and click. The recording pill is never in them.
- Accessibility: reads the element under the cursor (role, label, window, app), so an agent can find it in code.
How it works
- Press Control twice (or Start recording). A small pill with the time and Stop stays on top.
- Talk and point: rest the cursor on the thing you mean while you say "this", "here", "that one". Clicks count too.
- Press Control twice again. A soft sound plays and two lines go on the clipboard for any agent: what is new and how to get it.
- The agent calls
get_open_items, makes the change and callsmark_done. The window shows To fix, In progress (by which agent) or Done for each recording.
Esc while recording cancels it and moves it to the Trash.
Runs on your Mac
Speech is transcribed by Apple's on-device model. Recordings, screenshots, the queue and the MCP server all live on your
Mac, in ~/PointAndYap/sessions. No account, no server, no analytics. The app itself only goes online to check
pointandyap.com for updates (and macOS fetches Apple's speech model once).
Your coding agent is a different matter: when it reads an item, what you said and the crops go to whatever model it uses, like anything else you show it.
MCP tools
"/Applications/Point and Yap.app/Contents/MacOS/PointAndYap" --mcp is a local MCP server over stdio, named pointandyap. After you say
Connect, it is added to Claude Code (user scope), Codex (~/.codex/config.toml) and Cursor (~/.cursor/mcp.json), and a
skill is linked into ~/.claude/skills and ~/.agents/skills. Each config is backed up first to
~/Library/Application Support/PointAndYap/backups/. An existing entry named pointandyap that is not ours is left alone.
| Tool | What it does |
|---|---|
get_open_items |
The open items, oldest first, with what was said, the elements and the crops. Marks them in progress by the calling agent. |
start_items |
Marks specific open items in progress by you. |
list_items |
Every item with its state: open, in progress (by whom, since when) or done (with the note). |
mark_done |
Marks items done with a one-line note. Done items are never handed out again. |
release_items |
Gives items back to the queue, with a note why. |
list_sessions, get_session, get_frame |
The recordings themselves: the aligned timeline, crops and full screenshots. |
An item in progress that no one finishes goes back to the queue after two hours. One file lock keeps two agents from taking the same item.
Shell fallback
For an agent without MCP:
"/Applications/Point and Yap.app/Contents/MacOS/PointAndYap" --items
POINTANDYAP_AGENT=my-agent "/Applications/Point and Yap.app/Contents/MacOS/PointAndYap" --done <id> --note "what changed"
FAQ
Does anything leave my Mac? Point and Yap sends nothing anywhere; it only checks for updates. Your coding agent sends what it reads (your words and the crops) to its model, as with anything you give it. See Runs on your Mac.
Does pointandyap.com track me? It counts page views, downloads, installer runs and the app's update checks (with the app version and country), to know how many people come and use it. No cookies, no tracking script, no IP address kept; a visitor is a hash that changes every day.
Which agents work? Claude Code, Codex and Cursor are connected for you. Any MCP client that runs a local stdio server
works with the command above, and any agent with a shell can use --items and --done.
I skipped connecting. How do I connect later? Settings, Coding agents, Connect now.
How do I remove it? Quit it and delete it from Applications. To remove the agent entries: claude mcp remove pointandyap --scope user, the [mcp_servers.pointandyap] table in ~/.codex/config.toml, pointandyap in ~/.cursor/mcp.json, and the point-and-yap
links in ~/.claude/skills and ~/.agents/skills. Recordings are in ~/PointAndYap.
How do updates work? From 0.1.2 on, the app checks pointandyap.com every few hours, downloads a new version in the background (signed, checked before it is installed) and installs it when you quit; the window shows it with a Restart button. Settings has Check for updates.
Does it work on older Macs or Intel? Not yet. Only speech-to-text needs macOS 26 (SpeechAnalyzer, Apple's on-device model with word timings, which lines up each word with where the cursor was). Wider support is planned.
Can I build it myself? Yes: docs/DEVELOPMENT.md. swift test runs the tests over a synthetic
recording.
Contributing
Issues and pull requests are welcome: CONTRIBUTING.md. Report security issues privately as described in SECURITY.md. This project follows a code of conduct.
License
MIT. Provided as is, without warranty of any kind.
I made Point and Yap because it helps me build. Maybe it helps you too.
Point and Yap MCP questions
What tools does Point and Yap MCP provide?
It exposes 5 tools, including get_open_items, start_items, list_items, mark_done, release_items.
Does anything leave my Mac?
No, the app sends nothing anywhere except for update checks. Your coding agent will send the transcribed words and screenshots to its respective model.
Which coding agents are supported?
Claude Code, Codex, and Cursor are connected automatically, but any MCP client running a local stdio server can use it.
Does it work on Intel-based Macs?
Currently, it only works on Apple silicon Macs running macOS 26 or later due to requirements for Apple's on-device speech-to-text model.
How do I connect the agent if I skipped the initial setup?
You can reconnect the agents later through the Settings menu under Coding agents.