openreality MCP server
Commit Oct 30, 2026Converts phone videos into AI-queryable 3D scenes for measurement, path planning, and robot training data exports.
Also known as reality-opened/openreality, openreality-mcp
- Stars
- 658
- Tools
- 42
- Downloads, 30 days
- —
- Latest version
- —
- Last commit
- Oct 30, 2026
- License
- BSD-2-Clause
What is openreality MCP?
openreality converts phone videos into AI-queryable 3D scenes for measurement, path planning, and robot training data exports. It exposes 42 tools, such as workspace_list_scenes, workspace_upload_video, workspace_upload_recording.
Summary generated from the project's README
About openreality MCP server
Open Reality provides a way to transform standard phone videos into interactive 3D scenes that AI agents can query and manipulate. This server acts as a bridge between visual data and action, allowing users to perform spatial measurements, plan navigation paths, and generate datasets for robotics. It is designed for developers working with AI assistants like Claude, Cursor, or Codex, providing 42 specialized tools to interact with reconstructed environments.
Users can connect via the MCP protocol by running the server locally using npx or by adding it to their desktop configuration. The system supports a hosted service or complete self-hosting on a personal GPU machine or a Modal account. While the server handles the heavy lifting of 3D reconstruction, the client remains on your machine to manage credentials and large files. For testing without a GPU or account, an offline simulator is available to mock the entire workflow using fixture data.
What you can do with it
- 1Spatial measurement and calibrationAsk your assistant to measure distances between objects after providing a real-world reference for scale.
- 2Autonomous path planningRequest a specific route through a scanned area to reach a target object.
- 3Robotics dataset generationConvert a captured 3D scan into specialized formats for training robot models.
- 4Scene interrogationUse agents to identify and describe the contents of a reconstructed room.
Written from the project's README. Every use case is checked against a quote from it.
How to install openreality MCP
Formats verified 10 Nov 2026Add to CursorAdd to GooseInstall openreality MCP in Claude Code
$ claude mcp add openreality -- npx -y openreality-mcpInstall openreality MCP in Cursor
~/.cursor/mcp.json
{
"mcpServers": {
"openreality": {
"command": "npx",
"args": [
"-y",
"openreality-mcp"
]
}
}
}Install openreality MCP in VS Code
.vscode/mcp.json
{
"servers": {
"openreality": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"openreality-mcp"
]
}
}
}Install openreality MCP in VS Code CLI
$ code --add-mcp '{"name":"openreality","type":"stdio","command":"npx","args":["-y","openreality-mcp"]}'Install openreality MCP in Claude Desktop
claude_desktop_config.json
{
"mcpServers": {
"openreality": {
"command": "npx",
"args": [
"-y",
"openreality-mcp"
]
}
}
}Install openreality MCP in Codex
$ codex mcp add openreality -- npx -y openreality-mcpInstall openreality MCP in Codex config.toml
~/.codex/config.toml
[mcp_servers.openreality] command = "npx" args = ["-y", "openreality-mcp"]
Install openreality MCP in Gemini CLI
$ gemini mcp add -s user openreality npx -- -y openreality-mcpInstall openreality MCP in Gemini settings.json
~/.gemini/settings.json
{
"mcpServers": {
"openreality": {
"command": "npx",
"args": [
"-y",
"openreality-mcp"
]
}
}
}Started by HostedMCP on 16 Nov 2026
We ran this command for openreality-mcp 0.4.0 in an isolated container with no network and no keys. It answered and listed 42 tools.
openreality MCP tools 42
Read from the server with tools/listworkspace_list_scenesRead-onlyList all persisted scenes in the signed-in Open Reality account (newest first). Also reports which scan_ids are NEW since this machine last listed — the account is the sync layer: scans made on the phone or the web appear here. Use scene_card next for anything you want context on.
workspace_upload_videoActionUpload a local video file (.mp4/.mov/.avi/.mkv/.webm, ≤1 GB) into the account and start a reconstruction job. Returns {job_id, scan_id} immediately (202); follow with workspace_job_wait. The scene appears in workspace_list_scenes when the job is done.
workspace_upload_recordingActionUpload a robot recording (DimOS memory2 .db, ≤2 GB) into the account. Runs the recordings pipeline (recon → odometry-consistency score → QC gate); the scene persists as source="robot_recording" with the QC report under derived/demo/recordings/. Returns {job_id, scan_id}; follow with workspace_job_wait.
workspace_upload_splatActionImport a local .ply/.spz Gaussian splat as a scene (source="imported_splat", honestly degraded report). ≤64 MiB uploads inline and returns {scan_id, gaussian_count} synchronously; larger files go through the chunked lane and return {job_id, scan_id} to poll with workspace_job_wait. Non-gaussian PLYs are refused (422 not_a_gaussian_splat).
workspace_job_statusRead-onlyOne status read of an ingest/export job (GET /api/workspace/jobs/<job_id>): {status: queued|running|done|error, stage?, scan_id?, error?}.
workspace_job_waitRead-onlyPoll an ingest/export job until it is done or errored (or the timeout passes). Returns the final status plus the stage trail observed.
scene_cardRead-onlyThe Tier-1 context card for a scene: source, metric/anchor state, room + summary, closed-world object inventory, ground frame, and workspace artifacts (agent runs, measurements, paths). Small by design — pull details with the other scene tools.
scene_list_objectsRead-onlyThe closed-world object inventory from the persisted record (facts.objects): model query, operator relabel, confidence, world center/extent. These uids/labels are the ONLY objects that exist — never invent others.
scene_measure_distanceRead-onlyDistance between two WORLD-frame points (POST /measure). The response units are "m" ONLY when the scene has a metric anchor — otherwise "relative". ALWAYS report the units and scale_source with the value; never say metres for relative units.
scene_measure_angleRead-onlyAngle at the vertex (second point) of three WORLD-frame points (POST /measure, kind=angle). Returns degrees.
scene_plan_pathRead-onlyPlan a path through scanned free space to an object (by uid) or a world point (POST /nav/plan). Provide exactly one of object_uid | point_world. Honest failures: 422 no_geometry / no_floor / unreachable_goal (with a nearest-reachable suggestion). The response provenance line ("not certified navigation") must ride along with any use. The per-frame camera flythrough (poses) is left on disk by default and the waypoint list comes back as an evenly-sampled preview — both counts and the full-trajectory artifact key ride along; artifact_fetch(doc_key) for the whole thing.
scene_planesRead-onlyFloor + wall plane candidates from the scene geometry (POST /planes; cached per-scene). Used for restyle scoping and understanding room structure.
| workspace_list_scenes | List all persisted scenes in the signed-in Open Reality account (newest first). Also reports which scan_ids are NEW since this machine last listed — the account is the sync layer: scans made on the phone or the web appear here. Use scene_card next for anything you want context on. | Read-only |
| workspace_upload_video | Upload a local video file (.mp4/.mov/.avi/.mkv/.webm, ≤1 GB) into the account and start a reconstruction job. Returns {job_id, scan_id} immediately (202); follow with workspace_job_wait. The scene appears in workspace_list_scenes when the job is done. | Action |
| workspace_upload_recording | Upload a robot recording (DimOS memory2 .db, ≤2 GB) into the account. Runs the recordings pipeline (recon → odometry-consistency score → QC gate); the scene persists as source="robot_recording" with the QC report under derived/demo/recordings/. Returns {job_id, scan_id}; follow with workspace_job_wait. | Action |
| workspace_upload_splat | Import a local .ply/.spz Gaussian splat as a scene (source="imported_splat", honestly degraded report). ≤64 MiB uploads inline and returns {scan_id, gaussian_count} synchronously; larger files go through the chunked lane and return {job_id, scan_id} to poll with workspace_job_wait. Non-gaussian PLYs are refused (422 not_a_gaussian_splat). | Action |
| workspace_job_status | One status read of an ingest/export job (GET /api/workspace/jobs/<job_id>): {status: queued|running|done|error, stage?, scan_id?, error?}. | Read-only |
| workspace_job_wait | Poll an ingest/export job until it is done or errored (or the timeout passes). Returns the final status plus the stage trail observed. | Read-only |
| scene_card | The Tier-1 context card for a scene: source, metric/anchor state, room + summary, closed-world object inventory, ground frame, and workspace artifacts (agent runs, measurements, paths). Small by design — pull details with the other scene tools. | Read-only |
| scene_list_objects | The closed-world object inventory from the persisted record (facts.objects): model query, operator relabel, confidence, world center/extent. These uids/labels are the ONLY objects that exist — never invent others. | Read-only |
| scene_measure_distance | Distance between two WORLD-frame points (POST /measure). The response units are "m" ONLY when the scene has a metric anchor — otherwise "relative". ALWAYS report the units and scale_source with the value; never say metres for relative units. | Read-only |
| scene_measure_angle | Angle at the vertex (second point) of three WORLD-frame points (POST /measure, kind=angle). Returns degrees. | Read-only |
| scene_plan_path | Plan a path through scanned free space to an object (by uid) or a world point (POST /nav/plan). Provide exactly one of object_uid | point_world. Honest failures: 422 no_geometry / no_floor / unreachable_goal (with a nearest-reachable suggestion). The response provenance line ("not certified navigation") must ride along with any use. The per-frame camera flythrough (poses) is left on disk by default and the waypoint list comes back as an evenly-sampled preview — both counts and the full-trajectory artifact key ride along; artifact_fetch(doc_key) for the whole thing. | Read-only |
| scene_planes | Floor + wall plane candidates from the scene geometry (POST /planes; cached per-scene). Used for restyle scoping and understanding room structure. | Read-only |
Show all 42 toolsShow fewer tools
scene_ground_frameRead-onlyRead the recorded floor/ceiling frame (GET /ground_frame). If vertical_axis_known is false, the numbers are ABSENT on purpose — say so rather than guessing heights.
scene_ground_frame_fit(Re)fit the floor plane and write the result onto the scene metrics (POST /ground_frame). dry_run=true computes without persisting.
scene_anchorActionMetric-anchor calibration (POST /anchor): two picked world points + their real distance in metres. Non-destructive (writes NEW derived/anchor/* artifacts + the derived_latest pointer) and it is what unlocks units:"m" for measure/nav/export. materialize:"job" returns 202 with a job_id to poll via workspace_job_wait.
scene_keyframe_imageRead-onlyOne capture keyframe as an image (GET /keyframes/<blob_key>). blob_key comes from the scene detail keyframes[] refs. Real capture photos — cite as photo evidence.
scene_synthetic_viewsRead-onlyRenders of the scene's own splat registered as evidence (GET /synthetic_views). These are NOT photographs — always carry the "synthetic view" provenance when citing.
scene_synthetic_view_imageRead-onlyOne synthetic view PNG (a RENDER of the splat, not a photo — say so when citing).
scene_lodRead-onlyLevel-of-detail index status for the scene splat (GET /lod). Always 200 — read `status` (ready|none|running|error|not_needed).
scene_lod_buildActionSpawn the LOD decimation job (POST /lod → 202 {job_id}; 409 lod_job_active when one is already running). Poll with workspace_job_wait.
scene_imported_objectsRead-onlyStatus of geometry-derived objects for an imported splat (GET /imported_objects).
scene_imported_objects_runActionCluster an imported splat's geometry into facts.objects (POST /imported_objects). Labels carry geometry/synthetic-view provenance — confidence 0.0 is honest, not a bug.
scene_object_completeActionSAM-3D-Objects mesh completion for a segmented object (POST /objects/<uid>/complete → 202 {job_id}, broker-thread lane). Poll with scene_job_wait. Output is GENERATED geometry (honesty envelope in the artifact meta.json).
scene_object_variantsActionTRELLIS variant generation for a completed object (POST /objects/<uid>/variants → 202 {job_id}). Optional body fields pass through (see server docs). GENERATED assets.
scene_segmentActionSAM-3 click segmentation (POST /segment). Advanced: the body must match the server contract (click point + view selection — see routes_sam3d.py / the OS UI). Provided as a raw passthrough so the full surface stays reachable.
scene_job_statusRead-onlyOne status read of a broker-thread scene job (SAM-3D completion, variants, view renders). NOTE: these jobs honestly 404 across a broker restart.
scene_job_waitRead-onlyPoll a broker-thread scene job until done/error/timeout.
scene_share_accessRead-onlyOwner-only read of the share-link access log (GET /share/access): per minted link (access_id) — opened, request count, visits (new visit after 30 min idle), first/last seen, returned (a visit >= 24 h after the first), devices, and the Q&A questions the viewer asked. This is the behavioural evidence for customer discovery (L5 = returned). Match access_id to the person you sent the link to. events:true appends the raw event list.
scene_shareActionMint a read-only, single-scene share token + embed URL (POST /share; default TTL 30 days). Share tokens can NEVER reach the OS feature routes — viewer only.
agent_annotateActionStart the five-phase scene annotation run (survey → labels → description → dimensions → key features) on the server-side scene agent. 202 {run_id}; one active run per scene (409 agent_run_active carries active_run_id — or set attach_if_active). Costs bounded server-side LLM calls. Follow with agent_run_events wait_for_done.
agent_replayActionRe-serve a recorded run's persisted event log as a new (replay-badged, $0) run. Use for reviewing what an earlier run found without spending LLM calls.
agent_pilotActionGive the pilot agent a free-text instruction (navigate/inspect flows; tools: list_scene_objects, plan_path, measure_distance). 202 {run_id}; poll agent_run_events.
agent_chatActionOne chat turn with the scene agent (its own bounded run; the server agent may call its scene tools mid-turn). By default waits for the run to finish and returns the event transcript. The agent's numbers carry units/scale_source — repeat them honestly.
agent_run_eventsRead-onlyRead a run's event log from a cursor (GET /runs/<run_id>/events?after=). With wait_for_done, polls at ~1 Hz (the web client's cadence) until the run finishes.
agent_runsRead-onlyThe scene's run index (newest first) + which run is live: kind, status, cost_usd, llm_calls, findings. Persisted runs can be replayed for $0 with agent_replay.
export_manifestRead-onlyWhat an export WOULD contain (GET /demo/export/manifest): file tree + sizes, complete/absent components, and whether the zip route would work. No build, no writes. Formats: openreality | groot_lerobot_v2 | isaac_usd | mujoco_mjcf (the simulator formats isaac_usd / mujoco_mjcf are metric-gated — they need an anchor).
export_prepareActionBuild the export zip as a background job (POST /demo/export/prepare → 202 {job_id}; 409 export_job_active when one is running). With wait=true (default) polls the job and returns the prepared artifact (including download_path). Then artifact_fetch it. For MuJoCo prefer export_mujoco, which also downloads and unpacks the scene.
export_statusRead-onlyWhat is prepared for scene+format (GET /demo/export/prepared). Always 200 with {status: ready|running|error|none}; ready carries the artifact + download_path.
export_mujocoActionOne call from scene to a loadable MuJoCo model: builds (or reuses a fresh) mujoco_mjcf export, downloads it and unpacks it to disk, and returns the path to scene.xml — load it with mujoco.MjModel.from_xml_path(path) or `python -m mujoco.viewer --mjcf=<path>` (needs mujoco>=3.3). METRIC-GATED: an un-anchored scene is refused with 409 metric_scale_required — run scene_anchor first. Collision is primitives only (a floor plane + one upright box per detected object); the scan mesh is visual-only. A fresh build meshes the scene on a 64 GB container and takes minutes. Bytes never enter context.
artifact_fetchRead-onlyDownload any derived/... artifact (export zips, LOD levels, agent run logs, QC reports, masks) to a local file and return the path — bytes NEVER go into context. Default target: <artifacts_dir>/<scan_id>/<key>.
artifact_fetch_splatRead-onlyDownload the scene's original splat.ply to a local file and return the path. For a render-sized variant prefer the LOD levels (scene_lod → artifact_fetch of a level key).
artifact_fetch_cloudRead-onlyDownload the scene's cloud.ply to a local file and return the path.
| scene_ground_frame | Read the recorded floor/ceiling frame (GET /ground_frame). If vertical_axis_known is false, the numbers are ABSENT on purpose — say so rather than guessing heights. | Read-only |
| scene_ground_frame_fit | (Re)fit the floor plane and write the result onto the scene metrics (POST /ground_frame). dry_run=true computes without persisting. | |
| scene_anchor | Metric-anchor calibration (POST /anchor): two picked world points + their real distance in metres. Non-destructive (writes NEW derived/anchor/* artifacts + the derived_latest pointer) and it is what unlocks units:"m" for measure/nav/export. materialize:"job" returns 202 with a job_id to poll via workspace_job_wait. | Action |
| scene_keyframe_image | One capture keyframe as an image (GET /keyframes/<blob_key>). blob_key comes from the scene detail keyframes[] refs. Real capture photos — cite as photo evidence. | Read-only |
| scene_synthetic_views | Renders of the scene's own splat registered as evidence (GET /synthetic_views). These are NOT photographs — always carry the "synthetic view" provenance when citing. | Read-only |
| scene_synthetic_view_image | One synthetic view PNG (a RENDER of the splat, not a photo — say so when citing). | Read-only |
| scene_lod | Level-of-detail index status for the scene splat (GET /lod). Always 200 — read `status` (ready|none|running|error|not_needed). | Read-only |
| scene_lod_build | Spawn the LOD decimation job (POST /lod → 202 {job_id}; 409 lod_job_active when one is already running). Poll with workspace_job_wait. | Action |
| scene_imported_objects | Status of geometry-derived objects for an imported splat (GET /imported_objects). | Read-only |
| scene_imported_objects_run | Cluster an imported splat's geometry into facts.objects (POST /imported_objects). Labels carry geometry/synthetic-view provenance — confidence 0.0 is honest, not a bug. | Action |
| scene_object_complete | SAM-3D-Objects mesh completion for a segmented object (POST /objects/<uid>/complete → 202 {job_id}, broker-thread lane). Poll with scene_job_wait. Output is GENERATED geometry (honesty envelope in the artifact meta.json). | Action |
| scene_object_variants | TRELLIS variant generation for a completed object (POST /objects/<uid>/variants → 202 {job_id}). Optional body fields pass through (see server docs). GENERATED assets. | Action |
| scene_segment | SAM-3 click segmentation (POST /segment). Advanced: the body must match the server contract (click point + view selection — see routes_sam3d.py / the OS UI). Provided as a raw passthrough so the full surface stays reachable. | Action |
| scene_job_status | One status read of a broker-thread scene job (SAM-3D completion, variants, view renders). NOTE: these jobs honestly 404 across a broker restart. | Read-only |
| scene_job_wait | Poll a broker-thread scene job until done/error/timeout. | Read-only |
| scene_share_access | Owner-only read of the share-link access log (GET /share/access): per minted link (access_id) — opened, request count, visits (new visit after 30 min idle), first/last seen, returned (a visit >= 24 h after the first), devices, and the Q&A questions the viewer asked. This is the behavioural evidence for customer discovery (L5 = returned). Match access_id to the person you sent the link to. events:true appends the raw event list. | Read-only |
| scene_share | Mint a read-only, single-scene share token + embed URL (POST /share; default TTL 30 days). Share tokens can NEVER reach the OS feature routes — viewer only. | Action |
| agent_annotate | Start the five-phase scene annotation run (survey → labels → description → dimensions → key features) on the server-side scene agent. 202 {run_id}; one active run per scene (409 agent_run_active carries active_run_id — or set attach_if_active). Costs bounded server-side LLM calls. Follow with agent_run_events wait_for_done. | Action |
| agent_replay | Re-serve a recorded run's persisted event log as a new (replay-badged, $0) run. Use for reviewing what an earlier run found without spending LLM calls. | Action |
| agent_pilot | Give the pilot agent a free-text instruction (navigate/inspect flows; tools: list_scene_objects, plan_path, measure_distance). 202 {run_id}; poll agent_run_events. | Action |
| agent_chat | One chat turn with the scene agent (its own bounded run; the server agent may call its scene tools mid-turn). By default waits for the run to finish and returns the event transcript. The agent's numbers carry units/scale_source — repeat them honestly. | Action |
| agent_run_events | Read a run's event log from a cursor (GET /runs/<run_id>/events?after=). With wait_for_done, polls at ~1 Hz (the web client's cadence) until the run finishes. | Read-only |
| agent_runs | The scene's run index (newest first) + which run is live: kind, status, cost_usd, llm_calls, findings. Persisted runs can be replayed for $0 with agent_replay. | Read-only |
| export_manifest | What an export WOULD contain (GET /demo/export/manifest): file tree + sizes, complete/absent components, and whether the zip route would work. No build, no writes. Formats: openreality | groot_lerobot_v2 | isaac_usd | mujoco_mjcf (the simulator formats isaac_usd / mujoco_mjcf are metric-gated — they need an anchor). | Read-only |
| export_prepare | Build the export zip as a background job (POST /demo/export/prepare → 202 {job_id}; 409 export_job_active when one is running). With wait=true (default) polls the job and returns the prepared artifact (including download_path). Then artifact_fetch it. For MuJoCo prefer export_mujoco, which also downloads and unpacks the scene. | Action |
| export_status | What is prepared for scene+format (GET /demo/export/prepared). Always 200 with {status: ready|running|error|none}; ready carries the artifact + download_path. | Read-only |
| export_mujoco | One call from scene to a loadable MuJoCo model: builds (or reuses a fresh) mujoco_mjcf export, downloads it and unpacks it to disk, and returns the path to scene.xml — load it with mujoco.MjModel.from_xml_path(path) or `python -m mujoco.viewer --mjcf=<path>` (needs mujoco>=3.3). METRIC-GATED: an un-anchored scene is refused with 409 metric_scale_required — run scene_anchor first. Collision is primitives only (a floor plane + one upright box per detected object); the scan mesh is visual-only. A fresh build meshes the scene on a 64 GB container and takes minutes. Bytes never enter context. | Action |
| artifact_fetch | Download any derived/... artifact (export zips, LOD levels, agent run logs, QC reports, masks) to a local file and return the path — bytes NEVER go into context. Default target: <artifacts_dir>/<scan_id>/<key>. | Read-only |
| artifact_fetch_splat | Download the scene's original splat.ply to a local file and return the path. For a render-sized variant prefer the LOD levels (scene_lod → artifact_fetch of a level key). | Read-only |
| artifact_fetch_cloud | Download the scene's cloud.ply to a local file and return the path. | Read-only |
Automated checks
Ran Nov 16, 2026Full report →Not verified yet. A server needs a score of 80, no serious finding, and enough data for us to check. How checks work · What to fix for more points
- Source repository linked and reachablereality-opened/openreality is public and reachablegithub.com/reality-opened/openreality ↗Pass
- Package names this repository as its sourceopenreality-mcp names reality-opened/openreality as sourcenpmjs.com/package/openreality-mcp ↗Pass
- Build provenance (Sigstore attestation)No build attestation published for this releaseNot checked
- Open source license (OSI)BSD-2-Clausespdx.org/licenses/BSD-2-Clause.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-10-30github.com/reality-opened/openreality ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNo popular package one edit away from openreality-mcpPass
- No critical advisory in the package or its dependenciesNot checked
- No hidden instructions to the model42 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNo tool named like delete, drop or execPass
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubUtilities
| 🎥 Video in, 3D scene out | Upload a phone video. A few minutes later you have a persistent 3D scene. |
| 📏 Measurement | Distances and angles between any points. Numbers are only called metres after you calibrate with real distance; otherwise relative. |
| 🧭 Path planning | Plan a route through the scanned free space to an object or a point. |
| 🤖 Robot-training exports | Turns a scan into LeRobot / GR00T style datasets or an Isaac Sim or MuJoCo scene (hosted service). |
| 🕵️ Scene agents | Server-side agents that survey, label, and answer questions about a scene. |
| 🛠️ 42 tools for your AI | Everything is exposed through MCP. |
| 🧪 Offline simulator | A mock backend fakes the entire workflow with fixture data, so you can develop and demo with no account and no GPU. |
| 🏠 Self-hostable | The full server runs on your own GPU box or your own Modal account, no account with us needed. |
Quickstart
Add the tools to your AI assistant:
claude mcp add openreality -- npx -y openreality-mcp serve
codex mcp add openreality -- npx -y openreality-mcp serve
Add this to claude_desktop_config.json (Settings → Developer → Edit Config)
or ~/.cursor/mcp.json:
{
"mcpServers": {
"openreality": {
"command": "npx",
"args": ["-y", "openreality-mcp", "serve"]
}
}
}
Claude Code plugin (the MCP server plus the workflow skill in one step):
/plugin marketplace add reality-opened/openreality
/plugin install openreality@openreality
Claude desktop extension: download openreality-mcp-<version>.mcpb from the
latest release
and double-click it, or drag it into the Claude desktop window. No terminal needed.
Then sign in once (opens your browser, stores a revocable API key on your machine):
npx -y openreality-mcp login
Scan a room with your phone at open-reality.io, or just ask your assistant to upload a video file. Full per-client setup: open-reality.io/mcp.
Examples
Once connected, talk to your assistant like this:
"Upload
~/Videos/kitchen.mp4and reconstruct it."
"What objects are in my latest scan, and how big is the room?"
"The counter edge to the window is 2.4 m. Calibrate the scene, then measure the couch."
"Plan a path from the door to the desk and describe it."
"Export this scan as robot-training data and save the zip locally."
Self Hosting
The whole workflow is self-hostable. Read
server/docs/self-hosting.md; the short version:
git clone https://github.com/reality-opened/openreality
cd openreality/server
# Path A: your own GPU box (one process, local disk)
python -m server.selfhost --data-dir ~/openreality-data
# Path B: your own Modal account (CPU web server + GPU worker)
modal deploy modal_selfhost.py
Self-hosted servers need no account: a single token printed at first start is
your login, and the MCP client connects with OPENREALITY_URL plus that token.
[!IMPORTANT] Licensing. This repo is BSD-2-Clause, but the 3D reconstruction model a self-hosted server downloads (VGGT-1B and the VGGT code it runs on) is licensed by its owners as CC BY-NC 4.0, non-commercial use only. Nothing here redistributes it; your server fetches it from the source under their terms. For commercial use, use the hosted service (which runs a commercially licensed model) or get your own license from the model owners.
Repo index
| Directory | What it is | Ships as |
|---|---|---|
mcp/ |
The MCP server: 42 tools, scene resources, the offline simulator, and a full test suite. Developed here directly. | npm openreality-mcp |
server/ |
The backend: turns videos into persistent scenes and serves measurement, planning, agents, and exports over a plain REST API. | source (public mirror) |
core/ |
The 3D reconstruction library: camera tracking and dense geometry from ordinary video (the VGGT-SLAM 2.0 line), plus metric calibration, object detection, and splat export. | source (public mirror) |
server/ and core/ are curated mirrors of our private working repos, synced
by hand; each carries a MIRROR.md that says exactly what is included and how
it is synced. mcp/ is developed in this repo directly.
Technical details
flowchart LR
A["📱 phone video"] --> B["server: 3D reconstruction<br/>(GPU job)"]
B --> C[("persistent scene:<br/>geometry + objects + report")]
C --> D["41 MCP tools"]
D --> E["🤖 Claude / Codex / Cursor"]
E -->|"measure · navigate · export"| D
The MCP process always runs on your machine and holds your credentials; every tool call is a typed REST call to a server (ours or yours). Big files are written to your disk, never pasted into the AI's context. Server refusals and uncertainty labels reach the AI unedited, so it cannot pretend a relative number is metres.
The measurement rule is explained in Why our 3D tools refuse to say "metres" until you prove it.
🙏 Acknowledgements
Open Reality stands on other people's research and code. Thank you to:
- VGGT-SLAM by Dominic Maggio, Hyungtae Lim and Luca Carlone (MIT SPARK Lab). The reconstruction core in
core/is the VGGT-SLAM 2.0 line, and the scan demo above is theirs. - VGGT (Meta AI), the feed-forward 3D backbone a self-hosted server fetches under its own licence (see the note above).
- Depth Anything V2, the metric-depth model behind our scale-drift correction between map segments.
- gsplat from the Nerfstudio project, the optional GPU refinement in our splat export.
- SAM 3 (Meta AI) for segmentation, and SAM 3D Objects and TRELLIS (Microsoft) for object completion and variants on the hosted service.
- LeRobot (Hugging Face), NVIDIA GR00T and Isaac Sim, whose dataset and scene formats our robot-training exports target.
- GTSAM for factor-graph optimization, cordis for the plugin runtime, and the MCP TypeScript SDK.
🤝 Contributing
Issues and pull requests are welcome on any component. Changes to mcp/ land
here directly; fixes to server/ and core/ are folded back into the private
working repos and re-synced out. If you self-host and something breaks, an
issue with your logs is a gift: the self-host paths are young.
License
BSD-2-Clause for everything in this repository. Third-party models
are fetched from their owners under their own licenses (see the licensing note
above and server/docs/self-hosting.md).
openreality MCP questions
How do I add openreality MCP to Claude Code?
Run claude mcp add openreality -- npx -y openreality-mcp in your terminal. Add --scope user to make it available in every project.
How do I add openreality MCP to Cursor?
Open ~/.cursor/mcp.json and add the server under mcpServers, using the Cursor config shown in the install section above.
What tools does openreality MCP provide?
It exposes 42 tools, including workspace_list_scenes, workspace_upload_video, workspace_upload_recording, workspace_upload_splat, workspace_job_status.
How do I get started with Open Reality?
You can add the tools to your AI assistant using npx commands for Claude or Codex, or install the Claude desktop extension directly.
Can I run this server on my own hardware?
Yes, the entire workflow is self-hostable on your own GPU box or via a Modal account.
What kind of data can I export from a scan?
You can export scans as LeRobot or GR00T style datasets, or as Isaac Sim or MuJoCo scenes.