# openreality MCP server

> Converts phone videos into AI-queryable 3D scenes for measurement, path planning, and robot training data exports.

- Repository: https://github.com/reality-opened/openreality
- Page: https://hostedmcp.org/servers/reality-opened/openreality
- Transport: stdio
- Stars: 658
- License: BSD-2-Clause

## Install

### Claude Code

```sh
claude mcp add openreality -- npx -y openreality-mcp
```

### Cursor (`~/.cursor/mcp.json`)

```json
{
  "mcpServers": {
    "openreality": {
      "command": "npx",
      "args": [
        "-y",
        "openreality-mcp"
      ]
    }
  }
}
```

### VS Code (`.vscode/mcp.json`)

```json
{
  "servers": {
    "openreality": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "openreality-mcp"
      ]
    }
  }
}
```

### VS Code CLI

```sh
code --add-mcp '{"name":"openreality","type":"stdio","command":"npx","args":["-y","openreality-mcp"]}'
```

### Claude Desktop (`claude_desktop_config.json`)

```json
{
  "mcpServers": {
    "openreality": {
      "command": "npx",
      "args": [
        "-y",
        "openreality-mcp"
      ]
    }
  }
}
```

### Codex

```sh
codex mcp add openreality -- npx -y openreality-mcp
```

### Codex config.toml (`~/.codex/config.toml`)

```toml
[mcp_servers.openreality]
command = "npx"
args = ["-y", "openreality-mcp"]
```

### Gemini CLI

```sh
gemini mcp add -s user openreality npx -- -y openreality-mcp
```

### Gemini settings.json (`~/.gemini/settings.json`)

```json
{
  "mcpServers": {
    "openreality": {
      "command": "npx",
      "args": [
        "-y",
        "openreality-mcp"
      ]
    }
  }
}
```

## Tools (42)

- `workspace_list_scenes`: List all persisted scenes in the signed-in Open Reality account (newest first). Also reports which scan_ids are NEW since this machine last listed — the account is the sync layer: scans made on the phone or the web appear here. Use scene_card next for anything you want context on. (read-only)
- `workspace_upload_video`: Upload a local video file (.mp4/.mov/.avi/.mkv/.webm, ≤1 GB) into the account and start a reconstruction job. Returns {job_id, scan_id} immediately (202); follow with workspace_job_wait. The scene appears in workspace_list_scenes when the job is done.
- `workspace_upload_recording`: Upload a robot recording (DimOS memory2 .db, ≤2 GB) into the account. Runs the recordings pipeline (recon → odometry-consistency score → QC gate); the scene persists as source="robot_recording" with the QC report under derived/demo/recordings/. Returns {job_id, scan_id}; follow with workspace_job_wait.
- `workspace_upload_splat`: Import a local .ply/.spz Gaussian splat as a scene (source="imported_splat", honestly degraded report). ≤64 MiB uploads inline and returns {scan_id, gaussian_count} synchronously; larger files go through the chunked lane and return {job_id, scan_id} to poll with workspace_job_wait. Non-gaussian PLYs are refused (422 not_a_gaussian_splat).
- `workspace_job_status`: One status read of an ingest/export job (GET /api/workspace/jobs/<job_id>): {status: queued|running|done|error, stage?, scan_id?, error?}. (read-only)
- `workspace_job_wait`: Poll an ingest/export job until it is done or errored (or the timeout passes). Returns the final status plus the stage trail observed. (read-only)
- `scene_card`: The Tier-1 context card for a scene: source, metric/anchor state, room + summary, closed-world object inventory, ground frame, and workspace artifacts (agent runs, measurements, paths). Small by design — pull details with the other scene tools. (read-only)
- `scene_list_objects`: The closed-world object inventory from the persisted record (facts.objects): model query, operator relabel, confidence, world center/extent. These uids/labels are the ONLY objects that exist — never invent others. (read-only)
- `scene_measure_distance`: Distance between two WORLD-frame points (POST /measure). The response units are "m" ONLY when the scene has a metric anchor — otherwise "relative". ALWAYS report the units and scale_source with the value; never say metres for relative units. (read-only)
- `scene_measure_angle`: Angle at the vertex (second point) of three WORLD-frame points (POST /measure, kind=angle). Returns degrees. (read-only)
- `scene_plan_path`: Plan a path through scanned free space to an object (by uid) or a world point (POST /nav/plan). Provide exactly one of object_uid | point_world. Honest failures: 422 no_geometry / no_floor / unreachable_goal (with a nearest-reachable suggestion). The response provenance line ("not certified navigation") must ride along with any use. The per-frame camera flythrough (poses) is left on disk by default and the waypoint list comes back as an evenly-sampled preview — both counts and the full-trajectory artifact key ride along; artifact_fetch(doc_key) for the whole thing. (read-only)
- `scene_planes`: Floor + wall plane candidates from the scene geometry (POST /planes; cached per-scene). Used for restyle scoping and understanding room structure. (read-only)
- `scene_ground_frame`: Read the recorded floor/ceiling frame (GET /ground_frame). If vertical_axis_known is false, the numbers are ABSENT on purpose — say so rather than guessing heights. (read-only)
- `scene_ground_frame_fit`: (Re)fit the floor plane and write the result onto the scene metrics (POST /ground_frame). dry_run=true computes without persisting.
- `scene_anchor`: Metric-anchor calibration (POST /anchor): two picked world points + their real distance in metres. Non-destructive (writes NEW derived/anchor/* artifacts + the derived_latest pointer) and it is what unlocks units:"m" for measure/nav/export. materialize:"job" returns 202 with a job_id to poll via workspace_job_wait.
- `scene_keyframe_image`: One capture keyframe as an image (GET /keyframes/<blob_key>). blob_key comes from the scene detail keyframes[] refs. Real capture photos — cite as photo evidence. (read-only)
- `scene_synthetic_views`: Renders of the scene's own splat registered as evidence (GET /synthetic_views). These are NOT photographs — always carry the "synthetic view" provenance when citing. (read-only)
- `scene_synthetic_view_image`: One synthetic view PNG (a RENDER of the splat, not a photo — say so when citing). (read-only)
- `scene_lod`: Level-of-detail index status for the scene splat (GET /lod). Always 200 — read `status` (ready|none|running|error|not_needed). (read-only)
- `scene_lod_build`: Spawn the LOD decimation job (POST /lod → 202 {job_id}; 409 lod_job_active when one is already running). Poll with workspace_job_wait.
- `scene_imported_objects`: Status of geometry-derived objects for an imported splat (GET /imported_objects). (read-only)
- `scene_imported_objects_run`: Cluster an imported splat's geometry into facts.objects (POST /imported_objects). Labels carry geometry/synthetic-view provenance — confidence 0.0 is honest, not a bug.
- `scene_object_complete`: SAM-3D-Objects mesh completion for a segmented object (POST /objects/<uid>/complete → 202 {job_id}, broker-thread lane). Poll with scene_job_wait. Output is GENERATED geometry (honesty envelope in the artifact meta.json).
- `scene_object_variants`: TRELLIS variant generation for a completed object (POST /objects/<uid>/variants → 202 {job_id}). Optional body fields pass through (see server docs). GENERATED assets.
- `scene_segment`: SAM-3 click segmentation (POST /segment). Advanced: the body must match the server contract (click point + view selection — see routes_sam3d.py / the OS UI). Provided as a raw passthrough so the full surface stays reachable.
- `scene_job_status`: One status read of a broker-thread scene job (SAM-3D completion, variants, view renders). NOTE: these jobs honestly 404 across a broker restart. (read-only)
- `scene_job_wait`: Poll a broker-thread scene job until done/error/timeout. (read-only)
- `scene_share_access`: Owner-only read of the share-link access log (GET /share/access): per minted link (access_id) — opened, request count, visits (new visit after 30 min idle), first/last seen, returned (a visit >= 24 h after the first), devices, and the Q&A questions the viewer asked. This is the behavioural evidence for customer discovery (L5 = returned). Match access_id to the person you sent the link to. events:true appends the raw event list. (read-only)
- `scene_share`: Mint a read-only, single-scene share token + embed URL (POST /share; default TTL 30 days). Share tokens can NEVER reach the OS feature routes — viewer only.
- `agent_annotate`: Start the five-phase scene annotation run (survey → labels → description → dimensions → key features) on the server-side scene agent. 202 {run_id}; one active run per scene (409 agent_run_active carries active_run_id — or set attach_if_active). Costs bounded server-side LLM calls. Follow with agent_run_events wait_for_done.
- `agent_replay`: Re-serve a recorded run's persisted event log as a new (replay-badged, $0) run. Use for reviewing what an earlier run found without spending LLM calls.
- `agent_pilot`: Give the pilot agent a free-text instruction (navigate/inspect flows; tools: list_scene_objects, plan_path, measure_distance). 202 {run_id}; poll agent_run_events.
- `agent_chat`: One chat turn with the scene agent (its own bounded run; the server agent may call its scene tools mid-turn). By default waits for the run to finish and returns the event transcript. The agent's numbers carry units/scale_source — repeat them honestly.
- `agent_run_events`: Read a run's event log from a cursor (GET /runs/<run_id>/events?after=). With wait_for_done, polls at ~1 Hz (the web client's cadence) until the run finishes. (read-only)
- `agent_runs`: The scene's run index (newest first) + which run is live: kind, status, cost_usd, llm_calls, findings. Persisted runs can be replayed for $0 with agent_replay. (read-only)
- `export_manifest`: What an export WOULD contain (GET /demo/export/manifest): file tree + sizes, complete/absent components, and whether the zip route would work. No build, no writes. Formats: openreality | groot_lerobot_v2 | isaac_usd | mujoco_mjcf (the simulator formats isaac_usd / mujoco_mjcf are metric-gated — they need an anchor). (read-only)
- `export_prepare`: Build the export zip as a background job (POST /demo/export/prepare → 202 {job_id}; 409 export_job_active when one is running). With wait=true (default) polls the job and returns the prepared artifact (including download_path). Then artifact_fetch it. For MuJoCo prefer export_mujoco, which also downloads and unpacks the scene.
- `export_status`: What is prepared for scene+format (GET /demo/export/prepared). Always 200 with {status: ready|running|error|none}; ready carries the artifact + download_path. (read-only)
- `export_mujoco`: One call from scene to a loadable MuJoCo model: builds (or reuses a fresh) mujoco_mjcf export, downloads it and unpacks it to disk, and returns the path to scene.xml — load it with mujoco.MjModel.from_xml_path(path) or `python -m mujoco.viewer --mjcf=<path>` (needs mujoco>=3.3). METRIC-GATED: an un-anchored scene is refused with 409 metric_scale_required — run scene_anchor first. Collision is primitives only (a floor plane + one upright box per detected object); the scan mesh is visual-only. A fresh build meshes the scene on a 64 GB container and takes minutes. Bytes never enter context.
- `artifact_fetch`: Download any derived/... artifact (export zips, LOD levels, agent run logs, QC reports, masks) to a local file and return the path — bytes NEVER go into context. Default target: <artifacts_dir>/<scan_id>/<key>. (read-only)
- `artifact_fetch_splat`: Download the scene's original splat.ply to a local file and return the path. For a render-sized variant prefer the LOD levels (scene_lod → artifact_fetch of a level key). (read-only)
- `artifact_fetch_cloud`: Download the scene's cloud.ply to a local file and return the path. (read-only)

## FAQ

### How do I get started with Open Reality?

You can add the tools to your AI assistant using npx commands for Claude or Codex, or install the Claude desktop extension directly.

### Can I run this server on my own hardware?

Yes, the entire workflow is self-hostable on your own GPU box or via a Modal account.

### What kind of data can I export from a scan?

You can export scans as LeRobot or GR00T style datasets, or as Isaac Sim or MuJoCo scenes.
