MCP server de bug2context

bug2context

Turns a screen recording of a bug into structured, chronological context an AI agent can actually read.

Screenshots lose the story: bugs happen fast, a still frame has no timeline, and the logs and the narration live somewhere else entirely. LLMs don't take video. This distills a recording into one markdown chronology — key frames, the text on screen, device logs and spoken narration, all on the same clock.

• No SDK. Works on any video, including one someone sent you over WhatsApp.
• Local-first. Nothing leaves your machine. No account, no API key, no cloud.
• MCP-native. Claude Code calls it as a tool; you just point at the file.

uvx bug2context               # guided menu
uvx bug2context process bug.mp4 --out bundle/

Running it with no arguments opens a menu — record an Android device, process a video you already have, or list previous bundles — and writes to ~/.bug2context/, which is where the MCP server looks. Everything it does is reachable through the flags below; it just stops you having to remember them.

What comes out

- `[00:00]` 🖼 `frames/frame_001_00-00.png` — start of recording
      OCR: Cart 2 items
- `[00:04]` 🖼 `frames/frame_002_00-04.png` — screen changed
      OCR: Applying discount..
- `[00:04]` 📋 W/DiscountEngine: rate field missing
- `[00:06]` 🖼 `frames/frame_003_00-06.png` — screen changed
      OCR: TypeError: cannot read 'rate'
- `[00:06]` 📋 E/RideService: NullPointerException at RideRequest.kt:142
- `[00:06]` 📋 F/AndroidRuntime: FATAL EXCEPTION: main
- `[00:06]` 🎙 "ahí está, explota al aplicar el descuento"

A full run is in examples/report.md.

bundle/
├── report.md    the chronology above
├── frames/      the key frames, named by timestamp
└── meta.json    the same data, structured, for programmatic use

Install

Requires ffmpeg on PATH (brew install ffmpeg).

uvx --from 'bug2context[ocr]' bug2context          # OCR: Apple Vision on macOS, tesseract elsewhere
uvx --from 'bug2context[ocr,audio]' bug2context    # plus spoken narration

Or from source:

git clone https://github.com/emanueld92/bug2context
cd bug2context
uv sync --extra ocr --extra audio

Every stage is optional and degrades quietly: no OCR backend still gives you frames, no audio track still gives you the visual timeline.

Use it from Claude Code

claude mcp add bug2context -- uvx --from 'bug2context[ocr]' bug2context-mcp

Working from a local clone instead: uvx --from /path/to/bug2context bug2context-mcp.

Then just ask: "analiza este video del bug: ~/Desktop/crash.mp4".

Tool	Purpose
analyze_bug_video	Video → chronological report (returns the text itself)
get_frame	Fetch one frame as an image so the agent can look at it
list_bundles	Previously processed bundles, newest first

Bundles default to ~/.bug2context/<video>-<timestamp>/.

Android: screen + logcat in one command

uv run bug2context record --seconds 60 --package com.example.app --out bundle/

Reproduce the bug while it records. Both streams start from a single host timestamp — that anchor is what makes a stack trace land next to the frame showing the crash.

Pass --package. Level filtering alone is not enough on a real phone: a 2 min capture held 7,634 W/E/F lines and none came from the app under test — they were all AppOpsControllerImpl, GNSSMGT and friends. Scoping to the app's process left 35. Crash tags (AndroidRuntime, DEBUG) are kept regardless of process, since a crash report is the whole point.

Logs are captured unfiltered and scoped afterwards, so an app that crashes and restarts still has its death recorded. Consecutive identical entries fold into one with a (×N) count — framework chatter arrives in bursts of fifteen.

Already have a log file? Merge it into any video:

uv run bug2context process bug.mp4 --logcat logcat.txt --log-tag RideService

Reads both -v time and -v threadtime. --log-levels defaults to WEF and --log-max-lines to 80, ranked by severity so a cap never trades the crash for boot chatter. --log-pid scopes to a process when you already know it. If the device clock and the recorder disagree, --log-offset shifts everything by N seconds; with no --video-started-at, the first log entry becomes the anchor.

screenrecord caps at 3 minutes, so record refuses longer rather than handing back a silently truncated video. Ctrl-C cuts a recording short safely — the file is finalised on the device first, because killing the local adb leaves one ffprobe cannot open.

Videos over 10 minutes are refused with the ffmpeg command to trim them (--max-duration 0 overrides). Nothing scales badly with length except time, but it scales linearly: decoding alone runs at about a third of real time.

Narration

uv run bug2context process bug.mp4 --transcribe --language es

Off by default: it downloads a model on first use and is the slowest stage, while most recordings have no voice-over. Skipped automatically when there is no audio track, or the track is quieter than −50 dB — whisper invents confident sentences out of silence, so that guard is about output quality, not just speed.

How frames get picked

ffmpeg over-produces candidates (scene cuts and a fixed interval, so slowly changing screens are not skipped), then perceptual hashing collapses the near-duplicates. --max-frames caps the result, always keeping the opening frame and then whichever changed most.

Flag	Default	Notes
--scene-threshold	0.08	ffmpeg scene score for a cut
--interval-seconds	1.0	forced sample when no cut fires
--max-frames	auto	one frame per 2 s, between 20 and 40
--phash-distance	4	below this, frames count as duplicates
--ocr-upscale	2	enlarge before OCR; skipped above 1600 px wide

Measured end to end, OCR on:

Source	Result	Time
synthetic 3 min, 1080×1920	19 frames	8.5 s
Android capture, 2 min, 720×1612 @ ~12 fps	118 candidates → 20 frames	16 s
macOS capture, 90 s, 2880×1864 @ 60 fps	97 candidates → 18 frames	86 s

Recognised text is cached by frame content in ~/.cache/bug2context/, so re-running a video with different settings only pays for what actually changed — measured 9.0 s cold against 3.3 s warm on a 44 s clip, identical output. Extracting the same video twice yields byte-identical frames, which is what makes the cache safe. Delete the directory to reset it.

Retina desktop recordings are the slow case and there is no trick to remove: the cost is decoding 60 fps at 5.4 megapixels (26 s) plus OCR, which Vision charges at ~1.6 s per frame regardless of size. Budget roughly real time for those; phone captures stay far under it.

On --ocr-upscale: a stack trace on a recompressed 1080×1920 frame read as Null PointerSxception Ride Requestkt14 at 1× and correctly as NullPointerException Ride Request kt 142 at 3×. Raise it for badly recompressed sources — ask for the video as a file, not as a WhatsApp video. It is skipped on frames already 1600 px or wider, where it changed nothing measurable while building a 5760×3728 image per frame.

OCR also drops lines that are debris rather than text — », .lll (100 4, a misread status-bar clock. On a real capture that removed 41 of 210 report lines and no real text. A line that looks like a failure is never dropped.

Known limitation

Perceptual hashing compares visual structure. It sees navigation, dialogs and error banners, but it cannot detect a small text-only change on an otherwise identical layout — measured on 1080×1920 those differences fall inside the noise floor at every hash size, so no --phash-distance value separates them.

OCR does not rescue this: it runs on frames that survive deduplication, so a screen already collapsed is never read. Reading every candidate instead measured 48–59 s for a 3 min video, over the entire time budget. If your bug is a small text change on an identical screen, record cropped or at lower resolution so the text occupies more of the frame.

Development

uv sync --extra ocr
uv run pytest -q

183 tests. Device recording is mocked on purpose — a test suite should not record anyone's phone. Whisper is opt-in via BUG2CONTEXT_TEST_WHISPER=1 so the suite stays offline.

License

MIT — see LICENSE.