filegoblin MCP server

(o_o) filegoblin (filegoblin)

"Ingesting the messy world, spitting out clean context."

filegoblin is a single-binary CLI that "gobbles" almost any file or website — PDF, Word, Excel, PowerPoint, email, EPUB, HTML, whole repos — and converts it into clean, LLM-ready markdown.

Philosophy

• It just works. filegoblin document.pdf > context.md needs no manual. Power flags exist, but the default path is the product.
• Format invisibility. You shouldn't have to think about extensions; detection is extension-first with magic-byte fallback, so renamed and extension-less files route correctly.
• Fail loudly, never silently. An empty result exits non-zero, parse failures are reported in-band, and invalid flags are hard errors. An agent can trust every exit code.
• Local and private. Parsing and PII scrubbing happen entirely on your machine. No cloud APIs, no runtimes, no pip install.
• Not just a concatenator. cat * > out.txt exists; filegoblin adds structure (real markdown from Word, one consistent table dialect, readability-grade web extraction), token budgeting, and redaction.

---

📦 Installation

Every release comes in two flavours:

• with the PDF vision models for layout and reading order, formulas as LaTeX, tables, OCR on scanned pages, and a page reader for handwritten and math scans (~1.3 GB; ~510 MB without the page reader);
• lite, without them: the fast geometric converter, plus Apple Vision OCR on macOS.

Everything else is the same.

Channel	Lite	With the models
GitHub release (latest)	filegoblin-<target> archive	filegoblin-full-<target> archive: one binary with the models compiled in
PyPI	pip install filegoblin	pip install filegoblin, then filegoblin --install-models (the filegoblin[models] packages aren't on PyPI yet)
npm	npm i -g filegoblin	npm i -g filegoblin @filegoblin/models
Cargo	cargo install --git https://github.com/funkyfunc/filegoblin filegoblin	then filegoblin --install-models

Builds are published for macOS (Apple Silicon and Intel), Linux (x86_64 and aarch64), and Windows (x86_64).

Getting the models later.

• filegoblin --install-models downloads this version's model archive from the GitHub release, checks its SHA-256 and unpacks it into ~/.filegoblin/models. You can pass a directory instead, and set FILEGOBLIN_MODELS_URL to point at an internal mirror. filegoblin only downloads when you ask it to.
• filegoblin --models-status shows where each model was found.

Behind a corporate network. The npm model packages (and the PyPI ones, once they're published) carry the weights themselves, one package per model, with the two largest models in halves to stay under the registries' 100 MB per-file limit. They install from a company mirror (Artifactory, Nexus, CodeArtifact) like any other package, with no access to github.com.

Air-gapped machines. Copy the full download. Alternatively, unpack the filegoblin-vision-models.tar.gz archive on a shared read-only path and name that directory in FILEGOBLIN_MODELS (the archive itself is not read). That variable takes a PATH-style list. Licenses are in LICENSES.md inside the archive.

From source — needs a Rust toolchain (1.87+):

cargo install --git https://github.com/funkyfunc/filegoblin filegoblin

That gives you everything in the "always on" list below, including web fetching and crawling. The optional engines are opt-in because each one pulls in a heavy dependency:

Want	Add	Build needs
OCR on Linux/Windows	--features ocr	just get-brains for the weights (macOS uses Apple Vision, no flag)
PDF vision models (layout, reading order, formulas → LaTeX, tables, scan OCR)	--features vision (on in release builds)	The weights: filegoblin --install-models (the release's filegoblin-vision-models.tar.gz, SHA-256 checked) or just build-vision-models. filegoblin finds them in models/ next to the binary, in ~/.filegoblin/models, or in $FILEGOBLIN_MODELS. Add bundled-models with FILEGOBLIN_MODELS_DIR set to compile them into one binary
Audio/video transcription	--features transcribe	cmake + a C/C++ compiler; downloads a ~574 MB model on first use
Metal-accelerated transcription (macOS only)	--features transcribe-metal	Xcode Command Line Tools. Does not build on Linux/Windows
WASM plugins	--features wasm	—
BM25 search (--search)	--features rag	—

# Example: a macOS build with GPU-accelerated transcription
cargo install --git https://github.com/funkyfunc/filegoblin filegoblin --features transcribe-metal

Without the models, --vision smart (the default) behaves exactly like the fast converter, with one note per run on stderr; --vision off silences it. With the models, it uses them only on the pages that need them. Anyone building from source can make a single self-contained binary with --features vision,bundled-models and FILEGOBLIN_MODELS_DIR set.

---

Measuring quality. just gauntlet runs the 29-case adversarial gauntlet (just gauntlet off for the fast path). Each case gets pass/fail canaries and a 0–100 quality score graded against hand-checked gold excerpts in test_assets/gauntlet/gold/, plus checks that --format plain and --chunk-size lose nothing. just gauntlet-compare A B shows what a change did case by case. Results are in GAUNTLET_SCORECARD.md.

---

🏗 Current Project Structure

Note to Agent: Maintain this section as you add modules.

.
├── Cargo.toml        (Virtual Workspace Root)
├── crates/
│   ├── core/         (filegoblin-core: The Engine & Parsers)
│   │   ├── Cargo.toml
│   │   ├── build.rs  (Man page & shell completions)
│   │   └── src/
│   │       ├── lib.rs        (Core Pipeline Logic)
│   │       ├── cli.rs        (Clap Flag Definitions)
│   │       ├── formatters.rs (Output Assembly & Splitting)
│   │       ├── formatters/   (Plain text & specialized formatters)
│   │       ├── chunking/     (Layout-aware semantic chunking engine)
│   │       ├── curation/     (BM25 Search & Auto-Pruning)
│   │       ├── privacy_shield.rs (PII Redaction Engine)
│   │       ├── compressor/   (Token Compression Pipeline)
│   │       ├── parsers/      (File parsing engines; pdf_vision.rs = layout-guided PDF assembly,
│   │       │                  pdf_furniture.rs + pdf_edges.rs = page furniture, pdf_words.rs = glyph-level
│   │       │                  words, pdf_math.rs = inline-math clean-up, pdf_worker.rs = PDFs converted
│   │       │                  in child processes)
│   │       ├── vision/       (Bundled ONNX models on rten: layout, formula, table, OCR, page reader; hayro rendering)
│   │       └── pipeline/     (Pipeline execution stages)
│   └── cli/          (filegoblin: The Headless Binary)
│       ├── Cargo.toml
│       └── src/main.rs       (CLI Entry & Mascot)
│           └── walk.rs       (--walk interactive picker)
├── scripts/          (Dev tooling: run_gauntlet.py = graded gauntlet, olmocr_bench.py = olmOCR-bench
│                      on a fixed dev/test split, engines.py = other tools graded the same way
│                      (--engine; docs/COMPARISON.md), model packaging and release helpers)
├── npm/              (npx filegoblin Zero-Install Launcher)
│   ├── package.json
│   └── bin/filegoblin.js
└── .cargo/
    └── config.toml   (Aliases)

---

⚡ Current Capabilities (v1.17.0)

Some capabilities are gated behind opt-in Cargo features (built off by default to keep the binary lean). Feature-gated items are marked below.

• Multi-Format Output Engine (--format markdown|plain|json / -f): GFM markdown (default), syntax-stripped plain text with space-aligned tables for vector embeddings (e.g. OpenAI text-embedding-3, BGE) and BM25 search, and structured JSON documents
• Layout-Aware Semantic Chunking (--chunk-size, --chunk-overlap, --chunk-strategy layout|tokens): splits along structural headings (H1 > H2 > H3), protects tables and code blocks from mid-block slicing, automatically replicates table headers across chunk splits, and embeds hierarchical outline breadcrumbs
• Core Parsers (PDF, Word, Excel/CSV/TSV, PowerPoint, OpenDocument, Jupyter, Code, Email, RTF, HTML, EPUB) — always on, including Office's macro-enabled and template variants (.docm/.dotx/.dotm, .xlsm/.xltx/.xltm/.xlsb/.xlam, .pptm/.potx/.ppsx/.ppsm) and the pre-2007 binaries (.doc/.dot, .xls, .ppt/.pps/.pot)
• Legacy text encodings: plain text, markdown, source code, CSV/TSV, subtitles and stdin are decoded from a byte-order mark (UTF-8/UTF-16), as UTF-8, or else as Windows-1252/Latin-1 — binary files are still refused. CSV delimiters (, ; tab |) are sniffed, so European ; exports keep their columns
• PDF → markdown via pdf_oxide: headings, pipe tables, reading order, hyphenation — with per-page scanned-vs-text classification. Word spacing is re-read from glyph positions, so kerned, letter-spaced and ligature-split words ("infl uenced") and accents drawn apart come out whole; text outside the crop box (printer's slugs) is dropped; a password-protected PDF gets a clear error
• Scanned PDFs and images OCR automatically on macOS (Apple Vision, zero setup; parallelized page conversion, 30+ pages/s; every page of a multi-page TIFF, under ## Page N); Linux/Windows via --features ocr
• PDF vision pipeline (--features vision + model weights; --vision smart|all|off, default smart): pages are rendered in pure Rust (hayro) and read by bundled, permissively licensed models on rten — no Python, system libraries or general-purpose LLM at runtime. PP-DocLayoutV3 places every text-layer word in its region (title, heading, paragraph, caption, footnote, table, figure, running header/footer) in a learned reading order; display and inline formulas are re-read from the image as LaTeX (pix2text MFR); tables get rows/columns/spanning cells from Table Transformer and their cell values from the text layer; scanned pages, text regions the text layer leaves empty, and scans whose own OCR layer is too damaged to keep are read with PP-OCRv6 (medium). A scanned page PP-OCR is unsure of (handwriting) or that is full of formulas is read whole by the optional page reader, LightOnOCR-2-1B (Apache-2.0, 4-bit on rten, about half a minute to two minutes a page); every other page keeps its speed. --vision smart decides per page from cheap text-layer facts — scans, math fonts/symbols, column-aligned tables and forms, multi-column text, pictures, one- or two-page documents go through the models; plain single-column prose stays on the fast path. On the full olmOCR-bench (1,403 PDFs, official scorer; the 180 it was tuned on included) the models take filegoblin from 41.1% to 74.8% (arXiv math 0% → 87%, tables 35% → 80%, multi-column 50% → 73%, headers/footers 74% → 92%). On the 1,223 PDFs it was not tuned on, it scores 78.3% on the CPU alone (74.3% before the page reader); MinerU 79.0%, marker (fast mode) 65.6%, docling 54.6% and pymupdf4llm 42.3%, each in its default configuration (docs/COMPARISON.md). Ruled tables take their grid from the page's own rulings, and rotated (/Rotate) pages are read as displayed
• Geometric PDF page furniture removal: running headers/footers, page numbers, and line-number gutters (pleading paper, patents) are identified from position, cross-page recurrence, font size and alignment — before conversion — so prose, numbers and headings are never rewritten by text heuristics. One-off furniture that cannot recur (letterheads, print stamps, document IDs, URLs, copyright and "downloaded from" lines) is found on each page from its place, size and wording. Line-break hyphenation and words run together across visible gaps are repaired from page geometry plus the document's own vocabulary
• Semantic Chapter Boundaries: Automated detection and injection of thematic dividers and XML section markers (<!-- filegoblin:section ... -->)
• Word → real markdown: heading levels, nested lists, bold/italic, hyperlinks with URLs, and equations
• Word Resilient Fallback: Streaming OpenXML reader handling <w:gridSpan> merged tables and <mc:AlternateContent> floating text boxes
• Word Office Math (OMML): <m:oMath> and <m:oMathPara> converted directly to standard LaTeX ($ ... $ and $$ ... $$) with radicals, delimiters, matrices, and functions
• Embedded Office Charts: PowerPoint (.pptx) and Word (.docx) OpenXML DrawingML charts (c:chartSpace) deconstructed into clean Markdown pipe tables, Mermaid xychart-beta, and native Mermaid pie showData diagrams
• DrawingML Connectors & SmartArt: PowerPoint and Word shape connections, flowcharts, and SmartArt process hierarchies compiled into Mermaid flowchart diagrams
• Zero-install npm distribution: Execute via npx filegoblin in environments without Rust toolchains
• DOM-based HTML → markdown (readability-style content selection, link-farm pruning, preserved link URLs, title/source front-matter, ATX headings)
• One tabular dialect everywhere: standard markdown pipe tables across PDF, docx, pptx, OpenDocument, and CSV/XLSX
• Pure Rust Ruling-Line Table Extraction: Native port of pdfplumber.table vector ruling-line and intersection table algorithm, supporting collinear edge snapping, bounding box cycle reconstruction, multi-column merged headers, and fallback gutter detection
• Trafilatura DOM Content Extraction: Native port of trafilatura.core text/link-density heuristics, boilerplate token pruning, syntax-highlighted code block preservation, and citation-link dampening
• PowerPoint: manifest-true slide order + speaker notes + embedded chart tables
• Legacy Office binaries (Word, PowerPoint and Excel 97–2003) via office_oxide (pure Rust): .doc heading levels from the document's styles, tables, nested lists, links and footnotes; .ppt slides in the same shape as .pptx, with speaker notes; .xls sheets in the same layout as .xlsx, cells as Excel displays them. The file's own streams decide whether it is Word, PowerPoint or Excel, so renamed and extension-less files convert; RTF or HTML saved as .doc goes to its own parser; password-protected files and Outlook .msg get a clear error
• Magic-byte format detection — extension-first, falling back to content signatures, so extension-less files and files with an unknown extension still route correctly (zip, tar and gzip archives; Office documents are recognised by the parts inside their zip, legacy ones by the streams inside their OLE2 container). A known but wrong extension (a .docx renamed to .pdf) still goes to that extension's parser
• URL fetching routes by Content-Type: HTML → extractor, raw markdown/JSON/text → passthrough, binary documents (a PDF URL!) → the normal parser pipeline
• --browser: render JavaScript-built pages (SPAs) with your installed Chrome/Edge/Brave — headless, throwaway profile, no bundled browser
• Recursive crawling (--horde) + GitHub repo cloning — on by default in the CLI binary (web feature; disable with --no-default-features)
• Fail-loud contract for agents: exit 4 whenever nothing could be extracted, hard errors on invalid flags, quiet exit on broken pipes, explicit paths take precedence over piped stdin
• MCP server (--mcp): stdio Model Context Protocol server exposing a convert tool — plug filegoblin into Claude Desktop/Code with one config line
• Subtitles (.srt/.vtt): cue files → the same timestamped transcript markdown whisper produces, with speaker labels and rolling-caption dedup
• RSS/Atom feeds: a feed URL or saved feed file → a clean markdown list of entries
• Conversion cache: repeat URL fetches and OCR/transcription results come back instantly (~/.filegoblin/cache, --no-cache to bypass)
• Audio/video transcription (--features transcribe; add transcribe-metal on macOS for GPU): mp3/m4a/wav/flac/ogg/opus/mp4/mov/webm/mkv (audio track) → timestamped transcript markdown via whisper.cpp, on-device, model auto-downloads once
• Golden-file test suite: end-to-end snapshots of real fixture output per format (cargo insta review to inspect changes)
• WASM Plugin Extensibility (--plugin) — requires --features wasm
• Semantic BM25 Search with Relevance Scores (--search) — requires --features rag
• Conversion cache management: --no-cache to bypass, --cache-clear to empty it
• Glob Filtering (--include, --exclude) & Depth Control (--depth)
• Token Estimation (--tokens), Token-Only Mode (--tokens-only)
• Token Compression (--compress safe|contextual)
• Auto-Pruning to Token Budgets (--max-tokens)
• Manifest Table of Contents (--manifest)
• Git-Aware Diffing (--git-diff) with Unified Diff Output (--diff-format)
• Rust Symbol Extraction (--extract symbols) via tree-sitter
• Local PII / Secret Redaction (--scrub) — deterministic + entropy heuristics
• Output Splitting (--split), Chunking (--chunk), JSON (--json)
• Clipboard Ingestion (--clipboard), Clipboard Output (--copy), OS Open (--open), File Write (--write)
• Heuristic Project Summaries (--summary)
• Out-of-core streaming for the default dump path (flat RAM footprint)
• Pipeline-Safe Streams (-q for clean stdout)

---

✨ Key Features

• Parsers: pdf_oxide, docx-rs, tree-sitter, quick-xml, calamine, scraper (Statically Linked).
• OCR built in on macOS: Apple Vision (objc2) with no weights or flags — scanned PDFs and images just work. Linux gets pure-Rust ocrs/rten inference behind --features ocr.
• WASM Extensibility: wasmtime for pure-rust statically-linked component execution.
• Privacy First: 100% local PII/secret scrubbing — deterministic patterns (SSN, email, cloud/token prefixes) plus a Shannon-entropy tier that redacts high-entropy secrets. No model weights, no data egress.
• Structural Minification: Tree-sitter powered --extract symbols mode for Rust source.

Source	Path Example	Parser Used	Output Format
PDF	~/receipt.pdf	PdfGobbler	Markdown (headings, pipe tables, OCR for scans)
Word	~/notes.docx	OfficeGobbler	Markdown (headings, lists, emphasis, links, equations, charts)
Excel/CSV	~/data.xlsx	SheetGobbler	Markdown pipe tables (header row named)
PowerPoint	~/deck.pptx	PptxGobbler	Per-slide markdown + speaker notes + embedded charts
OpenDocument	~/doc.odt, ~/deck.odp	OpenDocGobbler	Markdown (headings, lists, tables)
Source Code	~/project/src	CodeGobbler	Full content (or --extract symbols)
Email	~/thread.eml, ~/inbox.mbox	EmailGobbler	Header block + markdown body + attachment list
RTF	~/notes.rtf	RtfGobbler	Plain text
Images	~/screenshot.png, ~/IMG_0042.HEIC	OcrGobbler (built-in on macOS)	OCR Text
Subtitles	~/movie.srt, ~/talk.vtt	SubtitleGobbler	Timestamped transcript
RSS/Atom	https://hnrss.org/frontpage	FeedGobbler	Entry list (title, link, date, body)
Webpages	https://news.ycombinator.com	HtmlGobbler (default build)	DOM extraction + front-matter

• Semantic Search: BM25 relevance scoring via tantivy with zero model weights.
• Manifest TOC: Auto-generated table of contents with per-file token counts.

⚙️ Development & Asset Pipeline

filegoblin maintains a strict zero-dependency profile for the end user. However, building the project requires assembling several logic assets.

build.rs generates the manpage and shell completions into target/generated/. The .rten tensor weights for pure-Rust OCR fallback on Linux/Windows are not downloaded automatically — fetch them with the provided justfile when building with --features ocr:

just get-brains

(On macOS, OCR uses Apple's Vision framework natively; no weights needed.)

---

🚀 Quick Start

Basic Gobble (Files):

# Output a single file to stdout
filegoblin my_notes.pdf > context.md

# Merge multiple files natively
filegoblin src/main.rs src/lib.rs README.md --write context.md

URL Ingestion (Web):

filegoblin https://example.com/api-docs > context.md

Web Horde (Split into mapping directory):

filegoblin https://bettercli.org/ --horde --split
# Translates to -> ./bettercli.org_gobbled/

Privacy Shield (Redact PII & Secrets locally):

# Typed redaction (default): values become [EMAIL], [AWS_KEY], [CREDIT_CARD], …
filegoblin ./src/api_keys.ts --scrub > safe_context.md

# Pipe a log through it as a pre-flight gate before any cloud LLM
cat prod.log | filegoblin - --scrub --scrub-style plain

# CI / pre-commit guard: non-zero exit if anything sensitive is found
git ls-files -z | xargs -0 cat | filegoblin - --scrub --scrub-fail -q || echo "secrets detected!"

# Exempt a known-public value, or add your own pattern via ./.filegoblin-scrub.json
filegoblin ./config --horde --scrub --scrub-allow "support@example.com"

# Consistent pseudonyms: the same value maps to [EMAIL_1] everywhere, so an LLM can
# still tell that an entity recurs — without ever seeing it.
cat ticket.txt | filegoblin - --scrub --scrub-style pseudonym

Detection is 100% local with zero model weights: validated rules (Luhn for cards, range checks for SSN, mod-97 for IBAN) plus cloud/token prefixes (AWS, GitHub, Stripe, Slack, Google, OpenAI, JWT, private-key blocks, connection strings) and a Shannon-entropy catch-all for unknown secrets. A value-free audit tally is printed to stderr (🛡️ Privacy Shield redacted 4 item(s): …). An optional .filegoblin-scrub.json supplies an allowlist and custom patterns.

What filegoblin does NOT do: maintain a secrets blocklist. It converts exactly what you point it at — including a .env file if you name one. Deciding what's sensitive is yours: the directory walker respects .gitignore (where secrets usually already live), --exclude filters by glob, and --scrub redacts what does get through.

Machine Contract (--json) — extract any file to clean text for agents/scripts:

# One weird binary format → clean text + metadata, in a single call
filegoblin report.pdf --json | jq '.[0] | {format, tokens}'
# → { "format": "pdf", "tokens": 1234 }

filegoblin ./src/ --horde -q --json | jq '.[].path'

--json emits a deterministic, alphabetically-sorted array. A successful record is { "path", "format", "tokens", "content" } (format is a coarse type like pdf, spreadsheet, code, image, text; tokens is an LLM token estimate). A file that fails to parse appears in-band as { "path", "format", "error" } rather than silently vanishing — so an agent can trust that every input is accounted for. Exit codes: 0 success, 4 when nothing could be extracted (empty result vs. total failure), 3 when --scrub --scrub-fail redacted something, 5 when the run went over its memory ceiling (--memory).

Multi-Format Output (--format markdown|plain|json / -f):

# Default GFM markdown with tables, math, and code fences
filegoblin document.pdf -f markdown > doc.md

# Strip markdown punctuation for dense vector embeddings & BM25 search
# (Pipe tables become space-aligned text columns; headings & math stripped of tokens)
filegoblin document.pdf -f plain > embedding_input.txt