TokenJar MCP server

MCP server that saves 70-95% tokens for AI coding assistants — without losing functionality.

TokenJar sits between your AI coding assistant and your codebase, intelligently compressing code reads, terminal outputs, and file operations to dramatically reduce token consumption, context compaction, and latency.

Works with Claude Code, Cursor, Antigravity (AGY), Windsurf, Continue.dev, and any MCP-compatible AI assistant.

---

✨ Features & Architecture

Module	What It Does	Token Savings
🦴 Code Skeletonizer	Extracts structural skeleton (signatures, types, docstrings) via Tree-sitter AST	80-95%
📖 Smart File Reader & Slicing Cache	L1 RAM + L2 Persistent SQLite cache with targeted line/symbol slicing (#L1-30) & auto-paging ceiling (MAX_OUTPUT_LINES = 80)	90-99.8%
🛡️ Lockfile & Asset Shield	Intercepts massive lockfiles & minified bundles with surgical version queries (query="react")	99%
🎯 Blast Radius & Hybrid Fuzzy Symbols	Instant global symbol lookup with AST parser and fuzzy Levenshtein + Trigram fallback suggestions (find_symbol_global)	85-95%
🖥️ Terminal Pruner	Compresses test/build/git terminal streams, keeps errors and summary info	60-90%
🗺️ Repo Map	PageRank & Graph Centrality codebase overview fitted into custom token budgets	Budget-fitted
🔄 Post-Update Auto-Sync	Automatically syncs IDE configs, slash commands, and project rules in a single command (tokenjar update)	Zero-maintenance
🧹 Zero-Trace Surgical Uninstall	Safely reverts IDE configs, purges caches, and cleans rules without touching user code	Safe & Zero-trace
🎨 On-Demand UI Dashboard	Lightweight standalone control panel (tokenjar ui) with Zero Background RAM	Instant

🛡️ Built-in Guardrails & Reliability

• Lockfile & Giant Asset Shield: Prevents context window destruction from 50,000-line lockfiles; supports 5-line surgical version queries.
• L1 RAM + L2 SQLite Persistent Cache: Survives MCP server restarts and IDE reboots (~/.tokenjar/cache.db with WAL mode).
• Fallback Safety Guard: If a test or command fails (exit_code != 0), TokenJar guarantees tracebacks and error contexts are preserved intact.
• Tiny File Anomaly Guard: If a diff header would consume more tokens than the file itself, the full content is returned to prevent token inflation.
• Runaway Stream Protection: Protects host memory from infinite loops by capping raw terminal buffers at 2MB with graceful truncation.
• SQLite Database Bloat Guard: Files larger than 5MB are cached by hash reference without bloating disk space.

---

🚀 Quick Start & Installation (v1.1.0 GA)

TokenJar is distributed in two official editions:

1. 🦀 Rust Native Engine (Recommended): High-performance, self-contained single binary with microsecond AST, 14 MB RAM, and zero Python dependencies.
2. 🐍 Python Edition: Pure Python FastMCP package for pip and virtual environments.

💡 Recommended Workflow (Best Practice — 3 Simple Steps)

[!TIP] Prevent broken paths when cleaning temporary files:

1. Place in a Permanent Folder: If downloading the standalone binary or zip manually, extract tokenjar into a permanent, safe directory that won't be accidentally deleted (e.g. C:\Users\<username>\.tokenjar\bin on Windows or ~/.local/bin on Linux/macOS). Avoid temporary folders like Downloads or Temp. (Note: The 1-click terminal scripts in Option 1 handle this placement automatically).
2. Run One-Time Install: Open a terminal in that folder and run:

   tokenjar install
   

   This automatically adds TokenJar to your system PATH, configures MCP across all detected AI assistants (Cursor, Claude Desktop, Antigravity, Windsurf, Claude Code), and sets up slash commands.
3. Activate in Your Project (CLI Recommended): Open a terminal in your project repository and run:

   tokenjar on
   

   (Running tokenjar on directly in your terminal/CLI is strongly recommended over typing /tokenjar on in the chat, as it creates rules before the assistant session starts and consumes zero conversation tokens). This injects repository steering rules (AGENTS.md) so your assistant automatically routes through TokenJar's token-saving MCP tools from turn 1.

📥 1-Click Direct Downloads (Precompiled Binaries)

Click your operating system below to download the latest v1.1.0 release:

Platform	Architecture	Click to Download	Format
🪟 Windows	x86_64 (64-bit)	⬇️ Download tokenjar-windows-x64.zip	Standalone .exe + Installer
🐧 Linux	x86_64 (64-bit)	⬇️ Download tokenjar-linux-x64.tar.gz	Standalone Binary
🍏 macOS	Apple Silicon (M1/M2/M3/M4)	⬇️ Download tokenjar-macos-arm64.tar.gz	Standalone Binary
🍏 macOS	Intel x86_64	⬇️ Download tokenjar-macos-x64.tar.gz	Standalone Binary
🐍 Python	Cross-platform	⬇️ Download tokenjar-python.zip	Python Wheel (.whl)

---

⚡ Option 1: Rust Native Engine (1-Click Terminal Install)

Best for: Highest speed, 14 MB RAM, microsecond tree-sitter AST, and zero Python dependency.

Copy and paste one line into your terminal to install and add tokenjar to your PATH automatically:

Windows (PowerShell):

iwr -useb https://raw.githubusercontent.com/Farukes/TokenJar/main/install.ps1 | iex

Windows (CMD / Command Prompt):

powershell -ExecutionPolicy Bypass -Command "iwr -useb https://raw.githubusercontent.com/Farukes/TokenJar/main/install.ps1 | iex"

# Linux & macOS (Bash):
curl -fsSL https://raw.githubusercontent.com/Farukes/TokenJar/main/install.sh | bash

Or install via Cargo (crates.io):

cargo install tokenjar

---

🐍 Option 2: Python Edition (pip)

Best for: Python-centric environments, custom script integration, or pip workflows.

# Install from PyPI
pip install tokenjar

# Or install directly from GitHub main:
pip install git+https://github.com/Farukes/TokenJar.git

---

🔄 Updating TokenJar

Keep your installation up to date with the latest features and engine optimizations:

# Self-update via TokenJar CLI (both 'update' and 'upgrade' work identically):
tokenjar update
# or:
tokenjar upgrade

# Force re-download:
tokenjar update --force

# If installed via pip:
pip install --upgrade tokenjar

# If installed via Cargo:
cargo install tokenjar --force

---

📊 Proven Performance & Stress Test Benchmark

Empirical results from our rigorous 100-Step Real-Life Developer Stress Test and 50-Cycle MCP Head-to-Head Benchmark comparing Standard Raw AI vs TokenJar Python vs TokenJar Rust Native Engine:

Metric	1. Raw AI (No TokenJar)	2. TokenJar Python	3. TokenJar Rust (v1.1.0)	Rust Advantage
Consumed Tokens (100 Steps)	622,892 tokens	95,492 tokens	68,641 tokens	89.0% net savings (554k tokens saved)
End-to-End Coding Savings	166,513 tokens	12,400 tokens	6,585 tokens	🚀 96.0% net savings (Surgical edits)
API Cost (per 100 Steps)	$1.8687	$0.2865	$0.2059	$1.66 saved per 100 steps
Total Runtime (100 Steps)	0.357 s (raw disk)	2.618 s	0.985 s	2.7x faster than Python
Warm Cycle Latency	N/A	23.6 ms	8.1 ms	3.0x faster execution
RAM / Memory Footprint	~30.0 MB	49.1 MB	15.0 MB	70% to 84% less RAM
Quality & Accuracy Score	100.0%	100.0%	100.0% (100/100)	100% functional completeness
Syntax Integrity & Zero Truncation	Ham (Unverified)	✅ Enforced	✅ Enforced	Zero placeholder comments

---

📝 Project Steering Rules (AGENTS.md)

Inject or remove TokenJar AI steering instructions into your repository rules:

# 🟢 Activate TokenJar in current project (creates/updates AGENTS.md, .cursorrules)
tokenjar on

# ⚪ Deactivate TokenJar in current project (cleanly removes TokenJar block)
tokenjar off

🎛️ Output Optimization Controls (CLI & Terminals)

Switch between compact surgical output and default unrestricted output with crystal-clear commands:

# 🟢 Enable compact surgical diffs & zero-truncation quality mandate
tokenjar output on

# ⚪ Revert AI assistant to default unrestricted output settings
tokenjar output off

# 📊 Check current output configuration status
tokenjar output

Slash commands are also supported in your AI assistant chat (/tokenjar output on, /tokenjar output off).

---

🔌 Setup with Your AI Assistant

⚡ 1-Click Automatic Setup (Recommended)

Automatically detects and configures TokenJar MCP server across Claude Desktop, Cursor, Antigravity, Windsurf, Claude Code, and VS Code:

# 📦 One-time setup: adds to PATH, enables MCP in all IDEs, installs slash commands
tokenjar install

# 🟢 Turn on TokenJar in your current project (generates AGENTS.md)
tokenjar on

# 🌐 Or enable/disable MCP server globally across all detected IDEs
tokenjar enable
tokenjar disable

Manual Configuration

If you prefer to configure manually or use other clients:

claude mcp add tokenjar -- python -m tokenjar

Create or update .cursor/mcp.json:

{
  "mcpServers": {
    "tokenjar": {
      "command": "python",
      "args": ["-m", "tokenjar"],
      "env": { "PYTHONUNBUFFERED": "1" }
    }
  }
}

Add to ~/.gemini/config/mcp_config.json:

{
  "mcpServers": {
    "tokenjar": {
      "command": "python",
      "args": ["-m", "tokenjar"],
      "env": { "PYTHONUNBUFFERED": "1" }
    }
  }
}

Add to ~/.codeium/windsurf/mcp_config.json:

{
  "mcpServers": {
    "tokenjar": {
      "command": "python",
      "args": ["-m", "tokenjar"]
    }
  }
}

Add to .continue/config.yaml:

mcpServers:
  - name: tokenjar
    command: python
    args: ["-m", "tokenjar"]

---

🛠️ Available MCP Tools

• find_symbol_global(query, root_path=".", exact=False): Search for functions, methods, or classes across the entire codebase by name without reading multiple files. Features hybrid fuzzy matching with Levenshtein distance fallback for typo-tolerant lookups.
• find_symbol_references(symbol_name, root_path=".", max_results=25): Blast radius reference analyzer. Finds all callers, imports, and usages across the entire codebase before editing or refactoring code.
• tool_get_code_skeleton(file_path): Extract structural skeleton of a file — classes, function signatures, docstrings, and type annotations with bodies replaced by .... (Supports Python, JS/TS, Go, Rust, Java, C/C++, C#, Ruby, PHP, Kotlin).
• tool_get_symbol(file_path, symbol_name): Extract the full implementation of a specific class or function by name after inspecting its skeleton.
• read_file_smart(file_path, symbol=None, force_full=False, query="", start_line=None, end_line=None): Differential file reader with session caching, targeted line range slicing, symbol extraction, and Lockfile Shield. Pass symbol="function_name" to extract symbols directly in 1 step. Supports start_line and end_line (1-indexed, inclusive) to surgically inspect specific line ranges with line numbers and enforces an auto-paging ceiling (MAX_OUTPUT_LINES = 80). Returns [CACHED] unchanged (~3 tokens) or unified diffs on edits. For lockfiles (package-lock.json, Cargo.lock, etc.), pass query="package-name" for surgical 5-line version blocks instead of 50,000 lines.
• run_command_smart(command, cwd="."): Executes shell commands and prunes verbose logs from pytest, jest, npm, cargo, and git.
• filter_output(output, output_type="auto"): Pure text filter for test runners, build pipelines, and version control logs without executing commands.
• get_repo_map_tool(root_path=".", max_tokens=1000): Graph centrality codebase map prioritized by cross-file import relationships.
• get_directory_tree_tool(root_path=".", max_depth=4): Lightweight directory tree honoring .gitignore and skipping binary folders.
• cache_stats(): Inspect session read hits, misses, diffs, and aggregate token savings.

📦 MCP Resources & Prompts

• Resources:
  • tokenjar://stats: Live cumulative token and financial savings dashboard.
  • tokenjar://guide: AI assistant best-practice optimization guidelines.
  • tokenjar://config: Active project configuration and ignore settings.
• Prompts:
  • optimize_coding_task(task_description): System prompt template steering assistants toward token-efficient workflows.

---

⚙️ Project Configuration (tokenjar.toml)

Create an optional tokenjar.toml in your repository root to customize exclusions and budgets:

[general]
ignore_patterns = ["tests/fixtures/*", "legacy/*", "*.bak"]
max_cacheable_bytes = 5242880 # 5 MB

[cache]
ttl_days = 30
max_entries = 5000

[repo_map]
default_budget = 1000

---

💻 CLI Commands & Shell Hooks

TokenJar also functions as an interactive command-line utility for human developers and local shell automation:

# 📦 1-Click System Setup & Integrations
tokenjar install            # One-time setup: adds to PATH, enables MCP across detected IDEs, sets up slash commands
tokenjar on                 # Turn on TokenJar in current project (creates/updates AGENTS.md rules)
tokenjar off                # Turn off TokenJar in current project (cleans AGENTS.md rules)
tokenjar on --global        # Configure MCP server globally across all detected IDEs
tokenjar off --global       # Uninstall MCP server globally from all detected IDEs
tokenjar enable             # Enable TokenJar MCP server in all detected AI assistants globally
tokenjar disable            # Disable TokenJar MCP server from all AI assistants globally

# 📊 Monitoring, Telemetry & Web Dashboard
tokenjar status             # Check operational status across AI assistants and IDEs
tokenjar stats              # View live performance, token reduction, and financial savings dashboard
tokenjar ui                 # Launch the interactive Web Dashboard in your browser (Zero Background RAM)
tokenjar ui --port 4141     # Specify custom port for Web Dashboard

# 🔄 Automatic Updates & Self-Healing
tokenjar update             # Upgrade binary + auto-sync IDE configs + slash commands + project rules
tokenjar update --force     # Force re-installation even if already on latest version

# 🧹 Cache & Telemetry Management
tokenjar clean              # Reset telemetry statistics and clear L2 SQLite cache
tokenjar clean --cache      # Only clear L2 SQLite cache
tokenjar clean --stats      # Only reset telemetry statistics
tokenjar cache-prune --ttl-days 30 --max-entries 5000 # Prune expired or excess entries from L2 SQLite cache

# ⚡ Developer Tools & Output Pruning
tokenjar run "pytest"       # Run shell command with intelligent token-saving output pruning
tokenjar run "npm test"     # Retains errors & summary, prunes repetitive logs
RAW=1 tokenjar run "pytest" # Temporary raw bypass (or pass --raw)
tokenjar hook               # Install transparent CLI interceptor hooks into shell profiles (PowerShell/Bash)
tokenjar unhook             # Remove transparent CLI interceptor hooks from shell profiles
tokenjar output on          # Enable compact surgical diffs & zero-truncation mode
tokenjar output off         # Revert AI assistant to default unrestricted output

# ⚠️ Zero-Trace Surgical Uninstall
tokenjar uninstall          # Safely revert IDE configs, clean project rules, purge cache, hooks, and PATH
tokenjar uninstall --yes    # Skip confirmation prompt and purge immediately

---

🔒 Enterprise Privacy & Security Guarantee

TokenJar is built strictly under a Zero-Telemetry, 100% Localhost design philosophy:

• 100% Local Execution: All parsing (Tree-sitter), caching (SQLite), and output filtering happen locally in-process on your CPU.
• Zero External Network Calls: No telemetry servers, no analytical trackers, no outbound pings, and no cloud dependencies whatsoever.
• Air-Gapped Compatible: Safely operates in classified, offline, or air-gapped corporate enterprise environments.
• Local Data Isolation: Persistent cache (~/.tokenjar/cache.db) and statistics (~/.tokenjar/telemetry.json) reside exclusively in your user directory and can be purged at any time with tokenjar clean --stats or by deleting the directory.
• Non-Invasive Architecture: Never modifies your project code without explicit assistant direction.

---

🦀 Enterprise & High-Performance Native Engine (Rust Edition)

For enterprise environments, massive monorepos (50,000+ files), CI/CD pipelines, or developer systems without a Python runtime, TokenJar provides an ultra-fast, zero-dependency native Rust binary (tokenjar.exe / standalone executable).

Why the Enterprise Native Engine?

• Zero Runtime Dependencies: No Python, pip, Node.js, or virtual environments required. Single standalone binary.
• Ultra-Low Latency: Instant startup (~3ms cold start vs 300ms Python startup) for zero-delay MCP tool responses.
• High-Concurrency Indexing: True multithreaded (Rayon + Tokio) parallel code parsing and symbol extraction.
• Embedded 15-Language AST Engine: Built-in Tree-sitter parsers for Rust, C, C++, Go, C#, Java, Python, JavaScript, TypeScript, PHP, Ruby, Bash, HTML, CSS, JSON statically linked inside the binary.
• Minimal Memory Footprint: Consumes only ~8-15 MB RAM under active load.

Enterprise Quick Start (Standalone Binary)

Download the precompiled binary from GitHub Releases or build directly with Cargo:

# Build optimized native release binary from source
cargo build --release --workspace

# The standalone binary is ready:
./target/release/tokenjar.exe status

Enterprise MCP Configuration (claude_desktop_config.json / Cursor)

Point directly to the native binary without any Python wrapper:

{
  "mcpServers": {
    "tokenjar": {
      "command": "C:\\path\\to\\tokenjar.exe"
    }
  }
}

---

🌍 Supported Languages

TokenJar uses Tree-sitter for AST parsing and supports 130+ programming languages out of the box, including:

Python · TypeScript · JavaScript · Go · Rust · Java · C# · C / C++ · Ruby · PHP · Swift · Kotlin · Scala · Dart · Lua · Elixir · Haskell · and more.

---

🧪 Development & Quality Assurance

TokenJar maintains dual test suites ensuring 100% parity across both implementations:

# Python (Community Edition & MCP SDK)
pip install -e ".[dev]"
pytest tests/ -v           # 72 tests passing (100% pass)

# Rust (Enterprise Native Engine)
cargo test --workspace    # 45 core tests + 3 benchmark/stress suites passing (100% pass)

---

📄 License & Intellectual Property

Copyright © 2026 Ömer Faruk Eskitürk. All rights reserved.

Licensed under the Business Source License 1.1 (BSL 1.1) with an automatic transition to the Apache License, Version 2.0.

• Free Use: Free for all personal, educational, research, evaluation, and internal business use.
• Commercial Restrictions: Cannot be hosted or provided as a paid commercial service or SaaS competing with the Licensor.
• Sunset to Apache 2.0: Converts automatically to 100% open-source Apache 2.0 on 2030-01-01.

See LICENSE for full legal terms.