CascadeGateway MCP server
Commit Oct 14, 2026Routes AI queries between local GPUs and cloud models using intelligent hardware-adaptive cascading to minimize token costs.
Also known as jadberro/CascadeGateway
- Stars
- 4
- Tools
- —
- Downloads, 30 days
- —
- Latest version
- —
- Last commit
- Oct 14, 2026
- License
- MIT
What is CascadeGateway MCP?
CascadeGateway routes AI queries between local GPUs and cloud models using intelligent hardware-adaptive cascading to minimize token costs.
Summary generated from the project's README
About CascadeGateway MCP server
Developers looking to reduce cloud API costs while maintaining high-quality coding assistance can use CascadeGateway to route queries between local GPUs and frontier models. The system automatically detects your hardware, such as an RTX 5090 or Apple Silicon, to select the best model size for your available VRAM. It functions as an intelligent proxy that handles routine tasks like unit testing and refactoring on local hardware via Ollama, only falling back to cloud providers like Gemini or OpenAI when context limits are exceeded or complex reasoning is required.
This gateway integrates with popular IDEs like Cursor and VSCode through an OpenAI-compatible API or a native Model Context Protocol endpoint. It features a unique architect-builder workflow where a reasoning model like DeepSeek-R1 drafts a plan for your approval before a local builder like Qwen 2.5 Coder synthesizes the actual code. Users can monitor real-time VRAM usage, temperature, and dollar savings through a dedicated desktop HUD or a web-based control center.
What you can do with it
- 1Minimize cloud token expensesRoute routine coding, unit tests, and refactoring tasks to your local GPU to achieve zero token cost.
- 2Execute complex architectural planningUse a high-reasoning local model to design system architectures and data contracts before generating code.
- 3Verify code with cloud auditorsCombine fast local code generation with a strict cloud-based critique to ensure production-grade quality.
- 4Manage GPU resources for gamingInstantly clear all loaded AI models from your video memory to free up resources for other applications.
Written from the project's README. Every use case is checked against a quote from it.
Automated checks
Ran Nov 16, 2026Full report →Not verified yet. A server needs a score of 80, no serious finding, and enough data for us to check. How checks work · What to fix for more points
- Source repository linked and reachablejadberro/cascadegateway is public and reachablegithub.com/jadberro/cascadegateway ↗Pass
- Package names this repository as its sourceNot checked
- Build provenance (Sigstore attestation)Not checked
- Open source license (OSI)MITspdx.org/licenses/MIT.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-10-14github.com/jadberro/cascadegateway ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNot checked
- No critical advisory in the package or its dependenciesNot checked
- No hidden instructions to the model0 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNot checked
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubCascadeGateway ⚡
Hardware-Adaptive Local LLM & Cloud Cascading Architecture
CascadeGateway is an intelligent, hardware-adaptive cascading proxy that routes AI queries between local GPUs (via Ollama at $0 token cost) and frontier cloud models (Google Gemini, OpenAI, etc.).
It automatically detects your GPU hardware and physical VRAM on startup—whether running a flagship RTX 5090 (32GB), RTX 4090 (24GB), mainstream RTX 3080 (10GB), or Apple Silicon—and dynamically selects and sizes the optimal models without requiring manual reconfiguration.
flowchart TD
Client["AI Client: Cursor / Continue / Antigravity"] --> Gateway["CascadeGateway :8000/v1"]
Gateway --> Phase1{"Phase 1: Structural Guard"}
Phase1 -->|"Context > 32k or Tools"| CloudFallback["Cloud Fallback: Gemini"]
Phase1 -->|"Passed"| Phase2{"Phase 2: Lexical Automaton <1ms"}
Phase2 -->|"Architecture / Reasoning"| LocalArchitect["Local Architect: DeepSeek-R1 / Gemma 4 (RTX 5090)"]
Phase2 -->|"Coding / Tests / Routine"| LocalBuilder["Local Builder: Qwen 2.5 Coder 32B (RTX 5090)"]
LocalArchitect --> Phase3{"Phase 3: 3-Token Lookahead"}
LocalBuilder --> Phase3
Phase3 -->|"Stalled / Timeout"| CloudFallback
Phase3 -->|"Healthy Stream"| LoopBreaker["Sliding-Window Loop Breaker"]
LoopBreaker --> ClientStream["SSE Stream to Client"]
CloudFallback --> ClientStream
VerifyRoute["Asymmetric Verification: /verify"] --> LocalDraft["Local Generator: RTX 5090"]
LocalDraft --> CloudCritic["Cloud Critic: Gemini Flash"]
CloudCritic --> ClientStream
Gateway -. Telemetry .-> HUD["🖥️ Always-On-Top Desktop HUD"]
🚀 Key Features
- 🖥️ Always-On-Top Desktop HUD: Sleek, frameless, draggable obsidian overlay (
#0b0f19) that sits above your IDEs next to the clock or anywhere on screen. Displays live routing decisions (🟢 Local Builder, 🟣 Local Architect, 🔵 Asymmetric Verify, 🟠 Gemini Cloud), real-time VRAM gauge (GB, %, °C, W), decision latency (sub-5ms), prompt snippet, dollar savings counter, pin toggle, compact mini-mode, and 1-click mode switcher buttons. - 🧠 Local GPU Architectural Reasoning: True multi-model orchestration on local silicon. System design, race condition, thread safety, and algorithmic queries are dispatched directly to the local reasoning model (
deepseek-r1:14borgemma4:26b) on the RTX 5090 rather than escaping to cloud tokens. Routine code generation is routed toqwen2.5-coder:32b. - Hardware-Adaptive VRAM Sizing: Automatically profiles your GPU (
nvidia-smi/ Metal / CPU) and assigns the optimal model tier. - Autonomous $0 Offload: Resolves 85–95% of routine coding, unit tests, refactoring, and QA directly on local VRAM.
- Dynamic Token Biasing Engine:
- Adaptive Mode: Automatically increases local GPU bias as cloud token budgets get consumed.
- Manual Slider: Fine-tune local preference from 0% (Quality First) to 100% (Maximum Local Offload).
- Strict Local: 100% execution on local VRAM with automatic cloud overflow only if context exceeds physical limit.
- 🧠 Multi-Task Workflow Modes & Interactive Review Gate:
- 🧠 Architect & Builder: Reasoning model drafts architecture, data contracts, and edge cases → Pauses at a Human Review Gate → You approve or refine → Builder synthesizes production code.
- 🚀 Solo Sprint: Instant local coding via
qwen2.5-coder:32bfor quick functions, tests, and scripts. - 🌐 Deep Context: Gemini 2.5 Flash ingests massive repository files (1M context) → modular execution on local 5090.
- 🔬 Math & Algo Proof: Deep Chain-of-Thought formal verification for cryptography and concurrency algorithms.
- 🛡️ Asymmetric Verification (
/verify): High-speed local draft synthesis on RTX 5090 ($0) combined with an anonymous strict cloud auditor (Gemini) providing production-grade critique and code enhancement without leaking full history.
- 🛡️ Sub-5ms Intelligent Routing & 3-Token Lookahead Failover: Three-phase classification pipeline: Phase 1 Structural & VRAM Budget Validation (<1ms, prevents swapping to system RAM), Phase 2 Single-Pass Lexical Scan (<1ms), and Phase 3 Resilient Streaming with an in-memory 3-token lookahead buffer that transparently replays stalled local requests to Gemini Cloud without dropping client connections or throwing IDE error popups.
- 🔁 Anti-Hallucination Sliding-Window Loop Breaker: In-flight ring buffer tracking token emission sequences (lengths 2, 3, 4 repeated ≥ 4×), terminating runaway generative loops immediately.
- ⚡ 1-Click IDE Auto-Configuration: Automated zero-friction setup endpoint (
/api/ide/auto-config) and UI button for VSCode Continue (~/.continue/config.json) and Cursor. - 💰 Zero-Surprise Dollar Savings Telemetry: Hardware-native token accounting (
eval_count) calculating real-time dollar savings based on commercial frontier rates ($3.00/1M tokens). - ⚡ Persistent GPU VRAM Residency & Zero-Cold-Start: Models remain permanently resident in GPU VRAM (
keep_alive: -1) across idle periods and tasks, ensuring instant zero-delay responses. Includes manual 1-Click "Pause GPU" (instant VRAM purge for AAA gaming) and "Warm GPU" (preloads weights into VRAM). - Native Model Context Protocol (MCP): Exposes a high-performance Streamable HTTP and Stdio MCP endpoint for Google Antigravity and Claude Desktop.
- 🎛️ Dynamic Model Selection: Select any installed local model for both the Architect role (
deepseek-r1:14b,gemma4:26b, etc.) and the Builder role (qwen2.5-coder:32b, etc.) directly from the Web UI toolbar or Windows Tray submenus. Automatically detects newly pulled models from Ollama without restarting the gateway. - Zero-Window Background Tray App: Runs silently in the system tray, boots with Windows/Linux, and includes single-instance mutex protection.
- 🎮 1-Click Pause & Free GPU (Instant VRAM Purge): Evicts loaded models from VRAM in <1s via a dedicated button on the Web UI, Windows Tray, or
POST /api/models/unload. Frees 20–30+ GB of VRAM immediately for AAA gaming, Blender, or video editing without terminating the server. Models reload automatically on demand when coding. - Live Hardware Telemetry: In-browser control center showing real-time VRAM allocation, GPU power draw (W), temperature (°C), lifetime token savings, and an interactive prompt runner.
- Standard OpenAI-Compatible API: Seamless drop-in replacement (
/v1/chat/completions) for Cursor, VSCode (Continue.dev), Aider, Claude Dev, and custom scripts.
🧠 The Interactive Review Gate
sequenceDiagram
autonumber
actor Dev as You (Developer)
participant Arch as 🧠 Architect - DeepSeek-R1 / Gemma 4
participant Gate as 🛑 Review Gate - You in the Loop
participant Build as ⚡ Builder - Qwen 2.5 Coder 32B
Dev->>Arch: "Add rate-limiting and burst protection to API routes"
Note over Arch: Deep CoT: Identifies race conditions,<br/>evaluates Redis vs in-memory,<br/>drafts interface contracts.
Arch->>Gate: Presents Architectural Blueprint + Edge Cases
Note over Gate: PAUSE: No code written yet.<br/>You review the proposed interfaces & strategy.
alt If you want adjustments
Dev->>Gate: "Use Redis, and add IPv6 CIDR subnet matching"
Gate->>Arch: Quick amendment - 100 tokens
Arch->>Gate: Updated spec
end
Dev->>Gate: "Approve & Build"
Gate->>Build: Sends final structured specification
Note over Build: Zero ambiguity.<br/>High-speed code synthesis - 70 t/s.
Build-->>Dev: Delivers complete implementation + unit tests - $0 Cost
🏛️ Hybrid Architect-Builder Execution (Google Antigravity & CascadeGateway)
When running agentic coding assistants like Google Antigravity on a Pro plan, repeatedly shipping 20k to 50k tokens of repo file dumps, compiler logs, and multi-turn message history quickly exhausts 5-hour quota refill blocks.
CascadeGateway implements the native Hybrid Architect-Builder Pattern:
flowchart TD
subgraph Cloud ["☁️ Cloud Tier (Pro Plan Quota)"]
Architect["Cloud Architect: Gemini Pro / Claude<br/>• High-level system decomposition<br/>• Milestone roadmap planning<br/>• Concise diff reviews"]
end
subgraph Gateway ["⚡ CascadeGateway :8000/v1"]
Router["Context Pruning & Intent Router"]
Clamp["Terminal Clamping (>60 lines → 15 head + 30 tail)"]
Dedup["Repeat File Buffer Deduplication"]
end
subgraph Local ["🖥️ Local Workstation Tier (NVIDIA RTX 5090 - 32GB VRAM)"]
Builder["Local Builder (vLLM / Ollama): Qwen 2.5 Coder 32B<br/>• Code generation, unit tests, refactoring<br/>• Iterative bug fixes & lint diagnostics<br/>• $0 Cloud Token Cost"]
LocalMCP["Local MCP Tools<br/>• AST Symbol Lookup (@llmtooling/code-search-mcp)<br/>• Targeted File Slicing"]
end
Architect -->|"Milestone Specifications"| Router
Router --> Clamp
Clamp --> Dedup
Dedup -->|"Iterative Coding Tasks"| Builder
Builder <-->|"Precise Symbol Queries"| LocalMCP
Dedup -. Escalation on repeated test failures .-> Architect
📉 Token Efficiency & Telemetry Audit Benchmark
Benchmarked using python -m cascadegateway.audit --compare baseline.jsonl optimized.jsonl:
| Metric | Baseline (Cloud Pass-Through) | Optimized (Architect-Builder on 5090) | Impact |
|---|---|---|---|
| Total Task Turns | 15 turns | 16 turns | Consistent scope |
| Average Prompt Tokens / Turn | 38,200 tokens | 3,897 tokens | -89.8% context bloat |
| Tokens Billed to Cloud Quota | 579,300 tokens | 3,550 tokens | -99.4% Cloud Token Reduction |
| Tokens Absorbed by Local GPU | 0 tokens | 68,500 tokens | 95.1% Local RTX 5090 Execution Ratio |
| Prefix Cache Hit Rate | 0.0% | 76.3% | Instantaneous prefill |
| Average Latency | 1,450 ms | 800 ms | Sub-second response |
🚀 Cross-Platform Local Engine Runners
-
Windows Native (Zero-Window Flash):
powershell -File scripts\start_local_engine.ps1Auto-detects Ollama or vLLM, warms up
qwen2.5-coder:32bin 32GB VRAM withkeep_alive: -1. -
Linux / WSL (vLLM with Automatic Prefix Caching):
./scripts/start_local_vllm.shLaunches
vllmwith--enable-prefix-caching,--enable-chunked-prefill, and--tool-call-parser hermes. -
Antigravity SDK & Standalone Orchestration:
python scripts/run_architect_builder.py --task "Draft and build target module"
📊 VRAM Hardware Sizing Matrix
| Hardware Tier | Supported GPUs | VRAM | Recommended Coding Model | General / Reasoning | Expected Speed |
|---|---|---|---|---|---|
| Tier 1 (Flagship) | RTX 5090, 4090, 3090, A6000 | 24 GB – 32 GB | qwen2.5-coder:32b |
gemma4:26b / deepseek-r1:32b |
60 – 75+ tokens/sec |
| Tier 2 (Enthusiast) | RTX 4080, 4070 Ti, 4070, 3080 12GB | 12 GB – 16 GB | qwen2.5-coder:14b |
mistral-small:22b-q4 |
45 – 55+ tokens/sec |
| Tier 3 (Mainstream) | RTX 3080 10GB, 3070, 4060 Ti, 2080 Ti | 8 GB – 10 GB | qwen2.5-coder:7b |
llama3.1:8b |
85 – 100+ tokens/sec |
| Tier 4 (Entry / CPU) | RTX 3050, 4060 laptop, Apple Silicon, CPU | < 8 GB | qwen2.5-coder:1.5b / 3b |
llama3.2:3b |
40 – 70 tokens/sec |
⚡ Quick Start
Prerequisites
- Ollama installed and running on your machine.
- Python 3.10+ installed.
- (Optional) A free API key from Google AI Studio for cloud fallback.
Windows (One-Click)
git clone https://github.com/jadberro/CascadeGateway.git
cd CascadeGateway
scripts\setup.bat
Creates .venv, installs dependencies, auto-generates your desktop shortcut, and places an auto-start shortcut in your Windows Startup menu.
Linux / macOS
git clone https://github.com/jadberro/CascadeGateway.git
cd CascadeGateway
chmod +x scripts/setup.sh
./scripts/setup.sh
🛠️ Connecting Your IDEs & Tools
Once running, the control center is available at http://127.0.0.1:8000/.
1. Google Antigravity
Add to ~/.gemini/config/mcp_config.json:
{
"mcpServers": {
"rtx5090-cascade": {
"serverUrl": "http://127.0.0.1:8000/mcp"
}
}
}
Antigravity automatically detects the toolset (query_local_5090, cascade_llm, get_cascade_metrics) and dynamically dispatches code generation directly to your GPU.
2. Cursor
- Navigate to Settings > Models > OpenAI API Key.
- Set Base URL:
http://127.0.0.1:8000/v1 - Set API Key:
cascading-local(any string) - Model:
cascade-auto
3. Continue.dev (config.json)
{
"models": [
{
"title": "Cascade Gateway (Local + Cloud)",
"provider": "openai",
"model": "cascade-auto",
"apiBase": "http://127.0.0.1:8000/v1",
"apiKey": "cascading-local"
}
],
"tabAutocompleteModel": {
"title": "Local Autocomplete",
"provider": "openai",
"model": "cascade-auto",
"apiBase": "http://127.0.0.1:8000/v1",
"apiKey": "cascading-local"
}
}
4. Claude Desktop (claude_desktop_config.json)
{
"mcpServers": {
"local-cascade": {
"serverUrl": "http://127.0.0.1:8000/mcp"
}
}
}
5. Python OpenAI SDK
from openai import OpenAI
client = OpenAI(
base_url="http://127.0.0.1:8000/v1",
api_key="cascading-local"
)
response = client.chat.completions.create(
model="cascade-auto",
messages=[
{"role": "user", "content": "Write an optimized LRU cache in Python."}
]
)
print(response.choices[0].message.content)
📁 Repository Architecture
CascadeGateway is structured as a modern, modular Python package:
CascadeGateway/
├── src/
│ └── cascadegateway/
│ ├── core/ # Hardware profiler, model sizing & routing engine
│ │ ├── config.py # YAML config & environment loader
│ │ ├── hardware.py # GPU VRAM auto-profiling (5090/4090/etc.)
│ │ ├── classifier.py # Phase 1 structural validation & Phase 2 regex automaton (<1ms)
│ │ ├── streaming.py # Phase 3 lookahead failover, loop breaker & asymmetric audit
│ │ └── router.py # Biasing engine, metrics & model resolution
│ ├── api/ # Modular FastAPI endpoints
│ │ ├── server.py # App initialization, SSE streaming & /v1/chat/completions
│ │ ├── pipeline.py # Architect, Review Gate & Builder endpoints
│ │ └── models.py # Pause & Free GPU (VRAM purge) & 1-click IDE auto-config
│ ├── web/ # Clean decoupled web assets & responsive UI
│ │ ├── templates/
│ │ │ └── dashboard.html # Responsive HTML5 dashboard & real-time telemetry
│ │ └── static/
│ │ ├── css/dashboard.css
│ │ └── js/dashboard.js
│ ├── hud/ # Always-On-Top Desktop HUD & Telemetry Overlay
│ │ ├── __init__.py
│ │ └── overlay.py # Tkinter floating dark obsidian window
│ ├── tray/ # Windows system tray controller
│ │ └── app.py # Pystray background service & menus
│ ├── mcp/ # Model Context Protocol endpoints
│ │ ├── __init__.py
│ │ └── server.py # Antigravity & Claude Desktop integration
├── scripts/ # Platform launchers & utilities
│ ├── start_cascade.bat # Windows batch launcher
│ ├── start_tray.vbs # Silent windowless tray runner
│ ├── start_hud.bat # Desktop HUD launcher
│ ├── start_hud.vbs # Silent windowless HUD launcher
│ ├── setup.bat / setup.sh # One-click installers
│ └── create_shortcuts.ps1 # Desktop & startup shortcut generator
├── tests/ # Comprehensive automated test suites
│ ├── test_gateway.py # End-to-end API integration tests
│ ├── test_resilience.py # Sub-5ms SLA, structural validation & lookahead tests
│ └── test_new_features.py # Loop breaker, IDE auto-config, dollar savings & /verify tests
├── assets/ # Icons & diagrams
├── config.yaml # Gateway configuration
├── pyproject.toml # Modern pip/uv packaging metadata
└── requirements.txt
📡 API Reference
| Endpoint | Method | Description |
|---|---|---|
/v1/chat/completions |
POST |
OpenAI-compatible chat completion endpoint supporting resilient SSE streaming, lookahead failover, loop breaking, and diagnostic headers (X-Cascade-*). |
/v1/models |
GET |
Returns available virtual model aliases (cascade-auto, local-5090) and physical Ollama models. |
/api/hud/state |
GET |
Desktop HUD Telemetry: Real-time status, active route, model, VRAM used/total, temp (°C), power (W), latency, and dollar savings. |
/api/hardware |
GET |
Real-time GPU telemetry: VRAM allocation, temperature, power draw (W), and detected hardware tier. |
/api/models/unload |
POST |
Instant VRAM Purge ("Pause GPU"): Evicts loaded models to 0 MB VRAM in <1s for AAA gaming or rendering. |
/api/models/preload |
POST |
Warm GPU: Preloads and pins model into VRAM with indefinite residency (keep_alive: -1). |
/api/ide/auto-config |
POST |
1-Click IDE Setup: Injects CascadeGateway configuration into ~/.continue/config.json and detects Cursor environments. |
/api/workflow/mode |
POST |
Sets active workflow mode (architect, builder, solo, verify, deep_context, math). |
/metrics |
GET |
Telemetry: exact hardware token accounting (eval_count), local offload percentage, and real-time dollar savings. |
/mcp |
POST |
Model Context Protocol streamable HTTP endpoint for Google Antigravity and Claude Desktop. |
/ |
GET |
Interactive browser control center, real-time hardware gauges, and prompt workbench. |
⚙️ Configuration (config.yaml)
server:
host: "127.0.0.1"
port: 8000
local:
provider: "ollama"
base_url: "http://127.0.0.1:11434"
# Set empty to let the hardware profiler auto-select the best model for your GPU
primary_model: ""
max_context_tokens: 32768
timeout_seconds: 120
cloud:
provider: "gemini"
default_model: "gemini-2.5-flash"
pro_model: "gemini-2.5-pro"
max_context_tokens: 1048576
biasing:
mode: "adaptive" # "adaptive", "manual", or "local_only"
bias_factor: 0.35 # Baseline local bias (0.0 to 1.0)
cloud_token_budget: 500000
allow_context_overflow_to_cloud: true
📜 License
This project is licensed under the MIT License.
CascadeGateway MCP questions
How does CascadeGateway save money on AI tokens?
It routes routine coding, unit tests, and refactoring tasks to your local GPU via Ollama at $0 token cost, only using cloud models for complex reasoning or large contexts.
Does it support different types of hardware?
Yes, it automatically profiles your GPU (NVIDIA, Metal, or CPU) and assigns the optimal model tier based on available VRAM.