icycache MCP server
Commit Nov 3, 2026Acts as a multi-upstream LLM gateway that provides normalization, semantic caching, and routing for local and cloud AI providers.
Also known as MlkBlk-IT/icycache
- Stars
- 1
- Tools
- 7
- Downloads, 30 days
- —
- Latest version
- —
- Last commit
- Nov 3, 2026
- License
- MIT
What is icycache MCP?
icycache acts as a multi-upstream LLM gateway that provides normalization, semantic caching, and routing for local and cloud AI providers. It exposes 7 tools, such as get_cache_stats, clear_cache, normalize_preview.
Summary generated from the project's README
About icycache MCP server
Developers seeking to reduce latency and costs by avoiding redundant LLM computations can use icycache as a multi-upstream gateway. It sits between any client, such as Claude Desktop, Cursor, or custom SDKs, and various providers including OpenAI, Anthropic, Ollama, and LM Studio. By acting as a single entry point for all traffic, it ensures that identical or semantically similar requests are served instantly from a local SQLite cache rather than re-running expensive model generations.
The system provides normalization to ensure that slight variations in phrasing do not bypass the cache. It supports 24 languages through a multilingual anchor engine and includes an opt-in semantic cache that utilizes embeddings and a Qdrant instance. Users can connect via stdio, SSE, or Streamable HTTP. The gateway manages 18 different upstreams, such as Groq, Mistral, and Google AI Studio, while providing real-time observability through a local web dashboard and WebSocket events.
What you can do with it
- 1Reduce redundant API costsA developer can route all LLM traffic through the gateway to serve cached responses for repeated prompts instead of burning tokens.
- 2Implement semantic request matchingAn AI client can use the semantic cache to recognize that two differently phrased questions are asking for the same information.
- 3Monitor multi-provider usageTeams can use the dashboard to track real token usage and latency across different providers like Anthropic and OpenAI.
Written from the project's README. Every use case is checked against a quote from it.
icycache MCP tools 7
Detected from the READMEget_cache_statsclear_cachenormalize_previewsearch_cached_promptsget_tool_cache_statsclear_tool_cachecheck_tool_loop
Automated checks
Ran Nov 14, 2026Full report →Not verified yet. A server needs a score of 80, no serious finding, and enough data for us to check. How checks work · What to fix for more points
- Source repository linked and reachablemlkblk-it/icycache is public and reachablegithub.com/mlkblk-it/icycache ↗Pass
- Package names this repository as its sourceNot checked
- Build provenance (Sigstore attestation)Not checked
- Open source license (OSI)MITspdx.org/licenses/MIT.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-11-03github.com/mlkblk-it/icycache ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNot checked
- No critical advisory in the package or its dependenciesNot checked
- No hidden instructions to the model7 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNo tool named like delete, drop or execPass
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHub⚠️ NOTICE — The rest of this repository (code, comments, scripts, docs, commits, issues) is written in French. Only this README is in English.
v0.4.0 — multilingual semantic anchors + a test suite that runs (2026-11-03)
Improvements
pytest tests/runs again. It used to exit with code 1 and zero output: three test files were scripts, not pytest modules —test_cacheforge*.pycallsys.exit()at import whenCF_BASEis missing, andtest_prefix_proxy.pyends withos._exit(), which killed the whole pytest process.granite_live_test.pymatched*_test.pyand was collected, starting a live LM Studio benchmark during collection.- The two async tests can pass now —
pytest-asynciowas missing from thedevextra and from the virtualenv (async def functions are not natively supported).- "plus de" is no longer corrupted.
normalize_numbers()replaced" plus "with"+"everywhere: "Donne-moi plus de détails" became "Donne-moi+de détails", changing the cache key and the embedding input. Operators are now converted only between two numeric operands, in a single implementation shared byicy_normalize.pyandicy_semantic.py.- Path anchors no longer swallow every slash (
"A/B"→"/b","12/24"→"/24"), so two paraphrases can finally share their hard anchors.- The entity guard is back on the MCP path. It compared the normalised (lower-case) prompt and therefore saw zero soft anchors — "Montreal" and "Quebec City" were no longer distinguished.
- The
RELAXacceptance path now requires shared grounding (identical hard anchors, or one common strong entity), so those two are no longer accepted at 0.99 similarity.What's new
- A multilingual anchor engine —
src/icy_lang.py+config/lang_profiles.json, 24 languages. Casing only marks an entity in some languages, so the engine tokenises per script (latin / CJK / cyrillic / arabic / devanagari / bengali), detects the language, and applies three casing models:sentence(en, fr, es, pt, it, vi, tr — a capitalised word outside a sentence start is an entity),noun(de, nl, pl, sv, da, cs, hu, fi, no, ru, uk — all nouns are capitalised, so they are filtered),none(zh, ja, ko, ar, hi, bn — no casing, only Latin acronyms and CamelCase count). No language is hardcoded in the source any more — adding one is a single JSON key.- Per-language soft anchors on the MCP path, taken from the raw text.
tests/conftest.pyisolates the test scripts from pytest collection;pyproject.tomlgains[tool.pytest.ini_options];pytest-asynciojoins thedevextra.- 84 tests pass — the suite was previously unrunnable.
⚠️ Breaking: the cache key format changed. The entries already in
db/cache.dbwere written with the old normalisation and will never be read again (TTL 30 days, orDELETE FROM cache; VACUUM;).✅ SWAP DONE — 2026-10-24. IcyCache was renamed from CacheForge. The universal multi-upstream gateway is now the single active service on
:8080, and the historical server holds the MCP layer.
src/server.py→ the gateway (18 upstreams inconfig/icycache.gateway.json: OmniRoute, LM Studio, Ollama, OpenAI, Anthropic, NVIDIA, Groq, Cerebras, OpenRouter, Kilo, Nous, Copilot, AI Gateway, HuggingFace, Google AI Studio, LLM7.io, Mistral, Cloudflare).src/mcp_server.py→ the MCP layer (SSE:8081, Streamable HTTP:8084, stdio).- Both start together via
tray.py;:8080is the single entry point for all LLM traffic.- Legacy entry point kept:
python src/server.py --port 8090for isolated testing.Update (2026-10-31) — proxy robustness
- Legacy cache keys (built before
_tools_sig) still hit, so v1 cache entries are not orphaned.- A client disconnect mid-stream is detected and logged; an aborted stream is not cached.
- Upstream errors now return the upstream's own status and body instead of a generic 502.
DELTA-002 — upstream resilience + opt-in semantic cache (2026-10-26)
- BUG-001 — 502 stuck after a 400: the httpx client is now drained, recycled, and the request retried once (
src/upstream_client.py). The gateway recovers instead of staying stuck.- Number normalization:
"two plus two"→2+2. 11 new rules (1111 → 1122).- Semantic cache: implemented (
src/icy_semantic.py) but off by default (semantic_enabled: false).- Tool registry: the 100 verified MCP tool anchors (DELTA-001) are registered and asserted at startup.
Available MCP transports (all three active at once)
Transport Endpoint Target client stdio python mcp_stdio.pyClaude Desktop, IDEs, any stdio client SSE (legacy) http://127.0.0.1:8081/sseHermes, older clients Streamable HTTP (standard) http://127.0.0.1:8084/mcpRecent MCP clients, MCP Inspector In stdio mode, go through
mcp_stdio.py: it reserves stdout for JSON-RPC. (Nothing is written to stdout anymore, but the wrapper remains the safety net.)
🔥 IcyCache — Universal LLM Gateway with local cache
"Forge the cache, not the answer." HTTP gateway that sits between any client and any upstream (local or cloud), caches responses and serves them back instantly — no CPU regeneration.
📖 Full configuration guide :
docs/CONFIGURATION.md— multi-upstream routing, per-client setup (Hermes, Claude Desktop, Cursor, VS Code, Ollama, direct SDK), MISS→HIT proof, troubleshooting, firewall enforcement.
What's new in v0.4.0
| Area | Change |
|---|---|
| Semantic cache | Multilingual anchor engine — 24 languages (src/icy_lang.py + config/lang_profiles.json). No language is hardcoded in the source any more: adding one is a single JSON key. |
| Tests | pytest tests/ runs again — 84 tests pass (the suite was previously unrunnable). pytest-asyncio added to the dev extra. |
| Normalization | "plus de" is no longer turned into "+" (it corrupted French prompts, the cache key and the embedding). One shared implementation for icy_normalize.py and icy_semantic.py. |
| Anchors | Paths no longer swallow every slash ("A/B" → "/b"). The named-entity guard is active again on the MCP path, and the RELAX path requires shared grounding. |
Full details in the notice at the top of this page and in the v0.4.0 release notes.
Dashboard

Read-only web dashboard (http://127.0.0.1:8080/dashboard) on a local instance, captured 2026-10-31.
The problem
| Problem | Consequence |
|---|---|
| Latency | An identical request burns CPU/RAM again for an already-computed answer |
| No semantic cache | Standard systems hash the raw prompt: Explain Hyper-V and Please explain Hyper-V thanks generate two distinct requests |
| No observability | Impossible to measure the tokens and time actually saved |
| Traffic bypassing the proxy | Every client talking directly to its provider breaks the cache promise |
IcyCache attacks all four: normalization before hashing, SQLite cache with TTL, exportable metrics, and a single entry point for all LLM traffic.
Semantic cache status: implemented but opt-in (
semantic_enabled: falseby default). It needs an embedding endpoint (LM Studio on:1234) and a Qdrant instance; when either is missing the cache degrades silently to a miss instead of failing the request. Seeconfig/semantic_cache.json.
Target architecture
Any client (Hermes, Claude Code, scripts, IDE, SDK, CLI)
base_url = http://127.0.0.1:8080
│
▼
┌───────────────────────────────────────────────────────────────────────┐
│ ICYCACHE GATEWAY 127.0.0.1:8080 (src/server.py) │
│ │
│ ROUTING URL prefix /<upstream>/v1/... │
│ header X-IcyCache-Upstream: lmstudio │
│ body "upstream": "ollama" │
│ default gateway.default_upstream (config) │
│ │
│ NORMALIZATION NFKD → lowercase → Aho-Corasick → accents → punctuation│
│ CACHE KEY SHA256 of the NORMALIZED payload (+ upstream + endpoint)│
│ DECISION HIT → instant response | MISS → forward + save │
│ TOKENS REAL usage from the upstream. Never an estimate. │
│ OBSERVABLE WebSocket /ws/events : live deltas + final │
│ MCP (facade) stats / clear / search / realtime / tool-cache │
└───────────────────────────────────────────────────────────────────────┘
│ ▲ │
▼ │ same path back │
{ OmniRoute | LM Studio | Ollama | OpenAI | Anthropic | … }
│
▼
SQLite db/cache.db (events: upstream, real tokens, latencies, ttft)
Rule: everything that leaves a client and comes back to it goes through IcyCache. OmniRoute is no longer the entry point: it is one upstream among others.
Components
| File | Role |
|---|---|
src/server.py |
Universal multi-upstream gateway (FastAPI): routing, cache, real tokens, WebSocket |
src/mcp_gateway.py |
MCP facade of the gateway: 7 tools, tool-cache, loop guard, mirror REST API |
src/icy_core.py |
Shared core: DB (events, agent_events, tool_cache), WebSocket hub, loop guard |
src/icy_normalize.py |
Standalone normalization engine (faithful port, no import side effects) |
src/icy_cache_store.py |
LLM cache store (compatible cache table, additive raw_hash column) |
src/upstream_utils.py |
Usage normalization, headers, cache_control/prompt_cache_key mutation, SSE meter |
src/mcp_stdio_bridge.py |
stdio bridge: reserves stdout for MCP JSON-RPC, logs and prints to stderr |
src/mcp_server.py |
MCP server (FastMCP): tools, tool-cache, loop guard — listens on :8081 (SSE) / :8084 (HTTP) |
config/icycache.gateway.json |
Gateway config: upstreams, cache, loop guard, tool cache |
config/normalization_rules.json |
1122 normalization rules (incl. number words → digits) |
src/upstream_client.py |
Resilient upstream client — one pool per base URL, drain + recycle + retry |
src/icy_semantic.py |
Opt-in semantic cache — embeddings + Qdrant store, silent miss when unavailable |
config/semantic_cache.json |
Semantic cache settings (semantic_enabled, thresholds, embedder URL) |
config/tools_registry.json |
Registry of the 100 verified MCP tool anchors (DELTA-001) |
tools/smoke_gateway.py |
Real multi-upstream smoke test (isolated ports, temp DB) |
tools/enforce_gateway.ps1 |
Firewall rules: makes bypass impossible |
tools/verify_enforcement.py |
Proof: can the client still reach an upstream directly? |
tests/test_gateway.py |
10 isolated tests (mock upstream, temp DB) |
tests/test_semantic_cache.py |
Semantic cache unit tests |
tests/test_upstream_recovery.py |
Upstream recovery / retry tests (BUG-001) |
tests/test_anchors.py |
Normalization anchor tests (DELTA-002) |
Routing: pick your upstream
| Method | Example | Priority |
|---|---|---|
| URL prefix | POST http://127.0.0.1:8080/lmstudio/v1/chat/completions |
1 |
| Header | X-IcyCache-Upstream: lmstudio |
2 |
| Body | {"upstream": "lmstudio", ...} |
3 |
| Config default | gateway.default_upstream |
4 |
No implicit default in the code: without an explicit source, the request is rejected with
400 listing the available upstreams. An unknown upstream is rejected with 404.
HTTP routes (gateway)
| Route | Method | Role |
|---|---|---|
/v1/chat/completions |
POST | OpenAI-compatible (streaming and non-streaming) |
/v1/messages |
POST | Native Anthropic (kind: anthropic) |
/v1/embeddings |
POST | Embeddings (cached) |
/v1/models |
GET | Upstream models |
/{upstream}/v1/{rest} |
* | Explicit routing by prefix |
/ws/events |
WS | Realtime events: live delta + final (real tokens) |
/health, /api/status |
GET | Health, reachable upstreams |
/api/metrics, /api/metrics/series, /api/metrics/export |
GET | Metrics (dashboard shapes) |
/api/agent-events |
GET | Agent / tool-cache events |
/api/cache-clear |
POST | Empty the cache — header X-IcyCache-Confirm: FORCE_DELETE |
/dashboard |
GET | Web dashboard (read-only) |
Confirmation headers: X-IcyCache-Confirm (unified naming). The old X-CacheForge-Confirm
is still accepted by the gateway for compatibility.
Tokens: the rule of the real
| Source | Value |
|---|---|
usage field of the upstream response |
always authoritative |
| OpenAI streaming | stream_options.include_usage=true injected → usage from last chunk |
| Anthropic streaming | message_start + message_delta |
| Usage absent | NULL + tokens_source: "n/a" — never an estimate |
| HIT | saved_prompt_tokens / saved_completion_tokens = real usage of the 1st call |
events columns: upstream, client, prompt_tokens, completion_tokens, cached_tokens,
cache_write_tokens, latency_ms, ttft_ms.
MCP tools (facade)
MCP server (src/mcp_server.py) — 11 tools
| Tool | Role |
|---|---|
get_cache_stats |
Entries, hits, breakdown by upstream/model |
clear_cache |
Empty the cache (confirm='FORCE_DELETE') |
normalize_preview |
Preview of normalization + cache key |
search_cached_prompts |
Search in normalized prompts |
proxy_tool |
Call a tool on a target MCP server (tool calling) |
proxy_tools |
Batch call of target tools |
list_proxy_servers |
Routable MCP servers |
list_proxy_tools |
Tools exposed by a target server |
get_tool_cache_stats |
Tool-cache state + agent events |
clear_tool_cache |
Empty the tool cache (confirm='FORCE_DELETE') |
check_tool_loop |
Detect an agent loop in an LLM history |
Gateway server (src/mcp_gateway.py) — 11 tools
The 7 observability tools (get_cache_stats, clear_cache, normalize_preview,
search_cached_prompts, get_realtime_tokens, list_upstreams, tool_cache_stats)
plus the 4 relay tools (proxy_tool, proxy_tools, list_proxy_servers,
list_proxy_tools).
MCP is a facade: it observes and drives. It is not in the request path (an MCP client calls its tools deliberately — you cannot intercept an OpenAI/Anthropic stream with it). The request path is the HTTP gateway.
Agent loops: icy_core.llm_loop_check() detects the same tool called again with the same
arguments and the same results (Anthropic and OpenAI formats). Threshold: 3 repetitions = warning.
Tool calling: safety rules
The tool proxy (src/tool_proxy.py) relays calls to target MCP servers.
Three non-negotiable rules:
- An MCP error is never cached and never returns
ok: true. AnisError: truebecomesok: falsewith the real message (no opaqueExceptionGroup). - Write tools are never cached. Replaying a cached response
must have no side effect; otherwise a write is silently skipped.
cache_toolspresent → strict allowlist (only tools listed astrueare cached).- otherwise
cache: true→ anti-write heuristic (create_,write_,delete_,push_, … excluded). - otherwise → no cache.
- MCP sessions are persistent (one process per server, not per call).
Without this, every tool call respawned a process: 21s measured on
npx, 12s onuvx.
Example
// via any MCP client
{"server": "github", "tool": "search_repositories", "args": {"query": "icycache"}}
// -> {"ok": true, "cached": false, "latency_ms": 4867, "result": {...}}
Target server configuration
config/icycache.json → tool_proxy.servers. Two portability tokens:
| Token | Value |
|---|---|
{PROJECT_ROOT} |
project root (moving IcyCache needs no edit) |
{PYTHON} |
project venv interpreter (the only one with fastapi/mcp) |
On Windows, .cmd commands (like npx) are resolved automatically:
CreateProcess cannot launch an npx.cmd script without a shell.
Enforcement: making bypass impossible
# 1. Install the rules (ELEVATED PowerShell)
powershell -ExecutionPolicy Bypass -File tools\enforce_gateway.ps1 -Action apply `
-TargetProgram "C:\Users\Tech\AppData\Local\hermes\hermes-agent\apps\desktop\release\win-unpacked\Hermes.exe" `
-AllowPort 8080 -BlockPorts 20128,1234,11434
# 2. The proof: can the client still reach an upstream directly?
python tools\verify_enforcement.py --program "D:\IcyCache\venv\Scripts\python.exe" --allow-port 8080
Strategy (Windows Firewall, per program): block upstream local ports including on
127.0.0.1, block all remote traffic, explicitly allow 127.0.0.1:8080.
The verify_enforcement.py program runs with the client binary (rules are per program)
and checks that only IcyCache remains reachable.
Environment variables (unified naming)
| Variable | Role | Old name (still read by src/mcp_server.py for ICYCACHE_*) |
|---|---|---|
ICYCACHE_DB |
Cache DB path | CACHEFORGE_DB |
ICYCACHE_LOG |
Log path | CACHEFORGE_LOG |
ICYCACHE_CONFIG |
Config file (gateway) | — |
ICYCACHE_TARGET |
Upstream override (isolated tests) | CACHEFORGE_TARGET |
ICYCACHE_TOOLPROXY_CONFIG |
Tool-proxy config (default: config/icycache.json) |
— |
Installation
Published: the repository, PyPI distribution and scoped npm package are available publicly. The Python package is
icycache. The public npm package is scoped as@mlkblkit/icycachebecause the unscoped name was unavailable; it exposes bothicycacheandicycache-mcp.
uv tool install icycache # or: pip install icycache
| Registry | Package | Page |
|---|---|---|
| PyPI | icycache |
pypi.org/project/icycache |
| npm | @mlkblkit/icycache |
npmjs.com/package/@mlkblkit/icycache |
This gives you the icycache console script:
icycache serve # the gateway, http://127.0.0.1:8080
icycache mcp # the MCP layer (SSE :8081 / HTTP :8084 / stdio)
icycache doctor # diagnostic: paths, config, upstreams, database
Installed packages never write into site-packages: db/ and logs/ go to
%LOCALAPPDATA%\IcyCache (Windows) or ~/.local/share/IcyCache.
Full guide: docs/INSTALL.md.
npm (Node-only clients)
The npm package is icycache — the same name as the repository. It exposes two binaries:
npx -y icycache --help # the CLI (serve / mcp / doctor)
npx -y -p icycache icycache-mcp # MCP server on stdio
From source (Windows)
# 1. Environment (Python 3.12.10)
python -m venv venv
.\venv\Scripts\Activate.ps1
pip install -e ".[dev]"
# 2. Manual start
.\start_icycache.ps1
# 3. Start at login (Task Scheduler, or Startup fallback without admin)
.\install_autostart.ps1
Build the packages yourself
python -m build # -> dist/icycache-<version>-py3-none-any.whl + .tar.gz
Building and publishing this package: see
PUBLISHING.md.
Tests
# Full suite (unit + integration + multilingual anchors) — no network, no LM Studio
.\venv\Scripts\python.exe -m pytest tests\ -v
# Isolated suite (mock upstream, temp DB, no port occupied)
.\venv\Scripts\python.exe -m pytest tests\test_gateway.py -v
# Isolated integration suite (temp DB + fake upstream :8190/:8191)
.\venv\Scripts\python.exe tests\run_isolated.py
# Offline PrefixProxy test (fake provider, no external call)
.\venv\Scripts\python.exe tests\test_prefix_proxy.py
# Real smoke test (LM Studio / OmniRoute, port 8090, temp DB)
.\venv\Scripts\python.exe tools\smoke_gateway.py --port 8090
# DELTA-002 suite (semantic cache, upstream recovery, normalization anchors)
.\venv\Scripts\python.exe -m pytest tests\test_semantic_cache.py tests\test_upstream_recovery.py tests\test_anchors.py -v
The smoke test touches neither the production service (:8080) nor the dashboard.
tests\run_isolated.py uses a temporary database (ICYCACHE_DB) and never touches
the production database.
License
MIT
Ownership and license
Project origin and evidence · License
Original project and documentation by Malik Belkacemi · MLKBLK-IT · © 2026
icycache MCP questions
What tools does icycache MCP provide?
It exposes 7 tools, including get_cache_stats, clear_cache, normalize_preview, search_cached_prompts, get_tool_cache_stats.
What is the purpose of IcyCache?
It is an HTTP gateway that sits between any client and any upstream provider to cache responses and serve them instantly to avoid CPU regeneration.