aiproxy MCP server

aiproxy

An OpenAI-compatible gateway that turns any LLM into a more capable one by fusing it with a reusable fabric of MCP servers.

📖 Documentation: https://sirmmo.github.io/aiproxy/

Point any OpenAI client at aiproxy, pick one of your configured assistants as the model, and the gateway runs the whole agentic tool loop for you — calling the wrapped LLM, executing tools against your Model Context Protocol servers, feeding results back — and returns a normal OpenAI response. From the client's side it just looks like a smarter model.

flowchart LR
    client(["Any OpenAI client"])
    client -- "POST /v1/chat/completions" --> gate
    subgraph proxy["aiproxy"]
        direction TB
        gate["auth chain<br/>static keys | Apiman"]
        loop(["agent loop"])
        backend["backend adapter<br/>OpenAI-compat | native Anthropic"]
        mcp["MCP servers<br/>fetch | filesystem | http ..."]
        gate --> loop
        loop -- "LLM turn" --> backend
        backend -. "assistant / tool_calls" .-> loop
        loop -- "tool calls" --> mcp
        mcp -. "results" .-> loop
    end
    backend -- "chat / messages API" --> llm(["Upstream LLM"])
    proxy -- "OpenAI response" --> client

Why

MCP gives you a growing ecosystem of tool servers (web fetch, filesystem, databases, search, your own APIs). But wiring those tools into every app and every model is repetitive. aiproxy makes that infrastructure reusable and model-agnostic: define your MCP servers and LLM backends once, compose them into named assistants, and every OpenAI-compatible app in your stack gets a tool-augmented model for free — no client changes, no SDK lock-in.

Features

• OpenAI-compatible API — /v1/chat/completions (streaming and non-streaming) and /v1/models. Works with the OpenAI SDKs, LangChain, LlamaIndex, curl, etc.
• Wraps any LLM — OpenAI-compatible backends (OpenAI, Groq, Together, Mistral, vLLM, Ollama, LM Studio, OpenRouter…) and native Anthropic (Messages API), behind one interface.
• Reflex models ("system one") — no, yes, maybe and random backends answer without any LLM: plain text, schema-valid JSON for response_format, tool calls, streaming. No-as-a-service, a deterministic stand-in for tests, a negative baseline for evals.
• Reusable MCP fabric — attach any number of MCP servers (stdio, sse, streamable-http) to each assistant. Tools are namespaced per server and executed transparently.
• Assistants as virtual models — each model a client can pick is a backend + system prompt + set of MCP servers + tool-loop budget.
• Two-model assistants — optionally split the job: a tool-calling specialist (tool_backend, e.g. needle-openai) decides which tools to call, gated on the confidence it reports, and a chat model that cannot call tools at all (e.g. openai-MobileMoE) writes the answer.
• Runtime admin API — add/edit/remove assistants, backends and MCP servers without a restart; introspect any server's tools.
• Pluggable auth — static API keys and Apiman key validation (gateway round-trip or trusted-header topologies) run in parallel.
• Browser clients welcome — CORS is on by default (CORS_ORIGINS), so in-browser clients such as Hollama can talk to /v1 directly.
• Docker-first — docker compose up and you have an endpoint. Node (npx) and uvx are baked in so most MCP servers install on demand.

Quick start (Docker)

Use the prebuilt image from GitHub Container Registry:

cp .env.example .env                # add your OPENAI_API_KEY / ANTHROPIC_API_KEY
cp config.example.yaml config.yaml  # define backends, MCP servers, assistants

docker run --rm -p 8000:8000 --env-file .env \
  -v "$PWD/config.yaml:/app/config.yaml:ro" \
  ghcr.io/sirmmo/aiproxy:latest

…or build from source with compose:

cp .env.example .env
cp config.example.yaml config.yaml
docker compose up --build

Then talk to it like OpenAI:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "research-assistant",
    "messages": [{"role": "user", "content": "Summarize https://modelcontextprotocol.io"}]
  }'

Or with the OpenAI Python SDK:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused-or-your-PROXY_API_KEY")

resp = client.chat.completions.create(
    model="research-assistant",              # an assistant, not a raw model
    messages=[{"role": "user", "content": "What's on the MCP homepage?"}],
)
print(resp.choices[0].message.content)

Streaming works exactly as clients expect (stream=True) — content tokens flow through while tool rounds run transparently between them.

Configuration

Everything is declared in config.yaml. ${VAR} / ${VAR:-default} are expanded from the environment, so keep secrets in .env.

mcp_servers:
  fetch:                                   # a reusable MCP server
    transport: stdio
    command: uvx
    args: ["mcp-server-fetch"]

backends:
  anthropic:                               # a wrapped LLM provider
    kind: anthropic                        # or "openai" for any compat endpoint
    base_url: https://api.anthropic.com/v1
    api_key: ${ANTHROPIC_API_KEY}

assistants:
  - name: research-assistant               # <- clients pass this as `model`
    backend: anthropic
    model: claude-sonnet-5
    system_prompt: "You are a meticulous research assistant. Cite your sources."
    mcp_servers: [fetch]
    max_tool_iterations: 8
    temperature: 0.2

See config.example.yaml for the fully annotated version, including sse/http MCP servers, local model backends, and API-key auth.

Two-model assistants

Small on-device models are often good at either picking a function or writing prose, rarely both. An assistant can name a second backend that only decides tool calls:

assistants:
  - name: ontorag-chat
    backend: mobilemoe            # writes the answer; never sees the tools
    model: MobileMoE-M-QAT
    tool_backend: needle          # decides tool calls
    tool_model: needle-2
    tool_confidence: 0.5          # run a call only at/above this reported confidence
    tool_context: turn            # `turn` (system prompt + current turn) or `full`
    tool_max_rounds: 1            # ask the specialist once per turn
    tool_fallback:                # if it declines, retrieve anyway
      tool: ontorag__answer
      arguments: {query: "{user}"}
    tool_citations:               # gateway-side citations from the results
      pattern: "[a-z0-9][a-z0-9-]+::[0-9]{4}"
      label: Sources
      skip_pattern: "no information|nothing on|nothing about"   # refusals carry no sources
    tool_empty:                   # nothing retrieved -> say so, skip the model
      pattern: '^\s*(\[\s*\])?\s*$|"matched_entities":\s*\[\s*\]'
      reply: The knowledge graph has nothing on that.
    tool_allowlist: [ontorag__answer, ontorag__search_entities]
    tool_result_max_chars: 6000   # cap retrieval payloads for a small answer model
    tool_arguments:               # hidden from the model, forced on every call
      ontorag__answer: {k: 3, expand: 1}
    tool_descriptions:
      ontorag__answer: Retrieve passages and facts for any question about the game world.
      ontorag__search_entities: Find factions, characters, species or places by name.
    mcp_servers: [ontorag]

Each round the gateway asks tool_backend first. If it returns tool calls whose confidence (x_needle.confidence, or a top-level confidence; backends that report none are trusted) clears tool_confidence, the calls run against the MCP servers and the round repeats. Otherwise backend answers from the conversation, tools withheld. Responses carry an x_aiproxy.decisions list showing every round's calls, confidence and whether they ran (and, with expose_tool_results: true, the retrieved text, for evaluation harnesses). An evaluation of this setup with RAGAS, TruLens, an ARES-style judge and Mintaka lives in evals/two_model_ontorag/. tool_citations appends the source ids found in the turn's tool results to the answer (small models do not cite reliably when asked; the gateway can), and tool_empty returns a fixed reply without calling the answer model when a round retrieved nothing, which small models otherwise paper over. tool_fallback makes retrieval unconditional: when the specialist's first decision of a turn runs nothing, the gateway calls the configured tool itself with {user} replaced by the user's message, so the answer model always has context instead of inventing some. tool_context: turn keeps specialists that truncate long inputs honest by showing them only the system prompt and the current turn, and tool_max_rounds: 1 asks them once per turn, before any tool result is in view. tool_allowlist, tool_descriptions, tool_arguments and tool_result_max_chars (usable on any assistant) trim what the models are told about and asked to decide: small tool-calling models degrade with large tool sets and long descriptions written for bigger models, should not be choosing page sizes or dataset ids, and a small answer model cannot absorb thousands of tokens of retrieval payload.

Backends

kind	Talks to	Auth header
openai	any OpenAI-compatible /chat/completions	Authorization: Bearer
anthropic	native Anthropic /messages	x-api-key

The Anthropic backend translates the canonical chat messages ↔ the Messages API (system prompt, tool_use/tool_result blocks, streaming events, stop-reason mapping), so tool use works first-class with Claude.

Reflex models ("system one")

Four more backend kinds answer without a model behind them. Each one has a polarity and shapes it to whatever the client asked for:

kind	Chat	response_format JSON	Tools on offer
no	No.	false, 0, "no", [], first "no"-like enum	never calls one
yes	Yes.	true, 1/maximum, "yes", one item, "yes"-like enum	calls the first tool with schema-valid arguments, then answers Yes.
maybe	Maybe.	null where allowed, midpoints, 0.5, coin flips, middle enum	never calls one
random	one of the above, picked per request by weights	idem	idem

backends:
  naas: {kind: no}
  yaas: {kind: yes, tool_calls: first}      # or `all` / `none`
  maas: {kind: maybe}
  raas: {kind: random, weights: {yes: 1, no: 1, maybe: 1}, seed: 42}

assistants:
  - {name: "no",  backend: naas}           # no `model` needed
  - {name: "yes", backend: yaas, mcp_servers: [echo]}

curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"no","messages":[{"role":"user","content":"Can I have a raise?"}]}' \
  | jq -r .choices[0].message.content
# No.

json_schema documents are filled field by field ($ref/$defs, anyOf, enum, const, minItems, minimum/maximum, date formats), so strict clients validate. Streaming emits the answer word by word; latency and stream_delay (seconds) simulate a slow model. text overrides the answer. Responses report usage estimates like any other model.

MCP servers

transport	Fields
stdio	command, args, env, cwd
sse	url, headers
http / streamable-http	url, headers

Tools are exposed to the model as "<server>__<tool>" and routed back to the right server on call. Sessions are persistent (one subprocess per stdio server, reused across requests) and started lazily on first use.

Admin API

Mutate the live registry without restarting (set ADMIN_API_KEY to protect it):

# Inspect current state (secrets redacted)
curl localhost:8000/admin/config

# See what tools a server actually advertises
curl localhost:8000/admin/mcp/fetch/tools

# Add/replace an assistant
curl -X PUT localhost:8000/admin/assistants/coder \
  -H "Content-Type: application/json" \
  -d '{"backend":"openai","model":"gpt-4o","mcp_servers":["filesystem"],
       "system_prompt":"You are a coding agent."}'

Method & path	Purpose
GET /admin/config	Dump current registry (secrets redacted)
GET/PUT/DELETE /admin/assistants[/{name}]	Manage assistants
GET/PUT/DELETE /admin/backends[/{name}]	Manage LLM backends
GET/PUT/DELETE /admin/mcp[/{name}]	Manage MCP servers
GET /admin/mcp/{name}/tools	Introspect a server's tools

Admin changes are in-memory. Use GET /admin/config to export current state and persist it into config.yaml yourself.

Auth

/v1/* auth is a pluggable chain — a request is authorized if any enabled provider accepts it, so static keys and Apiman run in parallel:

Provider	Enable with	Accepts a request when…
Static keys	non-empty proxy_api_keys	the caller's key matches an entry
Apiman gateway_probe	apiman.enabled: true, mode: gateway_probe	the key validates via a round-trip through the Apiman gateway (2xx)
Apiman trusted_header	apiman.enabled: true, mode: trusted_header	the request carries the shared secret the gateway injects

The caller's key is read from Authorization: Bearer <key>, the X-API-Key header, or the ?apikey= query param (matching Apiman's own conventions). If no provider is configured, /v1/* is open (handy for local dev).

• /admin/* — separate: if ADMIN_API_KEY is set, admin calls must send Authorization: Bearer <ADMIN_API_KEY>.

Apiman integration

Apiman is an open-source API-management layer (clients, plans, contracts, quotas, policies). Two topologies are supported:

gateway_probe — aiproxy stays reachable directly and validates each caller's key against Apiman itself. Register a tiny "auth check" API in Apiman whose backend is aiproxy's /health, then:

apiman:
  enabled: true
  mode: gateway_probe
  gateway_url: http://apiman-gateway:8080/apiman-gateway
  probe_api: aiproxy/authcheck/1.0    # {org}/{api}/{version}
  probe_path: health                  # backend path that returns 2xx
  cache_ttl: 60

On each request aiproxy calls GET {gateway_url}/aiproxy/authcheck/1.0/health with X-API-Key: <caller-key>; a 2xx authorizes the request (cached for cache_ttl seconds), 401/403 rejects it.

trusted_header — put the Apiman gateway in front of aiproxy and have an "Add Header" policy inject a shared secret; aiproxy trusts any request carrying it and never sees raw keys:

apiman:
  enabled: true
  mode: trusted_header
  header: X-Apiman-Gateway-Token
  secret: ${APIMAN_SHARED_SECRET}

Development & testing

The recommended path is Docker — it pins a clean Python 3.12 with node/uvx available, so you never fight a host interpreter or a leaked PYTHONPATH.

docker build -t aiproxy:latest .

# End-to-end check — spawns the demo MCP server and drives the full agent
# loop (streaming + non-streaming) with a scripted fake LLM. No API key needed:
docker run --rm aiproxy:latest python scripts/smoke_test.py
# Reflex backends (no / yes / maybe / random): schema filling, tools, streaming.
docker run --rm aiproxy:latest python scripts/test_reflex.py

# Run the server against your config.yaml (or just `docker compose up`):
docker run --rm -p 8000:8000 -v "$PWD/config.yaml:/app/config.yaml:ro" aiproxy:latest

uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python -e '.[dev]'
# Clear PYTHONPATH if your shell injects system site-packages that shadow the venv:
env PYTHONPATH= .venv/bin/python scripts/smoke_test.py
env PYTHONPATH= .venv/bin/uvicorn app.main:app --reload --port 8000

How a request flows

1. Client POST /v1/chat/completions with model: "<assistant>".
2. Gateway resolves the assistant → backend + MCP servers, and ensures those servers are connected.
3. It builds the OpenAI tool schema from the servers' tools and enters the agent loop:
   • call the backend with the messages + tools (an LLM, or a reflex that answers without one);
   • if the model returns tool calls, execute them concurrently against the MCP servers and append the results;
   • repeat until the model answers or max_tool_iterations is hit (the last turn drops tools to force a final answer).
4. Return an OpenAI chat.completion (or stream chat.completion.chunks), with the assistant name as the model and aggregated token usage.

Project layout

app/
  main.py            FastAPI app + lifespan
  config.py          YAML config models, ${env} expansion
  state.py           live registry (assistants/backends/MCP), runtime mutation
  mcp_manager.py     persistent MCP sessions + namespaced ToolSet router
  agent.py           the agentic loop (streaming + non-streaming)
  backends/          openai + anthropic adapters, and the reflex (no/yes/maybe/random) responders, behind one interface
  routes/            /v1 (chat) and /admin (registry) endpoints
  auth.py            pluggable auth chain (static keys + Apiman)
examples/echo_mcp_server.py   demo stdio MCP server
scripts/smoke_test.py         end-to-end test, no real LLM required
scripts/test_reflex.py        reflex backends: schema filling, tools, streaming, agent loop

License

MIT