MCP server AISIX AI Gateway

AISIX OSS

The open-source, Rust-native AI gateway for LLMs and AI agents

One OpenAI-compatible API in front of every model. Route, govern, secure, cache, and observe all your LLM and AI-agent traffic from a single control point — shipped as one static binary with low per-request overhead. Run it in your infrastructure for free, forever.

Built by the original creators of Apache APISIX.

Start free · Documentation · Quickstart · AISIX Enterprise and AISIX Cloud · Roadmap

---

AISIX OSS is the open-source product built around the Rust-native AISIX gateway. The gateway puts a single, OpenAI-compatible API in front of every LLM provider — OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI, DeepSeek, and any OpenAI-compatible endpoint. It gives platform teams one place to route, govern, secure, and observe LLM traffic, with first-class SSE streaming and low gateway overhead.

It runs as a single static binary — low cold-start, lock-free config reads, and hot configuration reloads with no restarts: declare resources in one resources.yaml and reload on SIGHUP, or point the gateway at etcd for a multi-replica cluster. Run the gateway in your infrastructure, or connect it to AISIX Enterprise or AISIX Cloud for centralized management with team governance, budgets, audit, and a dashboard.

AISIX OSS (this repo) is the open-source product. It runs without a control plane using declarative configuration or etcd. The same gateway binary serves as the data plane in the commercial products: customers host the control plane with AISIX Enterprise, while API7 hosts it with AISIX Cloud. In all three products, the gateway runs in your environment and calls providers directly; live AI traffic does not pass through either control plane. The proxy API is identical throughout. Talk to us about AISIX Enterprise or AISIX Cloud →

⚡ Quickstart

One container. No control plane, no database, no configuration store — the gateway reads every dynamic resource from one declarative resources.yaml.

# config.yaml
resources_file: /etc/aisix/resources.yaml
proxy:
  addr: "0.0.0.0:3000"
admin:
  enabled: false          # a declarative gateway needs no admin listener
observability:
  metrics:
    prometheus:
      enabled: true
      addr: "0.0.0.0:9090"

# resources.yaml
_format_version: "1"

provider_keys:
  - display_name: openai-main
    provider: openai
    api_key: ${OPENAI_API_KEY}        # interpolated from the environment

models:
  - display_name: my-model
    provider: openai
    model_name: gpt-4o-mini
    provider_key: openai-main

api_keys:
  - display_name: local-dev
    key_env: CALLER_API_KEY           # hashed at load; the plaintext is never stored
    allowed_models: ["my-model"]

export OPENAI_API_KEY="YOUR_PROVIDER_KEY"
export CALLER_API_KEY="YOUR_CALLER_KEY"

docker run -d --name aisix \
  -v "$(pwd)/config.yaml:/etc/aisix/config.yaml:ro" \
  -v "$(pwd)/resources.yaml:/etc/aisix/resources.yaml:ro" \
  -e OPENAI_API_KEY -e CALLER_API_KEY \
  -p 3000:3000 -p 127.0.0.1:9090:9090 \
  ghcr.io/api7/aisix:latest        # proxy → :3000, metrics + status → :9090
#                                  ^ the metrics/status listener is unauthenticated;
#                                    keep it on loopback or a private network

Then call the gateway exactly like OpenAI:

curl http://localhost:3000/v1/chat/completions \
  -H "Authorization: Bearer $CALLER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"my-model","messages":[{"role":"user","content":"hello"}]}'

Edit resources.yaml and send SIGHUP (docker kill -s HUP aisix) to apply changes with no restart — an invalid file is rejected whole and the last good configuration keeps serving. Check a file before booting with aisix validate --resources resources.yaml.

Full walkthrough: the Gateway Quickstart · every field: the resources file reference. For a multi-replica cluster, point the gateway at etcd instead — resources_file and etcd are mutually exclusive.

✨ Why AISIX

• One API, every model. Speak the OpenAI or Anthropic wire format in; the gateway translates to whichever provider each model points at. Point an OpenAI or Claude SDK at one base_url and switch models without changing code.
• A real gateway, in Rust. Single static binary, low cold-start, lock-free config reads on the hot path, native streaming.
• Open source, free forever. Apache-2.0 licensed and built to run in your infrastructure. Choose AISIX Enterprise or AISIX Cloud when you want centralized management through a control plane and dashboard.
• Production controls built in. Routing & failover, rate limits, guardrails, caching, and observability ship in the box. (Budgets and spend caps are AISIX Enterprise and AISIX Cloud features — the gateway enforces the control plane's decisions.)

🧩 Features — available today

Covered by 183 end-to-end scenario files (496 cases) that run against real gateway processes.

• OpenAI-compatible proxy (:3000) — chat/completions, completions, responses, embeddings, rerank, images/{generations,edits}, audio/{speech,transcriptions,translations}, videos (submit → poll → fetch), files, batches, fine_tuning/jobs, realtime, GET /v1/models, plus a root-level /passthrough/:provider/* escape hatch. Native SSE streaming, tool/function calling, JSON mode, vision/multimodal input, and reasoning-content support.
• Anthropic Messages API — POST /v1/messages as a first-class route, working against any configured upstream: requests and responses (including streaming) are translated both ways when a model points at a non-Anthropic provider.
• Routing & failover — virtual/routing models with six strategies: round_robin (smooth weighted round-robin), consistent_hash (session affinity keyed by header / cookie / API key / client IP), failover, plus metric-based least_cost, least_latency, and least_busy. Per-target priority tiers (active/backup pools), retry budgets, cooldowns, tag-conditional targets, and per-attempt timeouts.
• Ensemble models — fan one request out to a panel of models concurrently, then have a judge model synthesize a single answer, with a minimum-successful-responses threshold.
• Semantic routing — one virtual model that dispatches by the meaning of each request: it embeds the prompt, scores it against per-route example utterances, and routes to the best match (or a default). See the semantic routing docs.
• Rate limiting & concurrency — RPS/RPM/RPH/RPD + TPM/TPD + concurrency caps, AND-combined across caller keys, models, and policy scopes (api_key / model / team / member / team_member). Counters are per-process by default, or shared across replicas with the Redis backend.
• Guardrails — content-policy enforcement on input and output, in-process or through a provider: keyword/regex, built-in PII detection and redaction, Presidio, Lakera, OpenAI Moderation, AWS Bedrock Guardrails, Azure AI Content Safety (Prompt Shield + text moderation), and two Alibaba Cloud services. A block returns 422 content_filter; monitor mode records what would have happened without blocking.
• Caching — exact-match response cache with per-policy TTL and model/key scope matchers; memory and Redis backends; cost-saved telemetry on every hit. Separately, automatic prompt caching can be enabled per direct Anthropic model to inject cache breakpoints, so callers get provider-side prompt discounts without changing their requests.
• MCP gateway — front registered upstream MCP servers at /mcp with gateway-held credentials, per-server tool namespaces, and per-caller access. It serves every Streamable HTTP revision from 2025-03-26 through stateless 2026-07-28 without downstream sessions. Upstreams use initialize by default or server/discover with protocol_version: "2026-07-28". CI runs the official MCP suite's applicable tools-only protocol scenarios. Also exposes a REST API as MCP tools from its OpenAPI description.
• A2A agent gateway — front A2A (Agent-to-Agent) agents at /a2a/:agent, serving each agent's card with URLs rewritten to the gateway, over JSON-RPC 2.0.
• Inbound authentication — caller API keys (SHA-256 hashed, model allowlists, expiry, rotation), or OIDC/JWT bearer tokens validated against registered providers (Entra ID, Okta, Google Workspace, or any OIDC issuer) with JWKS caching.
• Observability — Prometheus /metrics, structured per-request access logs, usage events, OTLP/GenAI span export (Langfuse, Honeycomb, Grafana Cloud, or any OTLP receiver), plus dedicated Datadog and Aliyun SLS log exporters and object-storage (S3/GCS/Azure Blob) telemetry.
• Declarative configuration — one resources.yaml carries all ten resource collections (provider keys, models, caller keys, guardrails, MCP servers, A2A agents, cache policies, observability exporters, rate-limit policies, OIDC providers), validated against the same JSON Schemas the gateway uses at runtime. aisix validate checks a file offline; SIGHUP reloads it atomically.
• Operational endpoints — /livez and /readyz on the proxy listener; /status/config, /status/ready, /status/models, and Prometheus /metrics on a dedicated metrics listener (:9090). The admin listener (:3001) additionally serves a read-only resource surface, OpenAPI 3 with a Scalar UI, and a playground. Resources are managed declaratively — through the resources_file (reloaded on SIGHUP) or direct etcd writes — not through the admin listener; its former write endpoints were removed.

🔌 Supported providers

AISIX dispatches through five native adapter families — distinct wire-protocol bridges, not one generic relabel. Whatever the upstream protocol, the client-facing API stays OpenAI-shaped.

Adapter family	Reaches	Wire shape · auth
openai	OpenAI + any OpenAI-compatible vendor — DeepSeek, Groq, Mistral, Together, Fireworks, Perplexity, vLLM, Ollama, or self-hosted OpenAI-compatible endpoints	OpenAI chat completions · Bearer
anthropic	Anthropic Claude	Anthropic Messages · x-api-key
bedrock	AWS Bedrock — Anthropic, Meta Llama, Mistral, Cohere, Amazon Titan/Nova, AI21	Bedrock Converse + /invoke · SigV4
vertex	Google Vertex AI (Gemini)	Vertex :generateContent · OAuth2
azure-openai	Azure OpenAI	Azure deployments · api-key / Entra ID

Plus specialized handling for vendor quirks (e.g. DeepSeek reasoning content) and dedicated rerank / embeddings vendors (Cohere, Jina). Details in adapter protocol families.

☁️ Choose an AISIX Product

Same gateway binary, same proxy API — in every form the gateway runs in your environment. AISIX Enterprise adds a customer-hosted commercial control plane. AISIX Cloud adds an API7-hosted commercial control plane.

	AISIX OSS (this repo)	AISIX Enterprise	AISIX Cloud
License	Apache-2.0	Commercial	Commercial
Control plane	None	Customer-hosted	API7-hosted
Configuration	Declarative resources.yaml, or etcd for a cluster	Dashboard + AISIX Cloud Admin API	Dashboard + AISIX Cloud Admin API
Tenancy	Single instance / namespace	Organization-scoped environments, teams, and members	Organization-scoped environments, teams, and members
Provider keys	${VAR} references in resources.yaml; plaintext credentials in etcd	Encrypted at rest; operator-triggered in-place rotation	Encrypted at rest; operator-triggered in-place rotation
Budgets	— (rate and token limits only)	Per key, provider, environment, organization, and team	Per key, provider, environment, organization, and team
RBAC and audit	Local admin key; no organization layer	Organization roles, invitations, and audit history	Organization roles, invitations, and audit history
Operations	You operate the gateway	You operate the control plane and gateways	API7 operates the control plane; you operate gateways

→ Want centralized governance, budgets, and a dashboard? Talk to API7 about AISIX Enterprise or AISIX Cloud, or book a demo.

🏗️ Architecture

A single Cargo workspace; the aisix-server crate builds one binary named aisix that wires the crates together.

crates/
├── aisix-core           Config, snapshot, resource model, resources.yaml source, errors
├── aisix-etcd           Config provider + watch supervisor
├── aisix-gateway        Hub & bridge, SSE parser, provider trait
├── aisix-proxy          /v1/*, /mcp, /a2a handlers, routing, middleware
├── aisix-admin          Read-only resource surface + playground + OpenAPI
├── aisix-provider-*     openai · anthropic · azure-openai · bedrock · vertex
├── aisix-mcp            MCP gateway — server registry, tool ACL, transports
├── aisix-a2a            A2A agent gateway — agent cards, JSON-RPC bridge
├── aisix-ratelimit      fixed-window + token accounting + concurrency (local | redis)
├── aisix-cache          memory + redis backends
├── aisix-redis          shared Redis connection for cache + rate limits
├── aisix-guardrails     pre/post content-policy hooks
├── aisix-obs            tracing, metrics, access log, exporters
└── aisix-server         the `aisix` binary — bootstrap + CLI

🗺️ Roadmap

Highlights on the roadmap; tracked live in issues:

• Semantic (embedding-similarity) response caching
• More observability sinks — Langsmith, Helicone, Slack alerts
• Prompt templates managed as gateway resources
• Llama-Guard as a guardrail provider

Shipped since this list was last written: the MCP gateway, the A2A agent gateway, OIDC/JWT inbound auth, Redis-backed distributed rate limiting, and the Lakera, Presidio, PII, and OpenAI Moderation guardrails — see Features above.

🛠️ Development

Prerequisites: the Rust toolchain pinned in rust-toolchain.toml. Docker is only needed for the tests that exercise etcd, Redis, or provider emulators.

cargo check --workspace
cargo fmt --check
cargo clippy --workspace -- -D warnings
cargo test --workspace

# Coverage (matches the CI gate)
cargo llvm-cov --workspace --lcov --output-path lcov.info

# Run locally against a resources.yaml (no etcd needed). Copy the Quickstart's two files
# and change resources_file to the local path, e.g. resources_file: ./resources.yaml
cargo run -p aisix-server --bin aisix -- --config config.local.yaml

# Check a resources file without starting a listener
cargo run -p aisix-server --bin aisix -- validate --resources resources.yaml

💬 Community

• Discord — discord.gg/dUmRZ7Rvf
• Issues & discussions — github.com/api7/aisix/issues
• Contributing — CONTRIBUTING.md
• Website — api7.ai/ai-gateway

If AISIX is useful to you, a ⭐ helps other engineers find it.

📄 License

Apache 2.0.