MCP server de ollama-queue-proxy
Commit 13 nov 2026Proporciona un proxy de control de admisión multi-tenant para Ollama con colas de prioridad, autenticación por cliente y enrutamiento basado en modelos.
También conocido como TadMSTR/ollama-queue-proxy
- Stars
- 3
- Herramientas
- —
- Descargas, 30 días
- —
- Última versión
- —
- Último commit
- 13 nov 2026
- Licencia
- MIT
¿Qué es ollama-queue-proxy MCP?
Proporciona un proxy de control de admisión multi-tenant para Ollama con colas de prioridad, autenticación por cliente y enrutamiento basado en modelos.
Resumen generado a partir del README del proyecto
Acerca del MCP server de ollama-queue-proxy
Este proxy actúa como una capa de control de admisión para instancias compartidas de Ollama, permitiendo que múltiples usuarios o servicios compartan una misma GPU sin conflictos de recursos. Proporciona un entorno multi-tenant donde cada consumidor se identifica mediante una API key única. Estas llaves están vinculadas a límites de concurrencia y techos de prioridad específicos, asegurando que los trabajos por lotes (batch jobs) en segundo plano no afecten a las sesiones de chat interactivas. Está diseñado para desarrolladores que gestionan homelabs locales o servidores de inferencia compartidos y necesitan administrar cargas de trabajo de diversas herramientas como Open WebUI o LangChain.
Los usuarios despliegan el servidor mediante Docker y redirigen sus clientes actuales al puerto del proxy en lugar del puerto estándar de Ollama. El sistema cuenta con enrutamiento basado en modelos que prioriza hosts que ya tengan instalados modelos específicos, e incluye una caché basada en Valkey para solicitudes de embeddings con el fin de reducir la carga ascendente. Gestiona la autenticación mediante Bearer tokens o inyección basada en puertos para clientes que no admiten headers. El proxy también gestiona la persistencia de modelos inyectando configuraciones de keep_alive en las solicitudes.
Qué puedes hacer con él
- 1Evitar que los procesos batch afecten al chatAsigna un límite de prioridad bajo a los workers de embeddings en segundo plano para que los usuarios interactivos siempre tengan preferencia.
- 2Limitar solicitudes concurrentes por clienteRestringe a un cliente específico un número determinado de solicitudes simultáneas en curso para proteger los recursos de la GPU.
Escrito a partir del README del proyecto. Cada caso de uso se verifica con una cita del mismo.
Configuración
OLLAMA_HOSTOQP_AUTH__ENABLEDOQP_EMBEDDING_CACHE__ENABLEDOQP_OLLAMA__HOSTS__0__URLOQP_PROXY__PORTOQP_ROUTING__STRATEGY
| Nombre | Descripción | Tipo |
|---|---|---|
| OLLAMA_HOST | — | |
| OQP_AUTH__ENABLED | — | |
| OQP_EMBEDDING_CACHE__ENABLED | — | |
| OQP_OLLAMA__HOSTS__0__URL | — | |
| OQP_PROXY__PORT | — | |
| OQP_ROUTING__STRATEGY | — |
Verificaciones automáticas
Ejecutado 14 nov 2026Informe completo →Aún no verificado. Un servidor necesita una puntuación de 80, ningún hallazgo grave y suficientes datos para que podamos revisarlo. Cómo funcionan las verificaciones · Qué corregir para obtener más puntos
- Repositorio de origen vinculado y accesibletadmstr/ollama-queue-proxy is public and reachablegithub.com/tadmstr/ollama-queue-proxy ↗Aprobado
- El paquete indica este repositorio como su origenNo verificado
- Procedencia de la compilación (atestación de Sigstore)No verificado
- Licencia de código abierto (OSI)MITspdx.org/licenses/MIT.html ↗Aprobado
- Commit o lanzamiento en el último añoLast commit or release 2026-11-13github.com/tadmstr/ollama-queue-proxy ↗Aprobado
- OpenSSF ScorecardNo verificado
- El nombre del paquete no es similar a un paquete popularNo verificado
- No hay avisos críticos en el paquete ni en sus dependenciasNo verificado
- Sin instrucciones ocultas para el modelo0 tool descriptions and the README read; no override phrases, credential paths or hidden charactersAprobado
- Las herramientas destructivas están etiquetadasNo verificado
- El endpoint remoto responde al handshake de MCPNo verificado
README
Ver en GitHubollama-queue-proxy
Multi-tenant admission control for a shared Ollama. One endpoint that authenticates each consumer, binds its queue priority to its credential, caps its concurrency, and caches its embeddings — so that several tenants can share one GPU without the loudest one winning. Drop-in compatible: one config change in your consumers:
OLLAMA_HOST=http://localhost:11435
Everything else works as before. Streaming, /api/tags, /api/version — all pass through transparently, and metadata reads skip the queue entirely.
What it does
| Feature | What it gives you |
|---|---|
| Per-client API keys | Each consumer gets its own key with a priority ceiling — no shared secret |
| Authenticated priority | The ceiling is enforced against the key, so a client cannot promote itself by sending a header |
| Priority queuing | Three tiers (high / normal / low) with per-tier depth limits and expiry |
| Client injection | Port-based auth bypass for clients that can't send Bearer headers |
| Model-aware routing | Prefer a host that already has the model installed (/api/tags), else weighted round-robin |
| Embedding cache | Hash-keyed Valkey cache for /api/embed and /api/embeddings — repeated RAG requests skip upstream |
| keep_alive defaulting | Prevent Ollama from unloading models between bursty requests |
| Per-client concurrency caps | Hard ceiling per client so batch workloads can't starve interactive ones |
| Failover | On host failure, retry on the next configured host transparently |
The product only exists with
auth.enabled: true. With auth disabled — the shipped default, and the quick-start path —client_idcomes from a caller-suppliedX-Client-IDheader and the priority ceiling is not applied. Any client can then claim any identity and any priority, which also defeats the per-client concurrency caps. That is correct behaviour for a single-tenant quick start, but everything on this page describing policy assumes auth is on.
Just need auth? See ollama-auth-sidecar — a simpler tool if queuing, routing, and caching aren't needed.
Why this exists
The top r/ollama post of the past year was someone's open Ollama being exploited for weeks. Ollama ships with no authentication. This proxy puts auth in front with per-client keys and priority ceilings — without requiring any changes to consumers like Open WebUI, LangChain, or Continue.dev.
The other reality: in a homelab, one Ollama host quickly becomes a shared resource. Open WebUI, agent swarms, embedding workers, and overnight batch jobs all hit the same GPU. Without per-client policy and a queue, the loudest workload wins. This proxy makes one Ollama into a multi-tenant endpoint with an admission policy.
And the starvation problem: if you run embeddings at night and interactive chat hits the same server, the chat waits. One header fixes this:
X-Queue-Priority: high
Background jobs send low. Interactive tools send normal or high. The queue handles the rest.
The header is a request, not a grant. Each API key carries a max_priority, and the
requested tier is capped to it before the request is enqueued. A key issued at low
cannot obtain high by asking, so a batch worker cannot promote itself out of the tier
it was given. That distinction is the point of the whole design — see below.
How this compares to alternatives
The short version: several tools in this space queue, and several authenticate. The combination — a priority tier bound to an authenticated credential and enforced server-side — is what is hard to find elsewhere.
| Inbound client auth | Priority | Priority bound to identity | |
|---|---|---|---|
| ollama-queue-proxy | per-client API keys | 3 tiers, X-Queue-Priority |
yes — max_priority per key |
| Olla | none | — | n/a |
| ollamaMQ | none | VIP / Boost per user | no — X-User-ID is an unauthenticated header |
| LiteLLM | virtual keys | priority in request body (beta) |
no — caller-declared |
| ollama_proxy_server, LoLLMs Hub | API keys | none | n/a |
vs. Olla. Olla is the strongest tool here for fleet routing — Go, ~12 native
backends, circuit breakers, sticky KV-cache sessions, sub-millisecond selection, under
50 MB RAM. If routing intelligence across many backends is your problem, use Olla. What
it does not do is authenticate inbound clients at all; its docs are explicit that the
auth: block "has no bearing on how clients authenticate to Olla". So it cannot express
a per-tenant policy, because it has no notion of which tenant is calling.
vs. ollamaMQ. Closest in spirit — it has a real priority queue with VIP/Boost tiers
and a decaying fair-share score (which is better than this project's anti-starvation
story today). But priority is keyed off X-User-ID, an ordinary request header with
nothing verifying it, so any client can claim any tier.
vs. LiteLLM. The enterprise default for multi-provider proxying. If you run mixed
providers, use LiteLLM. Its scheduler does have a priority field, but it is
caller-declared rather than bound to the virtual key, it is marked beta, and it covers
only acompletion / atext_completion — not embeddings, which is precisely the
workload that starves interactive chat on a shared GPU. LiteLLM issue #13405,
"Support for Priority-Based Request Handling via API Keys", is an open request for
what this project already does.
vs. DIY Nginx + Redis. The "rate-limit Ollama with Nginx" pattern is in every blog
post, and it handles request-rate fine. It cannot express priority within a single key,
keep_alive injection, or an embedding cache — those need awareness of the Ollama
protocol, not just HTTP counts.
vs. ollama-auth-sidecar. The right tool when you have one Ollama host and just want auth, with no queue or policy.
How it works
flowchart TD
subgraph Entry["Entry Points"]
C1["Consumers\n(LibreChat, LangChain, agents)\nport 11435"]
C2["No-auth clients\nport 11436 / 11437"]
end
C1 -->|"Bearer token"| AUTH
C2 -->|"no header"| INJ["Client Injection\ninjects client_id + priority ceiling"]
INJ --> AUTH
AUTH{"Auth\ncheck"}
AUTH -->|"invalid"| E401["401 Unauthorized"]
AUTH -->|"pass"| EMBED
EMBED{"Embedding\nrequest?\n/api/embed\n/api/embeddings"}
EMBED -->|"yes"| CACHE{"Valkey cache\nSHA-256 key"}
EMBED -->|"no"| PRI
CACHE -->|"HIT"| HIT["Return cached response\nX-Cache: HIT\nskips queue + upstream"]
CACHE -->|"MISS"| PRI
PRI["Priority Queue\nhigh › normal › low\n(capped to key's max_priority)"]
PRI -->|"depth exceeded"| E503["429 / 503 + Retry-After"]
PRI --> WORKERS["Worker Pool\nmax_concurrent slots\nper-client semaphore"]
WORKERS --> KA["Inject keep_alive\ninto request body"]
KA --> ROUTER{"Model-aware\nrouter"}
POLLER["Background poller\nGET /api/tags every 30 s\n(every host, every interval)"]
POLLER -->|"installed model inventory"| ROUTER
ROUTER -->|"model installed on host"| H1["Ollama Host A\nprimary · weight 2"]
ROUTER -->|"model installed on host"| H2["Ollama Host B\nsecondary · weight 1"]
ROUTER -->|"no match → weighted\nround-robin"| H1
H1 -->|"connection failure"| H2
H2 -->|"all hosts failed"| E502["503 X-Failover-Exhausted"]
H1 --> RESP["Response\nX-Queue-Wait-Time\nX-Failover-Host\nX-Queue-Position"]
H2 --> RESP
Requests from multiple consumers enter the proxy, are authenticated (or identity-injected for consumers without Bearer support), and placed into one of three priority tiers — capped to the ceiling on the presented key. A worker pool drains the tiers in order. For each request, the router prefers a host that has the model installed. Embedding requests check the cache first — hits skip the queue and upstream entirely. keep_alive is injected into request bodies so Ollama doesn't unload models between requests. Per-client concurrency caps prevent any single client from monopolizing the queue.
Proxy overhead is roughly 1–2ms per request in local testing — negligible compared to Ollama inference time.
Quick start
git clone https://github.com/TadMSTR/ollama-queue-proxy
cd ollama-queue-proxy
cp config.example.yml config.yml
# Edit config.yml — set your Ollama host URL (see comment in file)
docker compose up -d
If Ollama runs natively (not in a container): set the host URL to
http://host.docker.internal:11434(Mac/Windows) orhttp://172.17.0.1:11434(Linux).
Then point your consumers at http://localhost:11435 instead of http://localhost:11434.
Warning: Default config has no authentication. If exposing beyond localhost, set
auth.enabled: trueand configure API keys. The docker-compose example binds to127.0.0.1for this reason.
Authentication
Set auth.enabled: true and add keys to config.yml:
auth:
enabled: true
keys:
- key: "sk-my-interactive-key"
client_id: "openwebui"
description: "Open WebUI"
max_priority: high
scope: inference # the default — may proxy, may not manage the queue
max_concurrent: 0 # unlimited (subject to proxy.max_concurrent)
- key: "sk-my-batch-key"
client_id: "memsearch-watch"
description: "Background embedding jobs"
max_priority: low
max_concurrent: 2 # cap at 2 concurrent so it can't starve interactive users
- key: "sk-my-admin-key"
client_id: "admin"
description: "Admin"
max_priority: high
scope: management
- key: "sk-my-readonly-key"
client_id: "homepage"
description: "Status widget — cannot spend GPU time"
scope: read
Consumers pass their key as a Bearer token:
Authorization: Bearer sk-my-interactive-key
Versus Ollama's built-in OLLAMA_API_KEY: Ollama supports a single shared key — one key for all consumers, no per-client control. This proxy gives each consumer its own key with its own priority ceiling, concurrency cap, and optional management access.
Priority ceilings: a key with max_priority: low that sends X-Queue-Priority: high is silently capped to low. The caller doesn't know — it just gets queued at its allowed tier.
Per-client concurrency caps: max_concurrent: N limits a client to N simultaneous in-flight requests. Setting to 0 is unlimited. The cap must be ≤ proxy.max_concurrent. Different clients have independent semaphores — a capped batch client never blocks an interactive client.
Key scopes
scope is one ordered axis with three cumulative levels. Each includes everything below it.
| Scope | Read status, summary, metrics, dashboard | Proxy inference to Ollama | Pause / resume / drain / flush |
|---|---|---|---|
read |
yes | no | no |
inference (default) |
yes | yes | no |
management |
yes | yes | yes |
inference is the default because it is what every key did before 0.5.0 — an existing config keeps working with no edit.
scope: read is the point of the feature. Before 0.5.0 there was no key that could not buy GPU time: management: false gated the four queue-control endpoints and nothing else, so any credential handed to a status widget also bought inference. A read key can see the proxy's own state and nothing upstream — it is refused at /api/generate, and also at cheap metadata reads like /api/tags, so the rule is "read sees the proxy, not Ollama" with no per-path exceptions to remember.
A valid key that is not authorized gets 403, not 401 — authenticated, but not enough. The refusal names only the scope that was required, never the one the caller holds.
Deprecated in 0.5.0:
management: true. It still works and maps toscope: management, logging a warning at startup that names theclient_id. Setting bothmanagement: trueand ascopethat is notmanagementis a startup error rather than a silent precedence rule — a config that contradicts itself about a privilege should not boot and pick a winner. An explicitmanagement: falseis not a contradiction with any scope: it is the field's default, so it asserts nothing, and leaving those lines in place while addingscope:elsewhere is a supported migration.
With
auth.enabled: falsenothing is enforced. No key is presented, so there is no scope to check — the same caveat that has always applied tomanagement, now coveringscopetoo. Every endpoint is open to anyone who can reach the port.
MCP consumer support: jobsearch-mcp and searxng-mcp both read OLLAMA_API_KEY from their environment and forward it as a Bearer token on all outgoing Ollama requests. Point them at the proxy and set their OLLAMA_API_KEY to their assigned key — no code changes required.
Where API keys come from
A key may be given three ways, and exactly one per entry:
auth:
enabled: true
keys:
- key: "literal-key-in-this-file" # simplest; fine for a private config
client_id: open-webui
max_priority: high
- key_env: OQP_KEY_SEARXNG # from the environment
client_id: searxng-mcp
max_priority: normal
- key_file: /run/secrets/oqp-memsearch # from a file — Docker/systemd secret style
client_id: memsearch-watch
max_priority: low
key_file strips trailing whitespace, because echo secret > file and most secret
managers leave a trailing newline, and a key that differs only by \n fails
authentication with nothing in the logs explaining why.
Why this exists. Every other setting can be supplied through the environment, but
OQP_AUTH__KEYS__0__KEY never worked: the override mechanism skips any path containing a
numeric component, and the list index trips it. So until 0.4.0 a literal in config.yml
was the only option — not as a design decision, but as a side effect. That is how API
keys end up committed to configuration repositories.
Failure modes are loud and quiet in the right places. A missing env var, an unreadable
file, zero sources, two sources, or a value that resolves to empty all fail at startup
with a message naming the client_id — and never the value. Resolved keys are excluded
from the model's repr, so a traceback or a debug dump of the config does not carry them.
Migrating an existing deployment is a separate step from this mechanism. Keys already committed to a repository's history stay compromised until they are rotated; moving them to
key_env:does not un-publish them.
Client injection
Some clients can't send a Bearer token — they're hardcoded to talk to Ollama directly with no auth header. Client injection solves this by binding extra ports that automatically inject a fixed identity, so those clients get full auth and priority enforcement without any client-side changes.
client_injection:
listeners:
- listen_port: 11436
inject_as: memsearch-watch # must match an auth.keys[].client_id
bind: 127.0.0.1 # default: loopback only
- listen_port: 11437
inject_as: localllm
allow_public_injection: false # must be true to bind injection ports to non-loopback
Point the client at the injection port. Its requests arrive with no Authorization header — the proxy fills in the identity and routes through the same queue, with the same priority ceiling and concurrency cap as the named key.
Security notes:
- Injection ports default to
127.0.0.1— accessible only from the local host. - Binding to a non-loopback address requires
allow_public_injection: true. This is appropriate behind a trusted network; be cautious exposing it to untrusted hosts. - If
allow_public_injection: trueANDauth.enabled: false, the proxy emits a startup security warning — any host on the network can consume GPU time with no credential. - The
Authorizationheader is stripped before forwarding to upstream — a token sent to an injection port is never relayed to Ollama.
Model-aware routing
When running multiple Ollama hosts (different GPUs or different model sets), the proxy can route each request to a host that already has the target model installed.
Installed, not loaded — and the difference matters. The router reads
GET /api/tags, which lists the models present on disk. It does not read/api/ps, which is what reports the models actually resident in VRAM. So this avoids sending a request to a host that would have to pull the model; it does not avoid cold-start latency, because a model installed on a host may still need loading into VRAM.The practical consequence: where every host has the same models pulled — the normal homelab case —
model_awarehas nothing to discriminate on and degenerates to weighted round-robin. Loaded-model routing via/api/psis tracked as an enhancement, not a promise.
ollama:
hosts:
- url: "http://forge:11434"
name: "primary"
weight: 2 # gets 2x the traffic of weight-1 hosts
model_sync_interval: 30 # seconds between /api/tags polls
- url: "http://helm:11434"
name: "secondary"
weight: 1
model_sync_interval: 30
routing:
strategy: model_aware # model_aware | round_robin
fallback: any_healthy # when no host has the model: pick any healthy host
model_poll_timeout: 3
How it works: a background poller hits GET /api/tags on each host every model_sync_interval seconds, maintaining a live (host → installed_models) map. Every host is polled every interval, whether or not it is currently reachable. Requests with a model field are routed to a host that has it. Weighted round-robin is deterministic (not stochastic) — a 2:1 weight ratio means exactly 2 requests to the heavy host for every 1 to the lighter host.
Requests without a model field use weighted round-robin across reachable hosts. If
no host is currently marked reachable, the proxy still picks one rather than refusing:
reachability is a cached observation, and one failed poll against a host that has since
recovered should not black-hole the proxy into 503s.
Fast-path invalidation: when a host returns "model not found" (404), the proxy immediately removes that (host, model) pair from the routing table — no waiting for the next poll cycle.
Startup: the proxy probes each host's /api/tags once before accepting requests. If no host responds, startup fails fast with a clear error.
Embedding cache
Repeated embedding requests (common in RAG, semantic search, and agent workloads) often re-embed the same strings. The embedding cache stores successful responses in Valkey (or any RESP-compatible store — Dragonfly is a supported drop-in) and serves hits without touching the queue or upstream.
embedding_cache:
enabled: true
backend: "redis://valkey:6379/0" # Valkey recommended; Dragonfly works as a drop-in
ttl: 86400 # seconds
max_entry_bytes: 32768 # skip caching responses larger than this
key_prefix: "oqp:embed:"
connect_timeout: 2
Scope: /api/embed and /api/embeddings only. /api/generate and /api/chat are never cached (non-deterministic, large, low repeat rate).
Cache key: SHA256 of model + \0 + canonical_json(input), truncated to 32 hex chars. Per-endpoint namespaces prevent cross-endpoint collisions (same text via /api/embed and /api/embeddings get separate keys — their response shapes differ).
Startup: if enabled: true, the proxy pings the backend at startup. If unreachable, startup fails fast. After startup, any RESP error degrades gracefully: logged at most once per minute, the cache is bypassed for that request, and no user request fails.
Metrics: oqp_embedding_cache_hits_total, oqp_embedding_cache_misses_total, oqp_embedding_cache_errors_total — all at /metrics with client, model, and endpoint labels.
keep_alive defaulting
Ollama unloads a model from GPU memory after 5 minutes of inactivity (configurable on the Ollama side). For bursty workloads — embeddings or agents that fire requests every few minutes — this causes expensive cold-load latency. The proxy injects a keep_alive value so the model stays loaded.
keep_alive:
default: "5m" # injected when the client doesn't send keep_alive
override: false # if true, always replace the client's value with default
Applies to: /api/generate, /api/chat, /api/embed, /api/embeddings.
Behavior: if override: false and keep_alive is absent in the request body, inject default. If override: true, always replace. Non-JSON bodies and bodies over max_request_body_mb pass through untouched.
Metadata fast path
/api/tags, /api/version, /api/ps, /api/show and / bypass the priority queue and
the worker pool. They are authenticated like everything else, then dispatched straight
upstream.
This matters more than it sounds. Before 0.4.0 every path entered the queue through the
catch-all handler, so with a small max_concurrent a UI polling /api/tags would queue
behind a multi-minute generation and appear to hang. Admission control should not be
spent on reads that carry no inference cost.
The trade-off: these paths are not subject to per-client concurrency caps. They are cheap reads against a local daemon, bounded by the shared HTTP connection pool.
Priority queuing
Three tiers: high, normal (default), low. Set the tier per-request:
X-Queue-Priority: low
Workers dispatch high before normal before low. Each tier has its own depth limit, max wait timeout, and high-watermark threshold for webhook events.
Consumer example:
# Background embedding job — uses low priority
import httpx
client = httpx.Client(
base_url="http://localhost:11435",
headers={
"Authorization": "Bearer sk-my-batch-key",
"X-Queue-Priority": "low",
}
)
The proxy caps the priority to the key's max_priority — a batch key configured with max_priority: low can't elevate itself to high regardless of what header it sends.
Preguntas sobre ollama-queue-proxy MCP
¿Cómo lo uso con clientes de Ollama existentes?
Es un reemplazo directo; solo necesitas cambiar el OLLAMA_HOST de tu cliente para que apunte a la dirección del proxy.
¿Puede un cliente aumentar su propia prioridad?
No, el límite de prioridad se aplica en el servidor mediante la API key autenticada, por lo que un cliente no puede promocionarse enviando un header.