ollama-queue-proxy MCP server
Commit Nov 13, 2026Provides a multi-tenant admission control proxy for Ollama with priority queuing, per-client authentication, and model-aware routing.
Also known as TadMSTR/ollama-queue-proxy
- Stars
- 3
- Tools
- —
- Downloads, 30 days
- —
- Latest version
- —
- Last commit
- Nov 13, 2026
- License
- MIT
What is ollama-queue-proxy MCP?
ollama-queue-proxy provides a multi-tenant admission control proxy for Ollama with priority queuing, per-client authentication, and model-aware routing.
Summary generated from the project's README
About ollama-queue-proxy MCP server
This proxy acts as an admission control layer for shared Ollama instances, allowing multiple users or services to share a single GPU without resource contention. It provides a multi-tenant environment where each consumer is identified by a unique API key. These keys are tied to specific priority ceilings and concurrency limits, ensuring that background batch jobs do not starve interactive chat sessions. It is designed for developers running local homelabs or shared inference servers who need to manage workloads from various tools like Open WebUI or LangChain.
Users deploy the server via Docker and point their existing clients to the proxy port instead of the standard Ollama port. The system features model-aware routing that prefers hosts with specific models already installed and includes a Valkey-based cache for embedding requests to reduce upstream load. It handles authentication through Bearer tokens or port-based injection for clients that lack header support. The proxy also manages model persistence by injecting keep_alive settings into requests.
What you can do with it
- 1Prevent batch jobs from starving chatAssign a low priority ceiling to background embedding workers so interactive users always get preference.
- 2Limit concurrent requests per clientRestrict a specific client to a set number of simultaneous in-flight requests to protect GPU resources.
Written from the project's README. Every use case is checked against a quote from it.
Configuration
OLLAMA_HOSTOQP_AUTH__ENABLEDOQP_EMBEDDING_CACHE__ENABLEDOQP_OLLAMA__HOSTS__0__URLOQP_PROXY__PORTOQP_ROUTING__STRATEGY
| Name | Description | |
|---|---|---|
| OLLAMA_HOST | — | |
| OQP_AUTH__ENABLED | — | |
| OQP_EMBEDDING_CACHE__ENABLED | — | |
| OQP_OLLAMA__HOSTS__0__URL | — | |
| OQP_PROXY__PORT | — | |
| OQP_ROUTING__STRATEGY | — |
Automated checks
Ran Nov 14, 2026Full report →Not verified yet. A server needs a score of 80, no serious finding, and enough data for us to check. How checks work · What to fix for more points
- Source repository linked and reachabletadmstr/ollama-queue-proxy is public and reachablegithub.com/tadmstr/ollama-queue-proxy ↗Pass
- Package names this repository as its sourceNot checked
- Build provenance (Sigstore attestation)Not checked
- Open source license (OSI)MITspdx.org/licenses/MIT.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-11-13github.com/tadmstr/ollama-queue-proxy ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNot checked
- No critical advisory in the package or its dependenciesNot checked
- No hidden instructions to the model0 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNot checked
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubollama-queue-proxy
Multi-tenant admission control for a shared Ollama. One endpoint that authenticates each consumer, binds its queue priority to its credential, caps its concurrency, and caches its embeddings — so that several tenants can share one GPU without the loudest one winning. Drop-in compatible: one config change in your consumers:
OLLAMA_HOST=http://localhost:11435
Everything else works as before. Streaming, /api/tags, /api/version — all pass through transparently, and metadata reads skip the queue entirely.
What it does
| Feature | What it gives you |
|---|---|
| Per-client API keys | Each consumer gets its own key with a priority ceiling — no shared secret |
| Authenticated priority | The ceiling is enforced against the key, so a client cannot promote itself by sending a header |
| Priority queuing | Three tiers (high / normal / low) with per-tier depth limits and expiry |
| Client injection | Port-based auth bypass for clients that can't send Bearer headers |
| Model-aware routing | Prefer a host that already has the model installed (/api/tags), else weighted round-robin |
| Embedding cache | Hash-keyed Valkey cache for /api/embed and /api/embeddings — repeated RAG requests skip upstream |
| keep_alive defaulting | Prevent Ollama from unloading models between bursty requests |
| Per-client concurrency caps | Hard ceiling per client so batch workloads can't starve interactive ones |
| Failover | On host failure, retry on the next configured host transparently |
The product only exists with
auth.enabled: true. With auth disabled — the shipped default, and the quick-start path —client_idcomes from a caller-suppliedX-Client-IDheader and the priority ceiling is not applied. Any client can then claim any identity and any priority, which also defeats the per-client concurrency caps. That is correct behaviour for a single-tenant quick start, but everything on this page describing policy assumes auth is on.
Just need auth? See ollama-auth-sidecar — a simpler tool if queuing, routing, and caching aren't needed.
Why this exists
The top r/ollama post of the past year was someone's open Ollama being exploited for weeks. Ollama ships with no authentication. This proxy puts auth in front with per-client keys and priority ceilings — without requiring any changes to consumers like Open WebUI, LangChain, or Continue.dev.
The other reality: in a homelab, one Ollama host quickly becomes a shared resource. Open WebUI, agent swarms, embedding workers, and overnight batch jobs all hit the same GPU. Without per-client policy and a queue, the loudest workload wins. This proxy makes one Ollama into a multi-tenant endpoint with an admission policy.
And the starvation problem: if you run embeddings at night and interactive chat hits the same server, the chat waits. One header fixes this:
X-Queue-Priority: high
Background jobs send low. Interactive tools send normal or high. The queue handles the rest.
The header is a request, not a grant. Each API key carries a max_priority, and the
requested tier is capped to it before the request is enqueued. A key issued at low
cannot obtain high by asking, so a batch worker cannot promote itself out of the tier
it was given. That distinction is the point of the whole design — see below.
How this compares to alternatives
The short version: several tools in this space queue, and several authenticate. The combination — a priority tier bound to an authenticated credential and enforced server-side — is what is hard to find elsewhere.
| Inbound client auth | Priority | Priority bound to identity | |
|---|---|---|---|
| ollama-queue-proxy | per-client API keys | 3 tiers, X-Queue-Priority |
yes — max_priority per key |
| Olla | none | — | n/a |
| ollamaMQ | none | VIP / Boost per user | no — X-User-ID is an unauthenticated header |
| LiteLLM | virtual keys | priority in request body (beta) |
no — caller-declared |
| ollama_proxy_server, LoLLMs Hub | API keys | none | n/a |
vs. Olla. Olla is the strongest tool here for fleet routing — Go, ~12 native
backends, circuit breakers, sticky KV-cache sessions, sub-millisecond selection, under
50 MB RAM. If routing intelligence across many backends is your problem, use Olla. What
it does not do is authenticate inbound clients at all; its docs are explicit that the
auth: block "has no bearing on how clients authenticate to Olla". So it cannot express
a per-tenant policy, because it has no notion of which tenant is calling.
vs. ollamaMQ. Closest in spirit — it has a real priority queue with VIP/Boost tiers
and a decaying fair-share score (which is better than this project's anti-starvation
story today). But priority is keyed off X-User-ID, an ordinary request header with
nothing verifying it, so any client can claim any tier.
vs. LiteLLM. The enterprise default for multi-provider proxying. If you run mixed
providers, use LiteLLM. Its scheduler does have a priority field, but it is
caller-declared rather than bound to the virtual key, it is marked beta, and it covers
only acompletion / atext_completion — not embeddings, which is precisely the
workload that starves interactive chat on a shared GPU. LiteLLM issue #13405,
"Support for Priority-Based Request Handling via API Keys", is an open request for
what this project already does.
vs. DIY Nginx + Redis. The "rate-limit Ollama with Nginx" pattern is in every blog
post, and it handles request-rate fine. It cannot express priority within a single key,
keep_alive injection, or an embedding cache — those need awareness of the Ollama
protocol, not just HTTP counts.
vs. ollama-auth-sidecar. The right tool when you have one Ollama host and just want auth, with no queue or policy.
How it works
flowchart TD
subgraph Entry["Entry Points"]
C1["Consumers\n(LibreChat, LangChain, agents)\nport 11435"]
C2["No-auth clients\nport 11436 / 11437"]
end
C1 -->|"Bearer token"| AUTH
C2 -->|"no header"| INJ["Client Injection\ninjects client_id + priority ceiling"]
INJ --> AUTH
AUTH{"Auth\ncheck"}
AUTH -->|"invalid"| E401["401 Unauthorized"]
AUTH -->|"pass"| EMBED
EMBED{"Embedding\nrequest?\n/api/embed\n/api/embeddings"}
EMBED -->|"yes"| CACHE{"Valkey cache\nSHA-256 key"}
EMBED -->|"no"| PRI
CACHE -->|"HIT"| HIT["Return cached response\nX-Cache: HIT\nskips queue + upstream"]
CACHE -->|"MISS"| PRI
PRI["Priority Queue\nhigh › normal › low\n(capped to key's max_priority)"]
PRI -->|"depth exceeded"| E503["429 / 503 + Retry-After"]
PRI --> WORKERS["Worker Pool\nmax_concurrent slots\nper-client semaphore"]
WORKERS --> KA["Inject keep_alive\ninto request body"]
KA --> ROUTER{"Model-aware\nrouter"}
POLLER["Background poller\nGET /api/tags every 30 s\n(every host, every interval)"]
POLLER -->|"installed model inventory"| ROUTER
ROUTER -->|"model installed on host"| H1["Ollama Host A\nprimary · weight 2"]
ROUTER -->|"model installed on host"| H2["Ollama Host B\nsecondary · weight 1"]
ROUTER -->|"no match → weighted\nround-robin"| H1
H1 -->|"connection failure"| H2
H2 -->|"all hosts failed"| E502["503 X-Failover-Exhausted"]
H1 --> RESP["Response\nX-Queue-Wait-Time\nX-Failover-Host\nX-Queue-Position"]
H2 --> RESP
Requests from multiple consumers enter the proxy, are authenticated (or identity-injected for consumers without Bearer support), and placed into one of three priority tiers — capped to the ceiling on the presented key. A worker pool drains the tiers in order. For each request, the router prefers a host that has the model installed. Embedding requests check the cache first — hits skip the queue and upstream entirely. keep_alive is injected into request bodies so Ollama doesn't unload models between requests. Per-client concurrency caps prevent any single client from monopolizing the queue.
Proxy overhead is roughly 1–2ms per request in local testing — negligible compared to Ollama inference time.
Quick start
git clone https://github.com/TadMSTR/ollama-queue-proxy
cd ollama-queue-proxy
cp config.example.yml config.yml
# Edit config.yml — set your Ollama host URL (see comment in file)
docker compose up -d
If Ollama runs natively (not in a container): set the host URL to
http://host.docker.internal:11434(Mac/Windows) orhttp://172.17.0.1:11434(Linux).
Then point your consumers at http://localhost:11435 instead of http://localhost:11434.
Warning: Default config has no authentication. If exposing beyond localhost, set
auth.enabled: trueand configure API keys. The docker-compose example binds to127.0.0.1for this reason.
Authentication
Set auth.enabled: true and add keys to config.yml:
auth:
enabled: true
keys:
- key: "sk-my-interactive-key"
client_id: "openwebui"
description: "Open WebUI"
max_priority: high
scope: inference # the default — may proxy, may not manage the queue
max_concurrent: 0 # unlimited (subject to proxy.max_concurrent)
- key: "sk-my-batch-key"
client_id: "memsearch-watch"
description: "Background embedding jobs"
max_priority: low
max_concurrent: 2 # cap at 2 concurrent so it can't starve interactive users
- key: "sk-my-admin-key"
client_id: "admin"
description: "Admin"
max_priority: high
scope: management
- key: "sk-my-readonly-key"
client_id: "homepage"
description: "Status widget — cannot spend GPU time"
scope: read
Consumers pass their key as a Bearer token:
Authorization: Bearer sk-my-interactive-key
Versus Ollama's built-in OLLAMA_API_KEY: Ollama supports a single shared key — one key for all consumers, no per-client control. This proxy gives each consumer its own key with its own priority ceiling, concurrency cap, and optional management access.
Priority ceilings: a key with max_priority: low that sends X-Queue-Priority: high is silently capped to low. The caller doesn't know — it just gets queued at its allowed tier.
Per-client concurrency caps: max_concurrent: N limits a client to N simultaneous in-flight requests. Setting to 0 is unlimited. The cap must be ≤ proxy.max_concurrent. Different clients have independent semaphores — a capped batch client never blocks an interactive client.
Key scopes
scope is one ordered axis with three cumulative levels. Each includes everything below it.
| Scope | Read status, summary, metrics, dashboard | Proxy inference to Ollama | Pause / resume / drain / flush |
|---|---|---|---|
read |
yes | no | no |
inference (default) |
yes | yes | no |
management |
yes | yes | yes |
inference is the default because it is what every key did before 0.5.0 — an existing config keeps working with no edit.
scope: read is the point of the feature. Before 0.5.0 there was no key that could not buy GPU time: management: false gated the four queue-control endpoints and nothing else, so any credential handed to a status widget also bought inference. A read key can see the proxy's own state and nothing upstream — it is refused at /api/generate, and also at cheap metadata reads like /api/tags, so the rule is "read sees the proxy, not Ollama" with no per-path exceptions to remember.
A valid key that is not authorized gets 403, not 401 — authenticated, but not enough. The refusal names only the scope that was required, never the one the caller holds.
Deprecated in 0.5.0:
management: true. It still works and maps toscope: management, logging a warning at startup that names theclient_id. Setting bothmanagement: trueand ascopethat is notmanagementis a startup error rather than a silent precedence rule — a config that contradicts itself about a privilege should not boot and pick a winner. An explicitmanagement: falseis not a contradiction with any scope: it is the field's default, so it asserts nothing, and leaving those lines in place while addingscope:elsewhere is a supported migration.
With
auth.enabled: falsenothing is enforced. No key is presented, so there is no scope to check — the same caveat that has always applied tomanagement, now coveringscopetoo. Every endpoint is open to anyone who can reach the port.
MCP consumer support: jobsearch-mcp and searxng-mcp both read OLLAMA_API_KEY from their environment and forward it as a Bearer token on all outgoing Ollama requests. Point them at the proxy and set their OLLAMA_API_KEY to their assigned key — no code changes required.
Where API keys come from
A key may be given three ways, and exactly one per entry:
auth:
enabled: true
keys:
- key: "literal-key-in-this-file" # simplest; fine for a private config
client_id: open-webui
max_priority: high
- key_env: OQP_KEY_SEARXNG # from the environment
client_id: searxng-mcp
max_priority: normal
- key_file: /run/secrets/oqp-memsearch # from a file — Docker/systemd secret style
client_id: memsearch-watch
max_priority: low
key_file strips trailing whitespace, because echo secret > file and most secret
managers leave a trailing newline, and a key that differs only by \n fails
authentication with nothing in the logs explaining why.
Why this exists. Every other setting can be supplied through the environment, but
OQP_AUTH__KEYS__0__KEY never worked: the override mechanism skips any path containing a
numeric component, and the list index trips it. So until 0.4.0 a literal in config.yml
was the only option — not as a design decision, but as a side effect. That is how API
keys end up committed to configuration repositories.
Failure modes are loud and quiet in the right places. A missing env var, an unreadable
file, zero sources, two sources, or a value that resolves to empty all fail at startup
with a message naming the client_id — and never the value. Resolved keys are excluded
from the model's repr, so a traceback or a debug dump of the config does not carry them.
Migrating an existing deployment is a separate step from this mechanism. Keys already committed to a repository's history stay compromised until they are rotated; moving them to
key_env:does not un-publish them.
Client injection
Some clients can't send a Bearer token — they're hardcoded to talk to Ollama directly with no auth header. Client injection solves this by binding extra ports that automatically inject a fixed identity, so those clients get full auth and priority enforcement without any client-side changes.
client_injection:
listeners:
- listen_port: 11436
inject_as: memsearch-watch # must match an auth.keys[].client_id
bind: 127.0.0.1 # default: loopback only
- listen_port: 11437
inject_as: localllm
allow_public_injection: false # must be true to bind injection ports to non-loopback
Point the client at the injection port. Its requests arrive with no Authorization header — the proxy fills in the identity and routes through the same queue, with the same priority ceiling and concurrency cap as the named key.
Security notes:
- Injection ports default to
127.0.0.1— accessible only from the local host. - Binding to a non-loopback address requires
allow_public_injection: true. This is appropriate behind a trusted network; be cautious exposing it to untrusted hosts. - If
allow_public_injection: trueANDauth.enabled: false, the proxy emits a startup security warning — any host on the network can consume GPU time with no credential. - The
Authorizationheader is stripped before forwarding to upstream — a token sent to an injection port is never relayed to Ollama.
Model-aware routing
When running multiple Ollama hosts (different GPUs or different model sets), the proxy can route each request to a host that already has the target model installed.
Installed, not loaded — and the difference matters. The router reads
GET /api/tags, which lists the models present on disk. It does not read/api/ps, which is what reports the models actually resident in VRAM. So this avoids sending a request to a host that would have to pull the model; it does not avoid cold-start latency, because a model installed on a host may still need loading into VRAM.The practical consequence: where every host has the same models pulled — the normal homelab case —
model_awarehas nothing to discriminate on and degenerates to weighted round-robin. Loaded-model routing via/api/psis tracked as an enhancement, not a promise.
ollama:
hosts:
- url: "http://forge:11434"
name: "primary"
weight: 2 # gets 2x the traffic of weight-1 hosts
model_sync_interval: 30 # seconds between /api/tags polls
- url: "http://helm:11434"
name: "secondary"
weight: 1
model_sync_interval: 30
routing:
strategy: model_aware # model_aware | round_robin
fallback: any_healthy # when no host has the model: pick any healthy host
model_poll_timeout: 3
How it works: a background poller hits GET /api/tags on each host every model_sync_interval seconds, maintaining a live (host → installed_models) map. Every host is polled every interval, whether or not it is currently reachable. Requests with a model field are routed to a host that has it. Weighted round-robin is deterministic (not stochastic) — a 2:1 weight ratio means exactly 2 requests to the heavy host for every 1 to the lighter host.
Requests without a model field use weighted round-robin across reachable hosts. If
no host is currently marked reachable, the proxy still picks one rather than refusing:
reachability is a cached observation, and one failed poll against a host that has since
recovered should not black-hole the proxy into 503s.
Fast-path invalidation: when a host returns "model not found" (404), the proxy immediately removes that (host, model) pair from the routing table — no waiting for the next poll cycle.
Startup: the proxy probes each host's /api/tags once before accepting requests. If no host responds, startup fails fast with a clear error.
Embedding cache
Repeated embedding requests (common in RAG, semantic search, and agent workloads) often re-embed the same strings. The embedding cache stores successful responses in Valkey (or any RESP-compatible store — Dragonfly is a supported drop-in) and serves hits without touching the queue or upstream.
embedding_cache:
enabled: true
backend: "redis://valkey:6379/0" # Valkey recommended; Dragonfly works as a drop-in
ttl: 86400 # seconds
max_entry_bytes: 32768 # skip caching responses larger than this
key_prefix: "oqp:embed:"
connect_timeout: 2
Scope: /api/embed and /api/embeddings only. /api/generate and /api/chat are never cached (non-deterministic, large, low repeat rate).
Cache key: SHA256 of model + \0 + canonical_json(input), truncated to 32 hex chars. Per-endpoint namespaces prevent cross-endpoint collisions (same text via /api/embed and /api/embeddings get separate keys — their response shapes differ).
Startup: if enabled: true, the proxy pings the backend at startup. If unreachable, startup fails fast. After startup, any RESP error degrades gracefully: logged at most once per minute, the cache is bypassed for that request, and no user request fails.
Metrics: oqp_embedding_cache_hits_total, oqp_embedding_cache_misses_total, oqp_embedding_cache_errors_total — all at /metrics with client, model, and endpoint labels.
keep_alive defaulting
Ollama unloads a model from GPU memory after 5 minutes of inactivity (configurable on the Ollama side). For bursty workloads — embeddings or agents that fire requests every few minutes — this causes expensive cold-load latency. The proxy injects a keep_alive value so the model stays loaded.
keep_alive:
default: "5m" # injected when the client doesn't send keep_alive
override: false # if true, always replace the client's value with default
Applies to: /api/generate, /api/chat, /api/embed, /api/embeddings.
Behavior: if override: false and keep_alive is absent in the request body, inject default. If override: true, always replace. Non-JSON bodies and bodies over max_request_body_mb pass through untouched.
Metadata fast path
/api/tags, /api/version, /api/ps, /api/show and / bypass the priority queue and
the worker pool. They are authenticated like everything else, then dispatched straight
upstream.
This matters more than it sounds. Before 0.4.0 every path entered the queue through the
catch-all handler, so with a small max_concurrent a UI polling /api/tags would queue
behind a multi-minute generation and appear to hang. Admission control should not be
spent on reads that carry no inference cost.
The trade-off: these paths are not subject to per-client concurrency caps. They are cheap reads against a local daemon, bounded by the shared HTTP connection pool.
Priority queuing
Three tiers: high, normal (default), low. Set the tier per-request:
X-Queue-Priority: low
Workers dispatch high before normal before low. Each tier has its own depth limit, max wait timeout, and high-watermark threshold for webhook events.
Consumer example:
# Background embedding job — uses low priority
import httpx
client = httpx.Client(
base_url="http://localhost:11435",
headers={
"Authorization": "Bearer sk-my-batch-key",
"X-Queue-Priority": "low",
}
)
The proxy caps the priority to the key's max_priority — a batch key configured with max_priority: low can't elevate itself to high regardless of what header it sends.
ollama-queue-proxy MCP questions
How do I use this with existing Ollama consumers?
It is a drop-in replacement; you only need to change your consumer's OLLAMA_HOST to point to the proxy address.
Can a client increase its own priority?
No, the priority ceiling is enforced server-side against the authenticated API key, so a client cannot promote itself by sending a header.