Feature request: optional external resource lock for shared GPU coordination

Use case

I run Ollama in Docker on a Linux/Unraid server with a single NVIDIA RTX 3050 6 GB GPU. The GPU is also used by Subgen/Whisper, and I plan to add Frigate.

These applications do not need to use the GPU simultaneously. I would like them to safely alternate without stopping containers or risking two models loading into VRAM at the same time.

My Ollama configuration already limits its internal concurrency:

OLLAMA_MAX_LOADED_MODELS=1
OLLAMA_NUM_PARALLEL=1
OLLAMA_KEEP_ALIVE=1m

Ollama also provides useful existing controls:

• GET /api/ps to list loaded models
• keep_alive: 0 to unload a model after a request
• ollama stop <model> to unload a model
• Ollama’s internal request queue

However, these controls only coordinate requests inside Ollama. They do not provide atomic coordination with another GPU application.

The race condition

An external manager can:

1. Check /api/ps.
2. Confirm that no Ollama model is loaded.
3. Start another GPU application.

But an Ollama client could submit a new request immediately afterward and load a model into the same GPU.

Likewise, an external manager cannot prevent Ollama from accepting new GPU work while another application owns the GPU.

Proposed functionality

Would you consider adding an optional external advisory resource lock, configured through an environment variable such as:

OLLAMA_RESOURCE_LOCK=/vram-lock/gpu.lock

When configured, Ollama would acquire an exclusive Linux flock before loading or using a local model.

Suggested behavior:

1. Acquire the lock before allocating GPU memory or loading a local model runner.
2. Wait for the lock when another application currently owns it.
3. Hold the lock while any Ollama model occupies the protected GPU resource.
4. Cover native generate, chat and embedding APIs, as well as OpenAI-compatible endpoints.
5. Keep the lock throughout streamed responses.
6. On keep_alive: 0, unload the model and release the lock only after the runner has released its GPU resources.
7. On client cancellation or disconnection, retain the lock until the underlying operation and cleanup have actually completed.
8. When models remain loaded through a positive or negative keep_alive, continue holding the lock for that loaded-model lifetime.
9. If the configured lock path cannot be opened, fail closed with a clear error rather than silently bypassing the lock.
10. When the environment variable is unset, preserve Ollama’s current behavior completely.

Parallel Ollama requests could share the already-held lock internally through reference counting. The external lock should only be released after the last protected runner has unloaded.

An optional timeout setting could also be useful:

OLLAMA_RESOURCE_LOCK_TIMEOUT=30m

A timeout could return a clear 503 GPU resource busy response rather than waiting forever.

Expected coordination flow

For example:

Subgen acquires shared lock
Subgen loads Whisper and transcribes
Ollama request waits
Subgen unloads Whisper and releases the lock
Ollama acquires the lock
Ollama loads its model and generates the response
Ollama unloads the model and releases the lock

The applications would not need to know anything about one another. They would only need access to the same bind-mounted lock file.

Current workaround

I have a proof-of-concept HTTP proxy in front of Ollama that:

1. Acquires a shared flock.
2. Forwards the inference request to Ollama.
3. Drains the entire streamed response.
4. Sends keep_alive: 0.
5. Checks /api/ps to confirm that the model is gone.
6. Releases the lock.

This works, but it requires every Ollama client to use the proxy. A client connecting directly to Ollama bypasses the protection. The proxy also has to reproduce Ollama’s streaming, cancellation, timeout and API behavior.

Native support in Ollama would eliminate that extra proxy and automatically protect all clients.

Alternative designs

If a shared file lock is not suitable for Ollama, an official equivalent could also help, such as:

• A server-wide drain/pause API that stops accepting new inference work
• A safe unload-all?wait=true endpoint
• A lease API for reserving and releasing the local GPU
• Lifecycle hooks that run before model loading and after complete unloading

The essential requirement is an atomic way to prevent new Ollama GPU work while another local application owns the GPU, and to know that Ollama has fully released the resource before ownership changes.

Thank you for considering this.