Concurrent cross-model request handling breaks when memory fits only one model

Concurrent cross-model request handling breaks when memory fits only one model

Describe the bug

When omlx is configured with memory limits that can fit only one model at a time, concurrent requests for different models are either rejected with HTTP 507 or blocked for 20-37 seconds. The already-loaded model's inference also stalls because the asyncio lock serializes all dispatch during another model's load.

This is NOT the same as omlx handling 2+ concurrent streams on the same model (which works fine via per-engine scheduler queues). This is a separate failure mode.

Observed symptom: The orchestrator's Qwen agent appears to stop when it tries to call a Gemma model — no error, no output, just stalled until either the Qwen request completes or is killed by the memory enforcer.

To Reproduce

# Start omlx with aggressive memory guard (ceiling ~38-42GB)
omlx serve --model-dir ~/models --memory-guard aggressive

# Qwen3.6-35B (~21GB) + Gemma-4-12B (~11GB) = 32GB > ceiling → only one fits

# Load Qwen (primary model), then send a concurrent Gemma request
curl -H "Authorization: Bearer KEY" \
  http://localhost:8000/v1/chat/completions \
  -d '{"model":"qwen3-6-35b-a3b-oQ4-mtp","messages":[{"role":"user","content":"test"}],"stream":true}'

# While Qwen is running:
curl -H "Authorization: Bearer KEY" \
  http://localhost:8000/v1/chat/completions \
  -d '{"model":"gemma-4-12b-it-qat-4bit","messages":[{"role":"user","content":"test"}],"stream":true}'

Expected (buggy): Either 507 InsufficientMemoryError or a 20-37s stall. Expected (correct): Request queues behind Qwen, returns when memory is freed.

Expected behavior

Cross-model requests should queue and wait for the busy model to drain. The asyncio lock blocking already-loaded models during another's load is not expected — concurrent streams on different loaded models should both be able to serve inference.

What's actually happening

1. EnginePool.get_engine() holds an exclusive lock during the entire load sequence (engine_pool.py:664-780)

The asyncio.Lock (defined at engine_pool.py:135) wraps the entire body of get_engine() — including the pre-admission check, all eviction loops, _load_engine() calls, and Metal buffer operations. This means:

• Already-loaded models' requests are blocked during another model's load
• A request for an already-loaded Qwen engine waits at the lock while Gemma loads (or vice versa)

2. _find_lru_victim() skips models with active requests (engine_pool.py:832-858)

if e.in_use > 0:
    continue
if self._entry_has_active_requests(e):
    logger.debug(f"Skipping victim '{mid}': has active requests")
    continue

When concurrent requests target different models and memory fits only one, all models have active requests → no evictable victim found → the admission loop raises InsufficientMemoryError (HTTP 507).

This conflates two distinct failure modes:

• ModelTooLargeError — model alone exceeds ceiling → genuinely impossible → 507 is correct
• InsufficientMemoryError — model would fit on a clean process but current usage leaves no room → the blocking model is busy, not pinned, and will drain → 507 is wrong; this should be a bounded wait

3. Single-threaded MLX executor compounds contention (engine_core.py:105-126)

get_mlx_executor() returns ThreadPoolExecutor(max_workers=1) — intentional to prevent Metal command buffer races (issue #85). Model loading, GPU inference, and Metal cache cleanup all compete for one thread.

4. No process-wide cross-model queue exists

The scheduler's waiting queue (scheduler.py:5790-5848) is per-engine only. There's no process-wide queue for cross-model swap waiting. When memory forces a choice between models, the second request has nowhere to wait.

5. Server returns 507 for both conditions (server.py:971-976)

except ModelTooLargeError as e:
    raise HTTPException(status_code=507, detail=str(e))
except InsufficientMemoryError as e:
    raise HTTPException(status_code=507, detail=str(e))

Both exceptions map to HTTP 507, making it impossible for callers to distinguish between "this model can never fit" and "wait for the current request to drain."

Log Evidence (June 15-22, 2026)

These are from my own ~/.omlx/logs/ on an M3 Max (~64GB RAM, Metal cap 44.0GB).

June 17 — FATAL crash from executor contention

05:38 → Hard memory pressure detected, no evictable models (stuck ~9 minutes)
05:40:26 → Request abort initiated for Qwen3.6-35B
05:40:45 → Unloading Qwen3.6-35B (immediate abort)
05:41:45 → FATAL "Engine teardown timed out after 60s"
   └─ shutdown submitted to single-threaded executor while MTP generation was running on that thread (engine_core.py:942-948)
05:42:09 → Supervisor restarts the process (server died from FATAL exit)

June 21 — Qwen↔Gemma swap cycles (deployment transition)

17:32 — Gemma loaded alone: 6.0s (baseline)
18:02 — Qwen loaded alone: 7.4s (baseline)
18:02 → Gemma loaded BEHIND Qwen: 37.0s (6.1x slower)
18:03 → Qwen evicted (memory pressure)
18:18 → Qwen reloaded BEHIND Gemma: 20.8s (4.0x slower)
18:19 → Gemma reloaded: 32.3s
18:20 → Qwen evicted for prefill headroom
18:22 → Qwen reloaded: 9.7s
18:28 → Gemma loaded: 11.9s, Qwen evicted
19:05 → Qwen loaded: 20.9s, Gemma evicted
19:13 → Server restart, Qwen loaded alone (single model)
20:06 → Qwen loaded alone: 5.2s (back to normal)

Normal single-model load times were 5.2-7.4s. Gemma loaded behind Qwen took 37.0s (6.1x). Qwen behind Gemma took 20.8s (4.0x). The Qwen↔Gemma swap cycle ran for about an hour, then I rolled back to single-model Qwen operation just to get stable service.

June 16 — First documented hard-pressure abort at ceiling

19:03:44 → Hard pressure first detected (43.8GB < 44.0GB ceiling)
19:03:51 → EMERGENCY ABORT (44.9GB > 44.0GB)
   "Emergency memory pressure: aborted 1 in-flight request(s)"

Key log entries for reference

Environment

• macOS Version: 27.0.0 (Darwin)
• Hardware: Apple M3 Max, ~64GB RAM
• Metal cap: 44.0GB (iogpu sysctl limit)
• Memory guard: aggressive tier, ceiling 37.7-42.0GB (--memory-guard aggressive)
• max_concurrent_requests: 2 (typical configuration)
• Models in scope: Qwen3.6-35B (~21GB), Gemma-4-12B (~11GB)
• Memory fits: only ONE model at a time (21 + 11 = 32GB > ceiling)

Notes for reproducing

This happens when an orchestrator (OpenClaw/OpenCode/Codex, etc.) routes tasks across models in a multi-tier setup where the user's hardware can't hold both loaded at once. The tier-1 model (Qwen3.6) serves most requests fine until the tier-3 model (Gemma-4) is needed. At that point tier-3 requests get either a 507 or a long stall, and tier-1 inference also blocks because of the asyncio lock.

I checked the relevant paths before filing — engine_pool.py:664-780 (the lock), _find_lru_victim() at line 832, get_mlx_executor() at engine_core.py:105. The background enforcer already has the busy-victim machinery (pending_unload_reason in process_memory_enforcer.py:1234) but it's not hooked into the request admission path.