server-beta: one hung provider call blocks the generation lane, and concurrency is hard-coded to 1

Runtime: server-beta (self-reports 13.23.1), BullMQ on external Redis, OpenRouter provider, six to eight concurrent Claude Code sessions on one project feeding the server_beta_generate_event lane.

Two related asks about the generation lane, both observed on the same day.

1. One hung provider call halts the whole lane

The lane runs a single BullMQ worker, and the provider call has no client-side timeout, so a request that never returns blocks every queued job behind it.

Observed 2026-09-16: the last line in the day's log is a lock with no matching completion, and nothing moves afterwards.

10:57:46.523 [SYSTEM] [bullmq:evt_363bc27c…] [generation] job locked for processing {jobId=a4e093ec…}
(no further lines)

While blocked: GET /api/health answers normally with active: 1, waiting climbing past 360, and — notably — stalled: 0, because the process is busy rather than dead, so BullMQ's stalled-job reclaim never fires. The bundled health check does spot it (⚠ 348 jobs queued >30m (worker stalled?)), but nothing recovers automatically; a launchctl kickstart of the server was the only way out, after which the locked job retried and the backlog drained normally.

Ask: bound the provider request (a timeout on the HTTP client, or a per-job TTL that fails the job into the existing retry path). Either turns "the lane is dead until a human notices" into one failed job.

2. No way to raise generation concurrency

ServerJobQueue's constructor takes concurrency and defaults to 1 (this.concurrency = e.concurrency ?? 1), and the generation lane is constructed without passing it, so the effective value is 1 with no environment override. Measured job times on this project: p50 16 s, p90 144 s, max 325 s — almost all of it waiting on the provider, so a small concurrency would scale nearly linearly.

Ask: an environment knob (for example CLAUDE_MEM_GENERATION_CONCURRENCY, default 1) passed through to the worker options.

Caveat worth noting in the design: on this account the provider itself rate-limits — 30 of 41 failures that day were OpenRouter rate limit (429) at concurrency 1 — so raising concurrency without backoff would mostly convert waiting into 429s. A knob plus retry with backoff on 429 and 5xx would be the useful combination; today a 429 consumes one of the three attempts like any other error.

Happy to test a patched build against this workload — it reproduces the queue depth easily.