[Bug] Request-log writer worker has no resourceLimits and re-queues the OOM batch at the head — logging dies silently and does not recover across restarts

Summary

The request-log writer worker is spawned without resourceLimits, so under sustained request volume it dies with Node's ERR_WORKER_OUT_OF_MEMORY:

Worker terminated due to reaching memory limit: JS heap out of memory

handleWriterFailure() then pushes the batch that just killed the worker back onto the head of the queue and restarts the worker. The fresh worker gets the same batch and dies again — a poison-batch loop.

The practical result is that request logging dies silently and permanently. The gateway keeps serving traffic and usage.sqlite keeps updating, so nothing looks broken, but request_logs stops gaining rows and every new bundle is dead-lettered. Because the backlog is on disk in the raw-trace spool, restarting the gateway does not recover it.

Environment

• @musistudio/claude-code-router 3.1.0 and 3.1.1 (verified byte-identical in this code path)
• Node v26.8.2, Arch Linux (kernel 7.2.6)
• Workload: Claude Code with heavy subagent fan-out, ~4k requests/hour
• observability: requestLogs: true, requestLogBodyCapture: "all", requestLogSuccessSampleRate: 1, requestLogMaxBodyBytes: 52428800 (all defaults)

Root cause

1 — the log workers have no heap cap. packages/core/src/observability/request-log-runtime.ts:708 (writer) and :848 (query):

const worker = new Worker(this.options.workerFile, {
  workerData: {
    dbFile: this.options.dbFile,
    mode: "writer",
    rawTraceSpoolDir: this.options.rawTraceSpoolDir
  }
});

No resourceLimits. Batches are structured-cloned into the worker (batchMaxBytes 4 MB / batchMaxItems 50, queueMaxBytes 128 MB), and the worker also holds SQLite bind buffers for them.

This is inconsistent with the project's own precedent — packages/core/src/routing/route-script-runtime.ts is the only file in the repo that sets resourceLimits, and it caps that worker at maxOldGenerationSizeMb: 64 / maxYoungGenerationSizeMb: 16 / stackSizeMb: 4. The log writer, which handles far more data, gets nothing.

2 — the failure handler re-queues the poison batch at the head. request-log-runtime.ts:797:

private handleWriterFailure(worker: Worker | undefined, error: Error): void {
  if (!worker || worker !== this.writerWorker) return;
  this.writerWorker = undefined;
  this.writerWorkerReady = undefined;
  void worker.terminate().catch(() => undefined);
  for (const [, batch] of [...this.inFlight].reverse()) this.queue.unshift(...batch.commands);
  this.inFlight.clear();
  rejectPending(this.writerRequests, error);
  if (this.closed) return;
  this.writerRestartCount += 1;
  const delayMs = Math.min(5_000, 100 * (2 ** Math.min(6, this.writerRestartCount - 1)));
  const timer = setTimeout(() => {
    this.ensureWriter().then(() => this.schedulePump(true)).catch(() => undefined);
  }, delayMs);
  timer.unref?.();
}

this.queue.unshift(...batch.commands) puts the failing batch first in line. There's no attempt counter on the batch, no split-and-retry, and no quarantine — so an OOM-inducing batch is retried forever at up to one attempt per 5 s.

There is also no worker.on("error") distinction between ERR_WORKER_OUT_OF_MEMORY and any other error, so an OOM is treated as a transient fault.

Observed behaviour

On a box running ~4k requests/hour with default observability settings:

• request_logs last row 12:58:24Z; zero rows written over the following 3+ hours while traffic continued

• usage.sqlite kept updating normally throughout — nothing else appeared broken

• gateway restart did not help: still zero rows after restart, because the backlog is in the on-disk spool

• raw-trace-spool/.ccr-dead-letter: 217 bundles, every one of them

  json复制

  {"acceptedAt":...,"attempts":0,"deadLetterReason":"inbox_capacity","lastError":"inbox_capacity"}

  attempts: 0 — these were dropped without ever being tried, purely because the drain was stuck.

• raw-trace-spool/.ccr-inbox: 174 bundles / 206 MB and growing

• disk: ~11 GB accumulated in 16 hours — request-log-bodies/ 7.7 GB, request-logs.sqlite 2.9 GB, spool 313 MB

Worth noting the bodies were not unusually large — max request body 2.63 MB, mean 0.42 MB, max response 8.42 MB, and request_body_truncated was 0 across all 16,831 rows, so the 50 MB requestLogMaxBodyBytes ceiling was never approached. This is driven by sustained volume, not by any single oversized payload.

Recovery required deleting raw-trace-spool/.ccr-inbox and .ccr-dead-letter — after that, rows started flowing again immediately and the inbox stayed at 0.

Suggested fix

1. Give the writer and query workers explicit resourceLimits, as route-script-runtime.ts already does — ideally sized from batchMaxBytes/queueMaxBytes rather than hardcoded.
2. Treat ERR_WORKER_OUT_OF_MEMORY as non-transient in handleWriterFailure: track per-batch attempts and, on repeat failure, split the batch or dead-letter it instead of re-queueing it at the head forever.
3. Re-queue failed batches at the tail, so one bad batch cannot block every subsequent one.
4. Surface the stall. Right now the only signal is an OOM line in a log; the UI shows a healthy gateway with a silently frozen request_logs. A counter for consecutive writer restarts / inbox_capacity dead-letters would make this visible.

Lowering requestLogSuccessSampleRate is an effective workaround for volume, and requestLogMaxBodyBytes does correctly truncate once lowered.

Related: #1534 — different call path (analyze() selecting body columns), same underlying theme of request-log memory pressure taking a process down.