bug(daemon): intermittent POST /session 504 — AcpSessionBridge newSession times out after 10000ms on shared ACP child

What happened?

Creating a new session intermittently fails in a running qwen serve deployment. The UI reports an error when the user submits the first message because POST /session fails before that message can be submitted to the new session.

The daemon reports an ACP newSession timeout, not a model response timeout:

qwen serve: newSession timed out after 10000ms on channel <channel-id>; decision=fence_shared

[WARN] [DAEMON] route=POST /session
  errorType=BridgeTimeoutError
  AcpSessionBridge newSession timed out after 10000ms

[WARN] [DAEMON] route=POST /session
  clientId=webui_<client-id> durationMs=10010 status=504

The forwarding service recorded the same request as HTTP 504 after 10014ms, consistent with the inner daemon timeout being propagated. Increasing an outer HTTP/proxy timeout alone would not remove this inner deadline.

Identifiers in the excerpts are placeholders. Error strings and measured durations are preserved from the observed incident.

Supporting evidence

The same running ACP child had intermittent responsiveness problems around the incident:

The event-loop stall observations are not samples taken inside the exact failing newSession call. They establish concurrent responsiveness issues, but do not identify the function responsible for this specific timeout.

Inspection of the installed bridge bundle confirmed that its newSession request uses initTimeoutMs and rejects with BridgeTimeoutError("newSession", initTimeoutMs). The incident's effective deadline was 10000ms.

Reproduction status

Observed in a downstream deployment; not yet reproduced on an unmodified upstream release, and no deterministic minimal reproducer is available.

Observed workflow:

1. Run a daemon with a shared ACP child and existing sessions.
2. Use the web UI to create another session and submit its first message.
3. Intermittently, POST /session returns 504 at the 10-second internal deadline.
4. Check daemon logs for newSession timed out and decision=fence_shared.

No user prompts, session transcripts, credentials, internal endpoints, or raw diagnostic bundles are attached.

What did you expect to happen?

• New-session creation should remain usable under transient load, with a clearly defined and configurable creation budget if it differs from the ACP handshake budget.
• When the deadline expires, clients should be able to distinguish an unstarted request from an operation that may still complete in the ACP child, so they can avoid duplicate sessions or unsafe automatic retries.
• Diagnostics should identify the slow creation phase (for example, queued work, configuration/extension refresh, or initialization) and expose the settlement outcome after the public timeout.
• Existing sessions sharing the child should remain usable when a new-session operation times out.

Client information

• Linux container, CPU quota equivalent to 2 CPUs.
• Installed Qwen Code package version: 0.24.0-dataworks.0 (downstream distribution; upstream commit mapping not established).
• qwen serve daemon with an ACP child and embedded web UI.
• Full /about output was not captured; the version above was read from the installed package metadata.

Login information

Authentication mode was not collected. Authenticated instance access succeeded; the observed failure was the daemon's BridgeTimeoutError. Credentials are intentionally omitted.

Additional context / possible confounders

Earlier, this container exhausted its inotify watch quota (8192 watches). To restore a separate MCP controller's configuration watcher, Qwen's chokidar watchers were switched to polling and Qwen was restarted:

CHOKIDAR_USEPOLLING=1
CHOKIDAR_INTERVAL=1000
CHOKIDAR_BINARY_INTERVAL=1000

The new-session timeout above occurred after that change. Polling may contribute overhead and should be considered when reproducing. However, an ACP event-loop stall of 3592.421375ms and an extension-refresh timeout were also logged before the change. We have not established polling as the cause.

A later 5-second CPU sample showed no increase in cgroup CPU throttling counters. This was not the failure interval, so it cannot rule out resource pressure during the incident.

Related: #12021. That report concerns existing-session load / channel.initialize timeout. This report concerns POST /session / newSession timeout on a shared child, with decision=fence_shared. We are not claiming the same cause or a child SIGKILL in this incident.

Requested investigation

1. Confirm the intended newSession deadline and its relationship to the initialization timeout.
2. Add or identify phase-level diagnostics for creation latency and event-loop stalls.
3. Verify that late completion / fencing / cleanup after a public timeout preserves existing sessions and does not leak or duplicate the requested session.
4. Add a regression scenario where a shared child's newSession response exceeds 10 seconds; document the expected client recovery behavior.

Expanded incident timeline: daemon startup → failed creation → later successful creation

This section correlates the daemon's own log, its captured stderr, and the forwarding service's access log. T+00:00.000 is the new daemon's HTTP-listening event; calendar times, process IDs, ports, workspace paths, and all request/session/channel identifiers have been removed. Relative order and measured durations are retained. S-A and S-B represent different existing sessions; R-fail and R-success represent different creation requests. Times from the forwarding service have second precision and are marked approximately.

This is the complete relevant lifecycle timeline available from these logs, not an unfiltered raw-log dump or a claim that every internal phase was logged. The daemon also reports suppressed access-log entries; exact ACP spawn/handshake and extension-scan start/end timestamps are not present in the captured evidence.

Does daemon startup scan extensions?

The installed runtime does perform awaited extension discovery during configuration initialization, but HTTP startup, configuration initialization, catalog reads, and explicit refresh requests are different events.

The installed downstream bundle contains this sequence in Config.initializeInternal (equivalent source location: packages/core/src/config/config.ts):

recordStartupEvent("config_initialize_extensions_initial_start");
if (!this.isSafeMode() && !this.getBareMode()) {
  await this.extensionManager.refreshCache();
} else if (!this.isSafeMode() && explicitExtensionNames.length > 0) {
  await this.extensionManager.refreshCache({ names: explicitExtensionNames });
}
recordStartupEvent("config_initialize_extensions_initial_end");
options?.signal?.throwIfAborted();
// Hook initialization follows.

The installed ExtensionManager.refreshCache() delegates to refreshCacheWithSnapshot(). Without a name filter, that method calls:

await this.extensionStore.readConsistent(async () => {
  const loaded = await this.loadExtensionsFromExtensionsDir(
    this.configDir,
    this.workspaceDir,
  );
  // Build the next extension cache and apply activation state.
});

The implementation distinguishes a cheap directory fingerprint from a full refresh; its comment describes refreshCache() as parsing manifests and re-listing extension skill directories. Thus the awaited refresh includes disk discovery, rather than only reading an already-populated in-memory map. This alone does not mean all extensions are activated, all MCPs are started, or remote updates are downloaded.

Separately, the installed bridge exposes refreshExtensionsForAllSessions(): it enumerates live sessions, issues the workspace-extension-refresh control method to their ACP connections, and tracks in-flight refreshes by session and refresh scope. Please investigate how this work is scheduled relative to newSession on a shared child. We are not claiming there is no refresh deduplication.

What is still missing: per-phase timestamps for the failed initialization, extension count/scan duration for that operation, lock/queue wait duration, and the final settlement/cleanup of R-fail. The later successful request does not answer the settlement question. Please correlate the startup markers (extensionsInitialStart/End, hooks, skills, final extensions, etc.) with session-level initialization deadlines so a future occurrence identifies the actual slow phase.

Additional attribution caution

Host event-loop lag warnings also occurred during this window (including 3015ms, 3815ms, and later 9087ms). These are from a different process than the ACP child's 11785.994239ms stall and should not be merged into a single measurement. The combination supports investigating shared runtime load and scheduling, but the current evidence does not prove extension scanning, polling, or host integration activity individually caused R-fail.

Startup and session-initialization work inventory (extensions, skills, MCP, JSONL, and other phases)

This inventory is based on the installed downstream JavaScript bundle, in addition to the observed timeline above. It describes execution paths and their conditions; it is not a claim that every conditional branch ran during R-fail. Per-phase timings were not captured for that request. In particular, daemon HTTP startup and per-session Config.initialize are separate lifecycle boundaries.

1. Daemon boot and deferred runtime

The observed restart proceeds through process launch → HTTP listening → health response → deferred runtime scheduling → workspace roots / reaper / ACP endpoint setup. The installed run-qwen-serve implementation guards startRuntime() against duplicate starts, clears the scheduling timers, calls buildRuntime(), retains the bridge/app, optionally starts bridge preheat, then completes runtime startup. Runtime construction failure follows its startup-failure cleanup path.

When preheat is enabled, bridge.preheat() runs asynchronously; on success it schedules workspace MCP discovery/runtime setup. Its failure message says the ACP preheat will be retried on the first session. Thus HTTP listening, mounted runtime, ACP preheat, and MCP readiness are distinct signals. Exact preheat/handshake completion times and whether all optional boot work ran are unavailable in the incident logs.

Host integration separately requests extension synchronization, extension/skill catalogs, MCP reload, and update checks, as shown in the timeline. These requests can overlap runtime/session activity; they are not all intrinsic daemon boot operations.

2. New-session path and awaited configuration work

The installed ACP newSession() validates/reserves a requested session ID, creates the session-initialization deadline, and profiles these phases in order:

settings_load
  → config_setup (newSessionConfig / Config initialization)
  → auth (non-provisional session)
  → file_system_setup (non-provisional session)
  → session_register (createAndStoreSession)
  → response_build

The deadline signal is checked between stages and passed into configuration/session creation. Failure has configuration cleanup and deadline/reservation disposal paths. The following work exists inside configuration initialization: