WebUI stalls under load: /api/session full-transcript reconciliation takes 4-7s and stacks threads on a single-threaded server

Summary

On a self-hosted instance (exp-v0.52.263, commit 3b9c632a), the WebUI repeatedly becomes unresponsive under sustained load. An external health probe sees the server answer 200 from a clean start, then accumulated request threads ramp until it stops responding; a systemd watchdog force-restart recovers it. Reproduced twice in one evening. 346 slow /api/session requests between restarts, another 211 after a clean restart in a ~90 min window (45 of them /api/session).

Root cause

Every GET /api/session — even the bounded ?msg_limit= tail-window path used for normal session switching — is slow, and the time is always concentrated in two stages that re-walk the entire session history:

Slow WebUI request completed: {"elapsed_ms": 5135.3, "method": "GET", "path": "/api/session",
  "stages": [{"t2_after_state_db_load": 0.0}, {"t3_after_model_resolve": 230.0},
             {"t4_after_compact_and_merge": 4680.4}, {"t5_after_redact": 187.0}]}

Slow WebUI request completed: {"elapsed_ms": 11556.9, "method": "GET", "path": "/api/session",
  "stages": [{"t4_after_compact_and_merge": 7165.2}, {"t5_after_redact": 4085.1}]}

Thread stack (captured from request_diagnostics.py) shows the time consumed inside sidecar reconciliation, on every load rather than incrementally:

Thread-460 (active): "process_request_thread"
    _message_visible_content_key (api/models.py:8984)
    _reconcile_api_content_sidecars (api/models.py:9402)
    merge_session_messages_append_only (api/models.py:9948)
    reconciled_state_db_messages_for_session (api/models.py:10628)
    regeneration_state (api/session_ops.py:232)

merge_session_messages_append_only is called on every session read and re-normalises full content for every message in that session (computing visible-content keys across content + api_content), then redact_session_data re-reads the entire settings file per request. With sessions of hundreds of messages, one in-progress stream plus several reconnect reloads (clients re-fetch after each dropped socket) launches many process_request_threads that stack faster than the single-threaded socketserver drains them, wedging the accept loop.

Contributing load (90-min window)

/api/projects       341
/api/settings       150
/api/health/agent   137
/api/models         112
/api/session/draft  110
/api/session         45 (each 1-5s+)

Suggestions

1. Make merge_session_messages_append_only / _reconcile_api_content_sidecars incremental — memoize per-session reconciliation keyed on max row id so a repeat read only processes rows newer than the last pass. The data is append-only; a full re-walk per read is not required.
2. Cache load_settings() reads — the function re-reads the settings JSON on every request path.
3. Backpressure in the frontend reconnect logic so a dropped socket doesn't fire N simultaneous /api/session reloads.

Environment

• Self-hosted, single node, served behind Tailscale on :8787
• Embedded Hermes agent sessions over the same process
• Memory peak 3.0G under load; ThreadingHTTPServer

Happy to attach full slow-request dumps / thread snapshots on request.