run_journal has no retention policy: terminal runs accumulate forever (916 MB of completed-run logs on one install)

Summary

~/.hermes/webui/sessions/_run_journal/ has no retention policy of any kind. delete_run_journal() has exactly one call site — session deletion (api/routes.py, the delete handler) — and there is no TTL, no size cap, and no age sweep anywhere in the codebase.

A long-lived or pinned session therefore accumulates one {run_id}.jsonl per run forever. Because the journal is only cleaned when the whole session is deleted, a session you keep is a session whose journal grows without bound.

Measured impact (one real self-hosted install)

The run journal had grown larger than every conversation sidecar combined. It was only found by accident while diagnosing unrelated WebUI latency (see below) — nothing surfaces it, and no code path would ever have reclaimed it.

Breakdown by terminal state

98.6% of the footprint is completed runs. Only the non-terminal remainder is genuinely load-bearing for recovery.

Why this is a bug and not an accepted trade-off

Upstream has invested in this journal's resource hygiene but only on the write and read paths, never on retention:

• #5784 — stop leaking _WRITER_LOCKS on delete
• #5799 — make appends O(1) instead of O(n²) over a session's lifetime
• #6291 — cache unchanged run summaries
• #6139 — bound summary readers to a tail read
• #7291 — stop journaling metering bloat (open)

Each of those reduces the cost of a journal that already exists. None of them bounds how much of it exists. #3802/#3811 added cleanup on session delete, which is the one case that doesn't apply to a long-lived session.

Suggested fix

An age/size-based sweep, and the predicate must be the run's own terminal event — not file age, and specifically not directory mtime:

• terminal: true → safe to retire (the run finished and its output was persisted).
• non-terminal → never touch. These are the crashed-run payloads the journal exists to recover; deleting them silently destroys the user's only recoverable output.
• Directory mtime is not a usable age signal. It updates whenever any file inside changes, so an actively-written session always looks fresh while its dir can still be reaped out from under a live writer. Use the file's own mtime.

Note this is cheap to implement: session_journal_fingerprint() in api/run_journal.py is already stat-only and O(runs), and delete_run_journal() already handles the safe-removal + writer-lock-eviction path.

Why I believe it's safe to retire old terminal journals

Every journal reader — find_run_summary(), read_run_events(), _sse_replay_run_journal_gap_checked(), and the runtime_journal status payloads — is keyed by a specific stream_id for a live or reconnecting stream. Nothing enumerates historical runs, so a terminal run that no client can still be attached to has no reader.

Local workaround (for anyone else hitting this)

A cron'd script that retires terminal runs older than 14 days (3 days when the journal exceeds 400 MB), moving them to an archive rather than deleting. On the install above it took 916 MB → 498 MB (328 MB archived), touching terminal runs only.

⚠️ Worth noting for anyone copying a quick fix from elsewhere: the obvious shell approach —

find "$JOURNAL_ROOT" -mindepth 1 -maxdepth 1 -type d -mtime +7 | xargs rm -rf

— is unsafe twice over. It has no terminal check, so it destroys non-terminal recovery payloads; and directory mtime makes the age test unreliable. A version of exactly that ran on this install and irreversibly removed 31 directories (138 MB) in a single pass, including resumable runs. It should not be adopted as a pattern.

Related (found while checking for duplicates)

• #7081 (open) — long sessions blocking unrelated requests; its stack shows the journal write on the done path, which matches how these journals get large in the first place.
• #6881 (open) — server.py at 100 GB RAM / 100% CPU on long sessions; the same install above showed 14.7 GB peak on a 16 GB box from the session cache, and the journal compounds it.
• #5839 / #5854 (closed) — large run journal inflating the session response; those fixed the load path, not on-disk retention.
• #6568 (open) — unbounded growth in session sidecars; same class of problem, different store.

Environment

• nesquena/hermes-webui @ 8d628f4e (2026-09-15), installed via git, single-host systemd
• 421 CLI sessions / 348 WebUI sidecars / 564 run-journal files
• Python 3.11.15