[Question]: Session detail & paginated endpoints return compaction-archived messages — missing active = 1 filter causes 50+ MB payloads and UI freezes

Title

Session detail & paginated endpoints return compaction-archived messages — missing active = 1 filter causes 50+ MB payloads and UI freezes

Please describe your issue

The Web UI's Node-side session reader does not apply the active = 1 filter that hermes-agent uses for every live-context read. After a session goes through context compression, the pre-compaction transcript is kept on disk as tombstones (active = 0, compacted = 1) and is never deleted — but the Studio API returns the entire transcript instead of the live context.

Observed: opening a single session returns 17,132 rows / 53.4 MB of JSON, where only 159 rows / 0.57 MB are live — a 93× amplification. The browser then parses that JSON and renders ~17,000 DOM nodes, so the history view hangs for tens of seconds or appears frozen.

Affected endpoints:

Expected: session detail / paginated endpoints return the live context only (active = 1), matching SessionDB.get_messages(..., include_inactive=False); the session list's message_count equals the number of messages the detail endpoint returns; total/hasMore are computed over live messages only. The pre-compaction transcript should stay reachable through search (unchanged).

Measured impact — session 39b106d3b734, profile bioinfo-brain:

Across all 84 sessions of that profile:

Control: profiles with no compaction are unaffected (default profile: 34.6 MB → 32.2 MB) — the fix is a no-op where nothing was compacted.

Beyond performance, this is a data-trust issue: the UI currently displays turns that the agent itself cannot see, so a user reviewing history may believe the agent has context it does not. There is also a visible contradiction in the UI — the sidebar shows message_count = 159 while opening the session fetches 17,132 rows.

Context

Root cause

hermes-agent defines the contract explicitly. hermes_state.SessionDB.get_messages():

def get_messages(self, session_id, include_inactive=False, limit=None, offset=0):
    """Load messages for a session in insertion order.

    By default only active messages are returned. Pass ``include_inactive=True``
    to load soft-deleted rows (e.g. for audit / debug views of rewound history).
    ...
    When ``limit`` is provided, returns at most ``limit`` messages starting from
    ``offset`` ... Enables pagination for the API endpoint to avoid loading
    entire transcripts.
    """
    active_clause = "" if include_inactive else " AND active = 1"
    sql = f"SELECT * FROM messages WHERE session_id = ?{active_clause} ORDER BY id"

archive_and_compact() documents the tombstone semantics:

# Soft-archive the live turns: active=0 hides them from the live context load,
# compacted=1 marks them as "summarized away" (vs rewind/undo's
# active=0+compacted=0, which means "user took it back").
# search_messages includes compacted=1 rows by default so the pre-compaction
# transcript stays discoverable; live-context loads (active=1 only) still
# exclude them.
conn.execute("UPDATE messages SET active = 0, compacted = 1 "
             "WHERE session_id = ? AND active = 1", (session_id,))

The Studio re-implements this read in dist/server/index.js using node:sqlite, dropping both the active filter and the pagination contract. Shipped SQL:

// Fot / I_
SELECT * FROM messages
WHERE session_id IN (${ids})            // <-- no `AND active = 1`
ORDER BY CASE session_id ... END, id

// fot
SELECT COUNT(*) AS total
FROM messages WHERE session_id IN (${ids})            // <-- total counts tombstones

SELECT * FROM messages
WHERE session_id IN (${ids})            // <-- no `AND active = 1`
ORDER BY CASE session_id ... END DESC, id DESC
LIMIT ? OFFSET ?

// _y
SELECT * FROM messages WHERE session_id = ? ORDER BY id

Two aggravating factors

1. SELECT * — the payload also carries reasoning, tool_calls, api_content etc., which the client mapper (l_) only partially uses.
2. Silent client fallback to the unbounded endpoint — the history view first calls the paginated endpoint and, on any failure/404, falls back to /api/studio/sessions/hermes/{id} (unbounded, whole parent/child chain). Message-list rendering is also not virtualized, and images travel as base64 inside the JSON.

Also: the paginated endpoint reports total = 17132 with hasMore permanently true, so "load older" pages forever through tombstoned turns instead of terminating at 159. Finally, Fot/I_ have no LIMIT at all — a 100-session parent/child chain can be concatenated into one response.

Reproduction

-- 1. find sessions where compaction has fired
SELECT s.id, s.message_count,
       (SELECT COUNT(*)    FROM messages m WHERE m.session_id = s.id) AS rows_total,
       (SELECT SUM(active) FROM messages m WHERE m.session_id = s.id) AS rows_active
FROM sessions s ORDER BY rows_total DESC LIMIT 5;
-- e.g. 17132 / 159

-- 2. open that session in the Web UI and watch the payload
--    GET /api/studio/sessions/hermes/<id>?profile=<profile>
--    => tens of MB

To make any session exhibit this, lower the compression trigger (compression.threshold, default 0.50) and run a long tool-heavy session — or simulate directly:

UPDATE messages SET active = 0, compacted = 1 WHERE session_id = '<id>' AND id < <cutoff>;

Suggested fix

1. Minimal — add the filter (4 queries, 1 line each)

- WHERE session_id IN (${ids})
+ WHERE session_id IN (${ids}) AND active = 1
- FROM messages WHERE session_id IN (${ids})
+ FROM messages WHERE session_id IN (${ids}) AND active = 1

Also update _y; ideally replace the delete guard with a pure existence check (SELECT 1 FROM sessions WHERE id = ?) so it can never be affected by message-level tombstones.

2. Do NOT change the search paths

cZe / iZe / Eot scan messages_fts and must keep matching compaction-archived rows — that is the documented contract ((m.active = 1 OR m.compacted = 1) in search_messages). Only conversation-body reads should be filtered.

3. Recommended — stop re-implementing the agent's private schema

active / compacted are hermes-agent internals with real semantics (rewind/undo also writes active = 0, with compacted = 0). Re-deriving those rules in Node couples the Studio to the schema; state.db is versioned (schema_version table) and migrated by the agent, so this can silently drift again. Prefer delegating to SessionDB.get_messages(session_id, limit, offset) through the existing agent bridge, so there is a single source of truth.

4. Defence in depth

• Remove (or bound) the client's silent fallback from the paginated endpoint to the unbounded GET /api/studio/sessions/hermes/{id}.
• Put a limit on the chain-merged full-detail queries (Fot/I_ currently have no LIMIT).
• Select only the columns the client actually uses (l_ reads ~13 fields) instead of SELECT *.

5. UI hardening

HistoryMessageList maps every message to a DOM node — no virtualization. A 1,000+ message session is still slow even when all rows are live (post-fix max is ~4.8 MB / 1,084 messages).

6. Regression test suggestion

For any session where SUM(active) < COUNT(*):

assert len(session_detail(id).messages) == sessions[id].message_count

Local verification performed

• active IS NULL count: 0; rewind rows (active = 0 AND compacted = 0): 0 — so in practice active = 1 is exactly equivalent to "not compacted", while also correctly hiding rewound turns.
• With the filter applied, "returned rows == sessions.message_count" holds for 84/84, 85/85 and 11/11 sessions across three profiles.
• Query plan improves (uses the purpose-built index): idx_messages_session(session_id) → idx_messages_session_active(session_id, active).
• Sessions that would render empty after filtering: 0 (compaction always inserts the summary with active = 1).
• Non-destructive placeholder check on the same store: hermes sessions optimize reclaimed 140.7 MB (848.9 → 708.3 MB); the remaining bulk is the dual FTS5 index (trigram + standard, ~76% of the file), which is unrelated to this bug.

Environment (if applicable)

• Ekko Studio Version: 0.7.21
• Agent Runtime and Version (if applicable): hermes-agent 0.19.0 — affected sessions are Hermes-profile sessions (source = tui / cli), read directly from ~/.hermes/profiles/<profile>/state.db
• Operating System: Linux (container, x86_64)
• Node Version: v26.4.0