Multiplex: async delegation from an api_server session of a secondary profile is still dropped as permanently-gone (#114609 fix does not cover raw API sessions)
Summary
Under gateway.multiplex_profiles, a background delegate_task started from an
api_server session of a secondary profile completes, and then the gateway
terminally drops its delivery:
WARNING gateway.run: Async delegation <deleg_id> targets permanently-gone session <session_id>;
terminally dropping delivery (result remains in the delegation records).The parent session is live. It exists only in profiles/<name>/state.db, has
ended_at = NULL, and takes further turns afterwards. This is the same symptom as
#114609, which was fixed in c632437c3bb3 (_completion_event_scope). That fix
resolves the owning profile from the event's session source. An api_server
session has no messaging source, so the fix does not apply to it, and the
pre-flight still runs in the launch profile's scope.
As requested when #114609 was closed, here is the ledger row from a version that has the fix.
Environment
- hermes-agent
v0.21.4+canary.20260929T070217Z(666f313d1d, 2026-09-29), Windows. - One multiplexed gateway. The affected profile is a secondary profile, not the launch profile.
- Client: our own server-side app. It talks to the profile through the native
session door
POST /api/sessions/{id}/chat/streamand first creates the session withPOST /api/sessions {"id": <id>, "source": "api_server"}. The client supplies the session id, sosession_history_delivery="1"(api_server.py, the native-session opt-in).
Ledger row (secondary profile's state.db, async_delegations)
| field | value |
|---|---|
| state | completed |
| delivery_state | pending |
| delivery_attempts | 0 |
| delivery_claim | NULL |
| origin_session | run_<hex> |
| parent_session_id / origin_session_id | <the live session id> |
| origin_ui_session_id | '' |
The event's session_key is run_<hex>, with no agent:<profile>:… lane, and
there is no platform, chat_type or chat_id.
Where it falls through (as read on our install)
_completion_event_scope(evt)→_build_process_event_source(evt).session_keyisrun_<hex>: no session-store entry, no cached source, and_parse_session_keygives no profile. Because_raw_process_event_session_id(evt)is truthy, it returnsNone("Raw API keys legitimately have no messaging source").source is None→async_launch_profile_scope_if_multiplexed(), which is the launch profile's scope._classify_completion_target(parent_session_id)→self._session_dbis the launch profile'sstate.db. The session is not there, soparent is None→"terminal"→ drop.
Scale
On our install, 164 drop lines between 2026-09-21 and 2026-10-01. Every one is for api_server sessions of the same secondary profile. None are for the launch profile.
Expected
The completion is persisted as the durable next-turn row described in the #104582
fix (6530eb41), in the owning profile's state.db, so the session's next turn
loads it.
Suggested direction
When the event has no messaging source, resolve the owning profile from the
delegation itself. The dispatch ran inside that profile's scope, so the event (or
the ledger row) can carry the profile name or home at dispatch time, and
_completion_event_scope can bind it when source is None. Also, a parent that
is missing from the launch profile's DB should not be classified as terminal
without checking the profile that owns the ledger row (the same point raised in
#114609).
Related
#114609 (same symptom, Discord, fixed by c632437c3bb3) · #104582 (api_server delegation delivery semantics) · #99055 (api_server async-delegation wake retries).
Happy to retest on a build with a fix.
Source: NousResearch/hermes-agent