#131279·hermes-agent

Multiplex: async delegation from an api_server session of a secondary profile is still dropped as permanently-gone (#114609 fix does not cover raw API sessions)

Author: northernhartdesignsCreated Oct 2, 2026Updated Oct 2, 2026
Labelstype/bugcomp/gatewaytool/delegateP2sweeper:risk-session-statesweeper:risk-message-deliveryarea/profiles

Summary

Under gateway.multiplex_profiles, a background delegate_task started from an api_server session of a secondary profile completes, and then the gateway terminally drops its delivery:

WARNING gateway.run: Async delegation <deleg_id> targets permanently-gone session <session_id>;
terminally dropping delivery (result remains in the delegation records).

The parent session is live. It exists only in profiles/<name>/state.db, has ended_at = NULL, and takes further turns afterwards. This is the same symptom as #114609, which was fixed in c632437c3bb3 (_completion_event_scope). That fix resolves the owning profile from the event's session source. An api_server session has no messaging source, so the fix does not apply to it, and the pre-flight still runs in the launch profile's scope.

As requested when #114609 was closed, here is the ledger row from a version that has the fix.

Environment

  • hermes-agent v0.21.4+canary.20260929T070217Z (666f313d1d, 2026-09-29), Windows.
  • One multiplexed gateway. The affected profile is a secondary profile, not the launch profile.
  • Client: our own server-side app. It talks to the profile through the native session door POST /api/sessions/{id}/chat/stream and first creates the session with POST /api/sessions {"id": <id>, "source": "api_server"}. The client supplies the session id, so session_history_delivery="1" (api_server.py, the native-session opt-in).

Ledger row (secondary profile's state.db, async_delegations)

field value
state completed
delivery_state pending
delivery_attempts 0
delivery_claim NULL
origin_session run_<hex>
parent_session_id / origin_session_id <the live session id>
origin_ui_session_id ''

The event's session_key is run_<hex>, with no agent:<profile>:… lane, and there is no platform, chat_type or chat_id.

Where it falls through (as read on our install)

  1. _completion_event_scope(evt) → _build_process_event_source(evt).
  2. session_key is run_<hex>: no session-store entry, no cached source, and _parse_session_key gives no profile. Because _raw_process_event_session_id(evt) is truthy, it returns None ("Raw API keys legitimately have no messaging source").
  3. source is None → async_launch_profile_scope_if_multiplexed(), which is the launch profile's scope.
  4. _classify_completion_target(parent_session_id) → self._session_db is the launch profile's state.db. The session is not there, so parent is None → "terminal" → drop.

Scale

On our install, 164 drop lines between 2026-09-21 and 2026-10-01. Every one is for api_server sessions of the same secondary profile. None are for the launch profile.

Expected

The completion is persisted as the durable next-turn row described in the #104582 fix (6530eb41), in the owning profile's state.db, so the session's next turn loads it.

Suggested direction

When the event has no messaging source, resolve the owning profile from the delegation itself. The dispatch ran inside that profile's scope, so the event (or the ledger row) can carry the profile name or home at dispatch time, and _completion_event_scope can bind it when source is None. Also, a parent that is missing from the launch profile's DB should not be classified as terminal without checking the profile that owns the ledger row (the same point raised in #114609).

Related

#114609 (same symptom, Discord, fixed by c632437c3bb3) · #104582 (api_server delegation delivery semantics) · #99055 (api_server async-delegation wake retries).

Happy to retest on a build with a fix.

Source: NousResearch/hermes-agent