#131292·hermes-agent

[Bug]: a custom provider whose pool rows sit under custom:<name> loses key rotation after a fallback round trip, on fallback to it, and after /model

Author: jonpol01Created Oct 2, 2026Updated Oct 2, 2026
Labelstype/bugcomp/agentarea/autharea/configP2sweeper:risk-security-boundary

Pre-flight

  • Searched existing issues and PRs by _rebind_fallback_credential_pool, resolve_runtime_pool_key, custom_provider_pool_key_candidates, _candidate_pool_exhausted, "fallback custom provider credential pool", "restore primary credential pool custom", "credential pool custom legacy key slug". #104457 fixes the fallback site only and resolves the key with resolve_runtime_pool_key, which returns the slug even when it is empty; #127759 changes only switch_model, the same way. #124906 / #124578 / #124593 are about a same-URL sibling's pool, a different problem in the same functions.
  • Confirmed on current main (95db0e2ada)
  • hermes debug share: N/A. Reproduced with a real config, a real auth.json, a real AIAgent and resolve_runtime_provider, fake keys, all traffic to 127.0.0.1.

Bug Description

A configured custom provider can keep its credential-pool rows under the providers.<key> slug or under the legacy custom:<display-name> key (the layout credential-pools.md documents). Startup (hermes_cli/runtime_provider_custom.py) tries both and takes the one that holds credentials. Every later rebind loads a single key:

  • Restore after fallback: restore_primary_runtime → _load_primary_pool uses resolve_runtime_pool_key(), which returns the slug even when it is empty. After one fallback round trip the primary comes back on an empty pool, and 429/401 key rotation is gone for the rest of the session (CLI, gateway, TUI and cron share this path).
  • Fallback to a custom provider: _rebind_fallback_credential_pool calls load_pool(fb_provider), so a fallback to a named custom provider with legacy rows (or provider: custom + base_url) attaches no pool.
  • /model: _swap_switch_runtime calls load_pool(new_provider), with the same gap.

Steps to Reproduce

  1. A custom provider whose pool rows are stored under custom:<name> (two keys), and a fallback provider.
  2. Trigger a fallback, then let the primary restore.
runtime: custom http://127.0.0.1:9/v1 pool: custom:relay-display
before fallback: pool custom:relay-display entries 2
fallback activated: True openrouter
restored: True custom
after restore: pool relay entries 0 has_credentials False

Fallback to that provider:

attached pool: None entries: 0
api_key in use: fake-relay-key-1

Expected Behavior

Restore, fallback and /model bind the same pool startup picked: the scoped key that holds the rows.

Actual Behavior

The restored primary runs on an empty pool, and a fallback or /model to the custom provider runs with no pool, so a 429 or 401 no longer rotates keys.

Affected Component

Agent Core (conversation loop, context compression, memory)

Messaging Platform (if gateway-related)

N/A (CLI only)

Debug Report

N/A, reproduced in code (see Pre-flight).

Operating System

macOS 26

Python Version

3.14

Hermes Version

main 95db0e2ada

Source: NousResearch/hermes-agent