[Bug]: a custom provider whose pool rows sit under custom:<name> loses key rotation after a fallback round trip, on fallback to it, and after /model
Pre-flight
- Searched existing issues and PRs by
_rebind_fallback_credential_pool,resolve_runtime_pool_key,custom_provider_pool_key_candidates,_candidate_pool_exhausted, "fallback custom provider credential pool", "restore primary credential pool custom", "credential pool custom legacy key slug". #104457 fixes the fallback site only and resolves the key withresolve_runtime_pool_key, which returns the slug even when it is empty; #127759 changes onlyswitch_model, the same way. #124906 / #124578 / #124593 are about a same-URL sibling's pool, a different problem in the same functions. - Confirmed on current
main(95db0e2ada) -
hermes debug share: N/A. Reproduced with a real config, a realauth.json, a realAIAgentandresolve_runtime_provider, fake keys, all traffic to 127.0.0.1.
Bug Description
A configured custom provider can keep its credential-pool rows under the providers.<key> slug or under the legacy custom:<display-name> key (the layout credential-pools.md documents). Startup (hermes_cli/runtime_provider_custom.py) tries both and takes the one that holds credentials. Every later rebind loads a single key:
- Restore after fallback:
restore_primary_runtime→_load_primary_poolusesresolve_runtime_pool_key(), which returns the slug even when it is empty. After one fallback round trip the primary comes back on an empty pool, and 429/401 key rotation is gone for the rest of the session (CLI, gateway, TUI and cron share this path). - Fallback to a custom provider:
_rebind_fallback_credential_poolcallsload_pool(fb_provider), so a fallback to a named custom provider with legacy rows (orprovider: custom+base_url) attaches no pool. /model:_swap_switch_runtimecallsload_pool(new_provider), with the same gap.
Steps to Reproduce
- A custom provider whose pool rows are stored under
custom:<name>(two keys), and a fallback provider. - Trigger a fallback, then let the primary restore.
runtime: custom http://127.0.0.1:9/v1 pool: custom:relay-display
before fallback: pool custom:relay-display entries 2
fallback activated: True openrouter
restored: True custom
after restore: pool relay entries 0 has_credentials FalseFallback to that provider:
attached pool: None entries: 0
api_key in use: fake-relay-key-1Expected Behavior
Restore, fallback and /model bind the same pool startup picked: the scoped key that holds the rows.
Actual Behavior
The restored primary runs on an empty pool, and a fallback or /model to the custom provider runs with no pool, so a 429 or 401 no longer rotates keys.
Affected Component
Agent Core (conversation loop, context compression, memory)
Messaging Platform (if gateway-related)
N/A (CLI only)
Debug Report
N/A, reproduced in code (see Pre-flight).
Operating System
macOS 26
Python Version
3.14
Hermes Version
main 95db0e2ada
Source: NousResearch/hermes-agent