Stale per-session model_provider survives model changes and routes agent construction through OpenRouter

Summary

A session whose stored model_provider no longer matches the canonical owner of its stored model keeps that stale provider forever. Every agent construction for such a session tries to build a client for the stale provider, fails, and logs three warnings. With a stale openrouter provider this also marks OpenRouter unhealthy for 60s on each occurrence.

Environment

• hermes-webui master @ 94fd2da8
• hermes-agent 0.21.3 @ 69fd61b0ef
• Ubuntu/Debian host, hermes-webui.service (systemd user unit), browser UI on :8787
• Active provider in config.yaml: model.provider: nous; no OpenRouter key configured (OPENROUTER_API_KEY commented out in ~/.hermes/.env)

Symptoms

Three WARNINGs appear together, always during agent construction (not during a turn):

WARNING agent.auxiliary_client: Auxiliary client: PAID lane engaged for auxiliary task — OpenRouter
  fallback model 'deepseek/deepseek-v4.1-flash' is not a :free SKU and may incur real spend. ...
WARNING agent.auxiliary_client: Auxiliary: marking openrouter unhealthy for 60s (payment / credit error). ...
WARNING agent.auxiliary_client: resolve_provider_client: openrouter requested but OpenRouter credential
  pool has no usable entries (credentials may be exhausted)

Observed 4 times over ~34 days, each within the same second that a new session was written.

Root cause

1. Session metadata in ~/.hermes/webui/sessions/<id>.json carries a top-level model_provider field. Sessions created while the user ran an OpenRouter model keep "model_provider": "openrouter" after the model and the global provider change. On this host: 38 records pinned to openrouter; 13 of them name a live Nous model (deepseek/deepseek-v4.1-flash, deepseek/deepseek-v4-pro) while still declaring openrouter.

2. The WebUI preserves that stored lane instead of repairing it:

   复制

   canonical_model_provider_lane("deepseek/deepseek-v4.1-flash", "openrouter")
     -> ("deepseek/deepseek-v4.1-flash", "openrouter")     # preserved
   canonical_model_provider_lane("deepseek/deepseek-v4.1-flash", None)
     -> ("deepseek/deepseek-v4.1-flash", "nous")           # config lane, correct

3. That lane reaches agent construction. Stack trace captured by wrapping agent.auxiliary_client._try_openrouter and constructing an agent with the stored lane:

   复制

   agent/auxiliary_client.py:4636  _resolve_openrouter_branch
   agent/auxiliary_client.py:5066  resolve_provider_client
   agent/agent_init.py:832         _routed_client_kwargs
   agent/agent_init.py:921         _init_openai_client
   agent/agent_init.py:964         _build_client
   agent/agent_init.py:2282        init_agent

   _routed_client_kwargs calls resolve_provider_client(agent.provider or "auto", model=agent.model). With agent.provider == "openrouter" this takes the OpenRouter branch and emits the three warnings. Construction then raises RuntimeError: No LLM provider configured. Run 'hermes model' ....

4. A repair routine already exists but does not fire for this case: api/routes.py:6540 _repair_foreign_session_model_provider. Its early-return guard requires the cached catalog to name exactly one owning provider for the stored model, and returns resolved_provider in every other case — so a stale provider that the catalog no longer lists as an owner is never rewritten.

Steps to reproduce

1. Ensure a session record has model_provider set to a provider the user has no credential for, while model names a model served by the configured provider.
2. Open or create that session (the WebUI writes/reads the record, agent construction runs).
3. Observe the three warnings in ~/.hermes/logs/agent.log and the construction failure.

Minimal local check without the UI:

from agent.auxiliary_client import resolve_provider_client
resolve_provider_client("openrouter", model="deepseek/deepseek-v4.1-flash")   # emits the trio
resolve_provider_client("nous",       model="deepseek/deepseek-v4.1-flash")   # no warnings

Impact

• Log noise that reads as if the user configured OpenRouter; it is a stale local record.
• OpenRouter is marked unhealthy for 60s on each occurrence.
• Agent construction fails after the warning, before the per-turn layer repairs the provider — the failure is invisible in the UI because the turn then runs normally on the configured provider.
• A user with a valid OpenRouter key would silently bill OpenRouter for these sessions.

Suggested fix

When the stored provider does not appear as an owner of the stored model in the cached catalog, rewrite model_provider to the canonical owner (or clear it so the config lane applies) and persist it. The existing _repair_foreign_session_model_provider is the natural place; its guard currently makes this a no-op. Also consider dropping the model_provider field on a model change that re-resolves the lane.

Workaround

Set model_provider to the canonical provider for the stored model in the affected session records (back up first; the files live outside sessions/ so they are not read as sessions). Verified on 13 records here: every repaired lane then resolves with no OpenRouter warning.