[Feature] Per-layer (L1/L2/L3) model overrides and multi-provider LLM config for standalone mode
Is this feature related to a problem? | 该功能需求是否与某个问题相关?
We run a standalone gateway (single user, deployMode: standalone) and hit two operational needs that the current single-channel LLM config cannot express:
1. One model config cannot serve all four pipeline stages well. The stages have opposite requirements:
- L1 extraction must return strict JSON inside a small budget. Reasoning/thinking models tend to spend the output budget on reasoning: with
maxTokens: 4096we repeatedly observed truncated responses (finishReason=length→No JSON array found), and per #1395 those batches were then silently skipped. - L2 scene synthesis / L3 persona are judgement + synthesis tasks that benefit from a larger budget and from thinking enabled.
Today llm.model / llm.maxTokens / llm.timeoutMs are global, so tuning one stage degrades another. There is currently no way — not even per-layer maxTokens — to express this in the config.
2. Only one provider can be configured at a time. llm.baseUrl / llm.apiKey / llm.model describe exactly one endpoint, and the provider key is already taken by the transport mode (openai | proxy). Switching between, say, a monthly-plan gateway and a pay-as-you-go fallback means editing the yaml and restarting; there is also no way to declare which env var holds a given key, so keys get pasted into the yaml file.
Describe the solution you'd like | 描述你期望的解决方案
Two additions, independent and backwards compatible (both default to the current behaviour):
(a) Per-layer overrides — llm.layers.{l1,l2,l3}
llm:
model: deepseek-v4-flash # global default (unchanged)
maxTokens: 4096
layers:
l1: # strict JSON, no thinking, small budget
model: deepseek-v4-flash
maxTokens: 4096
disableThinking: true
l2: # synthesis: bigger budget + thinking
maxTokens: 16384
disableThinking: false
l3:
model: deepseek-v4-pro
maxTokens: 16384Semantics we use locally and would propose: any omitted field falls back to the global value; an unknown layer key under layers fails fast at startup (typos must not be silently ignored); disableThinking has to be applied where the request body is built (in our environment the AI SDK compatible path did not serialize it, so we inject it at the fetch layer).
(b) Multiple providers + explicit active selection
llm:
activeProvider: opencode-go # switch = edit this one line + restart
providers:
- name: opencode-go
baseUrl: https://…/v1
apiKeyEnv: TDAI_KEY_OPENCODE # key lives in env, never in yaml
models: [deepseek-v4-flash, kimi-k2.7-code]
extra:
headers: { x-opencode-session: "…" } # some gateways require a session header
disableThinking: false
- name: zhipu
baseUrl: https://…/v4
apiKeyEnv: TDAI_KEY_ZHIPU
model: glm-5.3-flashNaming note: since llm.provider already means transport mode, a new key (activeProvider + providers[]) is needed. Validation should fail fast when activeProvider is not present in providers[], or when the referenced env var is empty, instead of silently falling back to the global channel.
Describe alternatives you've considered | 描述你考虑过的其他方案
- Env vars only (
TDAI_LLM_MODEL, …): what we do today. It cannot express per-layer or per-provider settings, and it hides the active configuration from the yaml, which is otherwise the single source of truth. - Runtime management API (patch the LLM config without restart): rejected locally — it would add a writable endpoint to a port that is not authenticated by default, for a monthly-frequency operation where a restart is acceptable.
- Rely on the
proxytransport for every provider: works when a proxy is deployed, but a single-machine deployment should be able to point directly at any OpenAI-compatible endpoint.
Additional context | 补充说明
- Relation to #1395: (a) is the prevention side of that report. Keeping L1 on a strict-JSON model with its own budget, while L2/L3 use thinking models with a bigger one, removes one of the two ways we lost data.
- We already run (a) and (b) locally, implemented as a parse-boundary normalization:
activeProvider+providers[]are folded into the existingStandaloneLLMConfigshape while parsing the yaml, so the resolver layer (src/gateway/llm-resolver.ts) and every runner construction point need no changes. Happy to open PRs along that shape if the direction sounds right — we would suggest landing (a) first, since it is small and self-contained. - Validated on: Windows 11 · Node 22 · standalone · OpenAI-compatible endpoints from several vendors, including monthly-plan gateways.
(中文摘要)两层需求:① L1 提取要严格 JSON + 小预算 + 关思考,L2/L3 是综合任务要大预算 + 开思考,而现在 llm.model/maxTokens 是全局的,调一层就伤另一层(我们本地已实证:L1 开思考 + 4096 会截断,正是 #1395 丢数据的两条诱因之一);② 现在只能配一个 provider(provider 字段已被传输模式占用),切换要改 yaml + 重启,且没有 apiKeyEnv,密钥只能写进 yaml。建议新增 llm.layers.{l1,l2,l3}(分层覆盖、缺省回落全局、未知层名 fail-fast)与 llm.activeProvider + llm.providers[](baseUrl/apiKeyEnv/models/extra.headers/extra.disableThinking;activeProvider 未命中或 env 缺 key 即拒启)。两项都默认保持现状行为;我们本地已按"解析边界折叠、下游零改动"的方式实现,方向认可即可按此提 PR(建议先落分层这项小改动)。
Source: TencentCloud/TencentDB-Agent-Memory