Gateway masks upstream context-length errors, breaking Claude Code auto-retry/compact ("All target providers failed")

English below / 英文在下方

问题描述

当上游 LLM 返回 context length 超限错误(如 vLLM/OpenAI 兼容端点的 400)时,CCR 网关的 fallback 聚合逻辑会用通用的 "All target providers failed." 掩盖真实错误。Claude Code CLI 无法识别这类报错,因此:

• 不会自动降低 max_tokens 重试——而面对 Anthropic 官方格式的超限错误,CLI 本来会这样做;
• 不会触发 auto-compact——重试 payload 不变,input tokens 也一样,导致请求永远卡在同一处失败。

对长会话(尤其经 CCR 接自托管/第三方 OpenAI 兼容后端)来说,用户只能手动 /clear 开新会话,上下文直接丢失。

真实案例(生产日志捕获)

上游 vLLM 风格的 400 报文:

{"object":"error","type":"BadRequestError","message":"Requested token count exceeds the model's maximum context length of 262144 tokens. You requested a total of 262462 tokens: 230462 tokens from the input messages and 32000 tokens for the completion."}

CCR 返回给客户端的却是:

{"error":{"message":"All target providers failed.","target_providers":["provider-infor-8080::openai_chat_completions"],"attempts":[...]}}

同一会话在 230,462 input tokens 处连续三次 400(间隔 21 分钟、23 分钟),每次都是完全相同的请求体——没有任何自愈行为。

根因

Claude Code CLI 识别这个精确格式后会自动自救:

input length and `max_tokens` exceed context limit: <input> + <max_tokens> > <limit>

解析点(@anthropic-ai/claude-code 内)会提取 <input>、<max_tokens>、<limit>,然后自动以 max_tokens = limit - input - 1000 重试。但 CCR 的聚合错误把上游报文替换成了通用消息,CLI 永远看不到上述格式。

修复建议

1. 在网关的错误分类层识别 context-length 超限(覆盖 vLLM / OpenAI / LiteLLM 等常见格式),然后把最终错误翻译成上述 Anthropic 风格格式。一个最小可行补丁(已在 @musistudio/claude-code-router 3.1.0 的聚合函数中本地验证,改写前先尝试从最后一条 attempt 的 details.message / message 提取):

message: (() => {
  try {
    const d = last && (last.details && last.details.message || last.message);
    if (typeof d === "string") {
      const m = /maximum context length of (\d+) tokens/.exec(d);
      const y = /(\d+) tokens from the input messages and (\d+) tokens for the completion/.exec(d);
      if (m && y) return `input length and \`max_tokens\` exceed context limit: ${y[1]} + ${y[2]} > ${m[1]}`;
    }
  } catch {}
  return "All target providers failed.";
})(),

本地打上此补丁后,同一场景下 Claude Code 立刻自动降 max_tokens 重试成功,死循环消失。

1. 更彻底的方案:识别到超限后,网关自己降 max_tokens 重试(限 1 次),再失败才返回翻译后的错误。这对不认识该格式的客户端(opencode 等)也有帮助——不过这会改请求体、动核心路径,可以作为后续讨论。

2. 该修复与 #1773(在聚合错误中透出 per-attempt 细节)方向一致,可协同实现;与 #1178 中社区 transformer 手工合成超限错误触发 compact 的做法相比,在网关层统一解决可以让所有 provider 受益。

Problem

When the upstream LLM returns a context-length-exceeded error (e.g. 400 from a vLLM/OpenAI-compatible endpoint), the gateway's fallback aggregation replaces it with the generic "All target providers failed.". Claude Code CLI cannot recognize that error, so:

• it never retries with a lower max_tokens — which it does do for Anthropic-format context-limit errors;
• it never triggers auto-compact — the retried payload is byte-identical, so the request fails at exactly the same place forever.

For long sessions routed through CCR to self-hosted / third-party OpenAI-compatible backends, the only escape is /clear and losing the context.

Real-world capture (production logs)

Upstream (vLLM-style) 400 body:

{"object":"error","type":"BadRequestError","message":"Requested token count exceeds the model's maximum context length of 262144 tokens. You requested a total of 262462 tokens: 230462 tokens from the input messages and 32000 tokens for the completion."}

What CCR returned to the client:

{"error":{"message":"All target providers failed.","target_providers":["provider-infor-8080::openai_chat_completions"],"attempts":[...]}}

The same session hit 400 three times at 230,462 input tokens (21 and 23 minutes apart), each time with an identical payload — no self-healing of any kind.

Root cause

Claude Code CLI self-heals when it sees exactly this format:

input length and `max_tokens` exceed context limit: <input> + <max_tokens> > <limit>

It parses <input>, <max_tokens>, <limit> and retries automatically with max_tokens = limit - input - 1000. CCR's aggregated error hides the upstream message, so the CLI never sees that format.

Suggested fix

1. Classify context-length errors at the gateway layer (covering common formats from vLLM / OpenAI / LiteLLM) and translate the final error into the Anthropic-style format above. A minimal patch (verified locally against @musistudio/claude-code-router 3.1.0, inside the aggregation functions, extracting from the last attempt's details.message / message first):

message: (() => {
  try {
    const d = last && (last.details && last.details.message || last.message);
    if (typeof d === "string") {
      const m = /maximum context length of (\d+) tokens/.exec(d);
      const y = /(\d+) tokens from the input messages and (\d+) tokens for the completion/.exec(d);
      if (m && y) return `input length and \`max_tokens\` exceed context limit: ${y[1]} + ${y[2]} > ${m[1]}`;
    }
  } catch {}
  return "All target providers failed.";
})(),

With this local patch applied, Claude Code immediately retried with a lowered max_tokens and succeeded; the loop disappeared.

1. More thorough option: on detection, the gateway itself could retry once with a reduced max_tokens before returning the translated error. That also helps clients that don't understand the format (e.g. opencode) — but it mutates the request body on the core path, so it may deserve separate discussion.

2. This aligns with #1773 (surfacing per-attempt details in aggregated errors) and could be implemented together; compared with the community transformers in #1178 that hand-craft context-limit errors per provider, solving it once at the gateway layer benefits every provider.