Reliability gap analysis: one bad key + stale models burned 70% of failover budget (WorkBuddy trace, 2026-09-14)

Summary

Tracing a real production session (WorkBuddy desktop client → local relay, 127.0.0.1:31415, DB requests table, 2026-09-14 02:33–03:02 UTC): 142 requests, 99 never-ok (70%). Every failure decomposes into three gaps in the current ladder logic. Two of them overlap with recently-landed fixes (#1201, #1143) that postdate v0.9.9 — verified NOT in the v0.9.9 desktop build — plus one genuinely unhandled provider wording.

Gap 1 — NavyAI "model does not exist" (400) is not caught by isModelNotFoundError

NavyAI's catalog served 24 models to this relay over the window; 0 succeeded, 65 errored. The recurring error is:

NavyAI API error 400: The model 'gpt-4.1' does not exist or is not supported for chat completions.

isModelNotFoundError (error-classify.ts:500) matches 'model does not exist' as a substring, but the quoted model id between model and does not exist breaks it:

"the model 'gpt-4.1' does not exist or is not supported..."
//  no contiguous substring 'model does not exist' — quoted id sits in between

So it falls through to provider_bad_request, skips only the single route, and every sibling model on the platform re-pays a round trip (each of these cost 2.5–10s). A regex like /model '?[^ ']*'? does not exist/ would catch this shape. Note #1143 fixed a sibling case ("No model found: ") with the same fall-through mechanism.

Also observed alongside: NavyAI API error 403: The Free plan is temporarily disabled due to abuse ×25 — this is exactly the case #1201 (bench whole key on account-suspended) fixes, confirming that fix matters on real traffic; it's just not in the shipped desktop build yet.

Gap 2 — slow-but-alive providers starve inside the 45s first-byte budget

Success TTFB by platform over the window:

11 successful requests had first byte after 45s. Meanwhile 12 exhaustion outcomes were timeout with "retry time budget (45000ms) expired with no first byte" — ollama/nemotron-ultra was killed at 45s and the same route served identical requests in 47s minutes later. The default budget is indistinguishable from the healthy TTFB of the relay's most-relied-on models. Since the budget is already runtime-tunable (fallback_time_budget_ms), a doc note or raising the default would both help; raising it is what actually fixed this instance (set to 120000 locally, immediate improvement).

Gap 3 — in-band error after a long attempt burns the whole budget before failover

Request id 1245: attempt 0 (nvidia nemotron-3-ultra) ran 140.7s before surfacing in-band provider error ... Service temporarily overloaded, leaving only scraps of budget for the next hop (kilo, truncated after 3.2s), then exhaustion at 144s. The in-band provider error classification correctly marks it retryable, but the attempt had no first-byte stall guard applied to it (error arrived late in the stream, not as a stall). Suggestion: consider an absolute per-attempt wall-clock cap independent of the request budget, or let the hedge-abort logic (HEDGE_BENCH_MIN_SILENT_FRACTION) also apply to in-band error frames that arrive with zero content deltas.

Data

Full error breakdown for the window (from the local requests table):

Happy to provide the full attempt-trace dump (request_attempts rows) if useful.

中文摘要

追踪一次真实会话（WorkBuddy → 本地 relay）：142 个请求 70% 最终失败。三个缺口：① NavyAI 的 400 "The model 'X' does not exist" 因模型 id 带引号插在中间，绕过了 isModelNotFoundError 的子串匹配，24 个模型逐个重付往返（#1143 修的是同类措辞的另一种写法，也没接住这条）；② 慢节点（ollama/nvidia TTFB 均值 57s）被 45s 无首字节预算反复误杀，本机调到 120s 后立竿见影；③ 单跳 in-band error 跑满 140s 才失败，把预算耗尽只给下一跳留了残渣。#1201（账号封禁熔断整个 key）确认正确且必要，但 v0.9.9 桌面版还没带上。附完整 attempt-trace 可随时提供。